released · 2026
CrawlSmart
A governed web crawling and question-answering application that follows site links, builds a searchable corpus, and exposes a conversational interface.
- RAG
- crawling
- Databricks Apps
- vector search
CrawlSmart turns a starting URL into a traceable knowledge corpus. It discovers linked pages, records crawl metadata, prepares content for retrieval, and exposes question answering over the resulting collection.
The project focuses on the less visible parts of a useful RAG demo: crawl boundaries, failed pages, source attribution, refresh behavior, and the ability to explain where an answer came from.