A local RAG system becomes more useful when its input can include web pages. The original experiment extended a document-based notebook with Crawl4AI, then used the extracted text in the same retrieval and generation pipeline.
The historical stack combined Crawl4AI, LangChain, all-MiniLM-L6-v2 embeddings, SQLiteVSS, Ollama, and Llama 3 8B. A short Wikipedia page kept the initial example small enough to inspect in a notebook.
Treat crawling as document preparation
Fetching a page and extracting useful text are separate steps. A successful HTTP response can still leave you with navigation, duplicated sections, or missing content. Read the crawler’s output before splitting it into chunks.
Keep the source URL and collection time with the text. When a page changes, that information helps explain why an answer produced from an earlier snapshot differs from what a reader sees today.
Reuse the retrieval pipeline
The original notebook split extracted text, calculated embeddings, stored them in SQLiteVSS, and ran a similarity search. Retrieved passages then became context for a local question-answering prompt.
Web page → extracted text → chunks + source metadata
Chunks → embeddings → vector store
Question → relevant passages → local model responseThe retrieval stage selects text; the model reads that text. Adding a crawler does not make every response current or correct. The collection schedule, extraction quality, and retrieval result still determine which evidence reaches the model.
Check failures where they happen
Inspect one example at each boundary: page to text, text to chunks, question to passages, and passages to answer. If an answer is wrong, this makes it possible to locate the failure instead of repeatedly changing the final prompt.
Web content is input data. Instructions found inside it should not gain authority over your application or its tools. For a question that the collected material cannot answer, the useful response is an explicit gap in the evidence.
The recovered article retains the historical notebook steps. For a new implementation, consult Crawl4AI’s project for its current installation and extraction interfaces. Start with the local RAG foundation if you have not yet verified retrieval over a simple document.