Retrieval-augmented generation adds relevant source material to a model’s input when a question arrives. A local RAG application combines a document collection, an embedding model, a retrieval store, and a language model running on your machine.
The original tutorial built this pipeline with Python, LangChain, Ollama, Llama 3, sentence-transformer embeddings, and SQLiteVSS. Its value is the visible sequence of operations: you can inspect what enters the system, what retrieval selects, and what the model says about it.
Build the document path first
Begin with a small text document. Load it, split it into passages, calculate embeddings, and store those vectors alongside the corresponding text. Keep a link or identifier for the source so a retrieved passage remains traceable.
Chunking is a practical decision. Very short chunks can separate an answer from its context. Very long chunks can dilute the relevant passage or crowd the model’s context window. Inspect a few chunks before treating the configuration as settled.
Inspect retrieval before asking for an answer
Embed the question using the compatible embedding model and run a similarity search. Read the returned passages directly. If they do not contain the answer, fluent generation will not repair the missing evidence.
Source document → chunks → embeddings → retrieval store
Question → query embedding → matching passages
Question + matching passages → local language model → answerThe language model receives retrieved text in its prompt. It does not read the vector database or answer directly from raw embedding vectors. The embeddings help the retriever find candidate passages; the passages provide the evidence.
Make uncertainty observable
A useful application lets readers compare the answer with its sources. Include questions whose answers are present, questions that require more than one passage, and questions the document cannot answer. The last group reveals whether the assistant admits missing information or fills gaps with a plausible story.
The original notebook used a retrieved-context prompt and a question-answering chain. Its package imports describe that historical environment; use the linked source when studying the notebook rather than assuming those imports match today’s libraries.
Once this small pipeline is understandable, web crawling can provide another document source. Keep the same inspection points when the input becomes larger or changes more frequently.