Installing your first local LLM: a small, useful starting point

A useful first local AI project is simply getting a model to answer a question on your own machine. That gives you a foundation for later work with retrieval, tools, and agents, while keeping the first troubleshooting task manageable.

The original home-lab setup used an Ubuntu virtual machine on Proxmox, 16 virtual CPUs, 16 GB of RAM, Ollama, and Llama 3 with 8 billion parameters. Those are the specifications of that experiment, not a minimum requirement or a performance guarantee for every model.

Establish one working path

Install Ollama using its official instructions for your operating system. Start the service, then download and run the model. These were the core commands in the original experiment:

ollama pull llama3
ollama run llama3

The first command downloads model files. The second opens an interactive session. Ask a small question, wait for the response, and check whether the speed is acceptable before adding a web interface. Model size, quantization, available memory, and CPU or GPU execution all affect the experience.

Keep the model name and runtime version in your notes. When you later change either, you will have a useful baseline for comparing output and latency.

Keep the network boundary explicit

Ollama binds to the loopback interface by default. That is enough when your client and model server run on the same machine. The original article also described connecting from a different computer, but remote access is a separate configuration decision.

The current Ollama FAQ documents the host setting and service configuration. Do not expose an inference endpoint publicly just to make a desktop client connect. Decide which machines need access and configure the network accordingly.

Add an interface after inference works

Open WebUI and AnythingLLM were the interfaces considered in the original roadmap. A graphical client can make ongoing conversations easier, but it also adds configuration and storage to understand.

Local inference can work offline after the necessary files are downloaded. Web search, remote tools, cloud models, and application telemetry have their own network behaviour. Check the whole application before describing the complete workflow as offline.

Once the basic conversation works, the next useful experiment is retrieval over a small local document.