What Is Ollama and Why It Matters

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

Ollama is an open-source runtime that lets you pull, run, and serve large language models on your own hardware - Mac, Linux, or Windows - with a single binary. No Python environment, no CUDA headaches to start, no data leaving your machine. For teams with privacy requirements or developers who want sub-100ms latency without paying per token, it is the fastest path from zero to a running model.

Installation

macOS: Download the .dmg from ollama.com or install via Homebrew:

bash

brew install ollama

Linux:

bash

curl -fsSL https://ollama.com/install.sh | sh

Windows: Download the installer from the Ollama GitHub releases page. WSL2 is supported and recommended if you need GPU acceleration.

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Pulling and Running Your First Model

Once installed, pull a model from the Ollama model library:

bash

ollama pull llama3.1:8b
ollama run llama3.1:8b

The run command opens an interactive chat. Type /bye to exit. For a one-shot query:

bash

ollama run llama3.1:8b "Summarize the CAP theorem in two sentences"

REST API at localhost:11434

Ollama exposes a REST API automatically when the daemon is running:

bash

curl http://localhost:11434/api/generate   -d '{"model":"llama3.1:8b","prompt":"What is PagedAttention?","stream":false}'

The API docs cover /api/chat, /api/embeddings, /api/pull, and more.

OpenAI-Compatible Endpoint

Ollama ships a /v1/ endpoint that matches the OpenAI API surface exactly. This means any library or tool that works with OpenAI will work with Ollama by changing base_url:

bash

curl http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"Hello"}]}'

In Python with the openai package:

bash

pip install openai

python

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "What is entropy?"}],
)
print(response.choices[0].message.content)

Modelfile: Custom System Prompts and Defaults

A Modelfile lets you bake in a system prompt, adjust temperature, or set a stop sequence, then ollama create a named model from it:

bash

cat > Modelfile <<'EOF'
FROM llama3.1:8b
SYSTEM "You are a concise technical writer. Never use bullet points."
PARAMETER temperature 0.3
EOF
ollama create tech-writer -f Modelfile
ollama run tech-writer

Docker Compose Example

For a containerised setup with GPU support:

bash

version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
volumes:
  ollama_data:

Memory Requirements by Model

Model	Quantization	VRAM / RAM
Llama 3.1 8B	Q4_K_M	~5 GB
Llama 3.1 70B	Q4_K_M	~40 GB
Mistral 7B	Q4_K_M	~4.5 GB
Gemma 2 27B	Q4_K_M	~16 GB

CPU-only inference works but is 3 - 10x slower. For interactive use, aim for at least a 16 GB M-series Mac or an NVIDIA GPU with matching VRAM.

Ollama is the fastest local LLM setup available today. Pair it with Open WebUI for a ChatGPT-style interface, or point any OpenAI SDK at localhost:11434/v1 to drop it into an existing app.

#ollama#local-llm#self-hosted#privacy#gpu

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.