Ollama is an open-source runtime that lets you pull, run, and serve large language models on your own hardware - Mac, Linux, or Windows - with a single binary. No Python environment, no CUDA headaches to start, no data leaving your machine. For teams with privacy requirements or developers who want sub-100ms latency without paying per token, it is the fastest path from zero to a running model.
Installation
macOS: Download the .dmg from ollama.com or install via Homebrew:
brew install ollama
Linux:
curl -fsSL https://ollama.com/install.sh | sh
Windows: Download the installer from the Ollama GitHub releases page. WSL2 is supported and recommended if you need GPU acceleration.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
The run command opens an interactive chat. Type /bye to exit. For a one-shot query:
ollama run llama3.1:8b "Summarize the CAP theorem in two sentences"
REST API at localhost:11434
Ollama exposes a REST API automatically when the daemon is running:
curl http://localhost:11434/api/generate -d '{"model":"llama3.1:8b","prompt":"What is PagedAttention?","stream":false}'
The API docs cover /api/chat, /api/embeddings, /api/pull, and more.
OpenAI-Compatible Endpoint
Ollama ships a /v1/ endpoint that matches the OpenAI API surface exactly. This means any library or tool that works with OpenAI will work with Ollama by changing base_url:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "What is entropy?"}],
)
print(response.choices[0].message.content)
Modelfile: Custom System Prompts and Defaults
A Modelfile lets you bake in a system prompt, adjust temperature, or set a stop sequence, then ollama create a named model from it:
cat > Modelfile <<'EOF'
FROM llama3.1:8b
SYSTEM "You are a concise technical writer. Never use bullet points."
PARAMETER temperature 0.3
EOF
ollama create tech-writer -f Modelfile
ollama run tech-writer
CPU-only inference works but is 3 - 10x slower. For interactive use, aim for at least a 16 GB M-series Mac or an NVIDIA GPU with matching VRAM.
Ollama is the fastest local LLM setup available today. Pair it with Open WebUI for a ChatGPT-style interface, or point any OpenAI SDK at localhost:11434/v1 to drop it into an existing app.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Open Code Review – An AI-powered code review CLI tool: A Practical Overview
Open Code Review is an open-source CLI tool from Alibaba that uses AI to review code changes. It runs locally, supports multiple LLMs, and costs about $0.01 per review. Here's a practical breakdown.
