Chroma: The Easiest Vector Database for LLM Apps

Updated
6 min read
Chroma: The Easiest Vector Database for LLM Apps
TL;DR

Chroma runs in-process with zero setup, embeds text automatically with a default model, and scales to a client-server deployment when you outgrow local mode.

Why Chroma for Prototyping

Chroma is the fastest way to add a vector store to a Python LLM app. It runs in-memory or on-disk with a single import - no Docker, no server, no external dependencies. When you are ready for production, flip to client-server mode with the same API.

Installation

bash
pip install chromadb

In-Memory Mode

python
import chromadb

client = chromadb.Client()  # ephemeral, in-memory
collection = client.create_collection("docs")

collection.add(
    documents=["PagedAttention manages KV cache as pages.", "HNSW is a graph ANN index."],
    ids=["doc1", "doc2"],
)

results = collection.query(
    query_texts=["how does LLM memory work?"],
    n_results=2,
)
print(results["documents"])

Chroma uses all-MiniLM-L6-v2 (via the chromadb default embedding function) automatically - you do not need to manage embeddings yourself.

Persistent Mode

bash
client = chromadb.PersistentClient(path="./chroma_db")

Data persists across process restarts in the specified directory. This is sufficient for single-server production deployments with millions of documents.

Metadata Filtering

python
collection.add(
    documents=["LangGraph is a state machine framework.", "Instructor adds Pydantic to LLMs."],
    metadatas=[{"category": "agents"}, {"category": "tooling"}],
    ids=["doc3", "doc4"],
)

results = collection.query(
    query_texts=["build an agent"],
    where={"category": "agents"},
    n_results=1,
)

The where filter uses MongoDB-style operators: $eq, $ne, $gt, $in, $and, $or.

Custom Embedding Function

Use any embedding model by wrapping it:

python
from chromadb import EmbeddingFunction
from sentence_transformers import SentenceTransformer

class LocalEmbedder(EmbeddingFunction):
    def __init__(self):
        self.model = SentenceTransformer("all-mpnet-base-v2")

    def __call__(self, input: list[str]) -> list[list[float]]:
        return self.model.encode(input).tolist()

collection = client.create_collection("docs", embedding_function=LocalEmbedder())

LangChain Integration

bash
pip install langchain-chroma
python
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma(
    collection_name="docs",
    embedding_function=OpenAIEmbeddings(),
    persist_directory="./chroma_db",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

Client-Server Mode for Production

bash
chroma run --host 0.0.0.0 --port 8000

Connect from your app:

python
client = chromadb.HttpClient(host="localhost", port=8000)

Chroma vs Qdrant vs Pinecone

Click any column to sort
4 rows
Self-hostYesYesNo
In-processYesNoNo
Hybrid searchNoYesYes
Production scaleMediumHighHigh

Full documentation at docs.trychroma.com.

#chroma#vector-database#embeddings#local#python

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.