PristrenMLflow 2.x adds native LLM tracing, prompt versioning, and model registry support - bringing the same experiment discipline from ML training to LLM application development.
Mahmudul Haque Qudrati
CEO & ML Engineer
// tags
A practical guide to building reliable agentic AI systems covering structured outputs, observability, fallbacks, and cost controls with real code examples.
MLflow was built for traditional ML experiment tracking (hyperparameters, metrics, artifacts). In version 2.x it added first-class LLM support: automatic tracing of LLM calls, prompt versioning, and a model registry that understands fine-tuned checkpoints. If your team already uses MLflow for ML, adding LLM observability is a natural extension.
pip install mlflow openai
mlflow server --host 0.0.0.0 --port 5000
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
MLflow 2.x auto-patches OpenAI, LangChain, and LlamaIndex:
import mlflow
import openai
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("llm-experiments")
# Enable autologging - automatically traces all OpenAI calls
mlflow.openai.autolog()
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain HNSW indexing"}],
)
Every call appears as a trace in the MLflow UI with input, output, model name, token counts, and latency.
with mlflow.start_run(run_name="rag-v2-eval"):
mlflow.log_param("model", "gpt-4o-mini")
mlflow.log_param("chunk_size", 512)
mlflow.log_param("retrieval_k", 4)
# Run your RAG evaluation
scores = evaluate_rag_pipeline()
mlflow.log_metric("faithfulness", scores["faithfulness"])
mlflow.log_metric("answer_relevancy", scores["answer_relevancy"])
mlflow.log_metric("latency_p95_ms", scores["p95_latency"])
Compare run metrics in the MLflow UI to see which prompt or chunking strategy performs best.
Decorate any function to create a span:
@mlflow.trace(span_type="CHAIN")
def run_rag(question: str) -> str:
docs = retrieve_docs(question)
return generate_answer(docs, question)
@mlflow.trace(span_type="RETRIEVER")
def retrieve_docs(query: str) -> list[str]:
# vector search
return results
@mlflow.trace(span_type="LLM")
def generate_answer(docs: list[str], question: str) -> str:
# LLM call
return answer
Log a fine-tuned model checkpoint as a registered model:
with mlflow.start_run():
mlflow.log_artifact("./fine-tuned-llama-3.1-8b/", artifact_path="model")
mlflow.register_model(
model_uri=f"runs:/{mlflow.active_run().info.run_id}/model",
name="llama-3.1-8b-customer-support",
)
Promote versions through Staging → Production using the UI or API:
client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
name="llama-3.1-8b-customer-support",
version=3,
stage="Production",
)
pip install mlflow langchain-openai
mlflow.langchain.autolog()
# All LangChain chain/agent calls are automatically traced
Full documentation at mlflow.org/docs/latest/llms and the GitHub repo.
Chat, meetings, project management, and time tracking — all wired together with AI. Stop context-switching between five tools.
Try Zlyqor freePractical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
SpaceX is buying Cursor, the AI-powered code editor. The deal signals a shift in how AI coding tools are valued and deployed. Here's a practical breakdown of what's happening and what it means for developers.
