PristrenMLflow 2.x adds native LLM tracing, prompt versioning, and model registry support - bringing the same experiment discipline from ML training to LLM application development.
Mahmudul Haque Qudrati
CEO & ML Engineer
// tags
OpenAI Codex issue #2847 about excluding sensitive files from context is still unresolved. This post covers the problem, current workarounds, and why it's a critical missing feature for production use.
MLflow was built for traditional ML experiment tracking (hyperparameters, metrics, artifacts). In version 2.x it added first-class LLM support: automatic tracing of LLM calls, prompt versioning, and a model registry that understands fine-tuned checkpoints. If your team already uses MLflow for ML, adding LLM observability is a natural extension.
pip install mlflow openai
mlflow server --host 0.0.0.0 --port 5000
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
MLflow 2.x auto-patches OpenAI, LangChain, and LlamaIndex:
import mlflow
import openai
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("llm-experiments")
# Enable autologging - automatically traces all OpenAI calls
mlflow.openai.autolog()
client = openai.OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Explain HNSW indexing"}],
)
Every call appears as a trace in the MLflow UI with input, output, model name, token counts, and latency.
with mlflow.start_run(run_name="rag-v2-eval"):
mlflow.log_param("model", "gpt-4o-mini")
mlflow.log_param("chunk_size", 512)
mlflow.log_param("retrieval_k", 4)
# Run your RAG evaluation
scores = evaluate_rag_pipeline()
mlflow.log_metric("faithfulness", scores["faithfulness"])
mlflow.log_metric("answer_relevancy", scores["answer_relevancy"])
mlflow.log_metric("latency_p95_ms", scores["p95_latency"])
Compare run metrics in the MLflow UI to see which prompt or chunking strategy performs best.
Decorate any function to create a span:
@mlflow.trace(span_type="CHAIN")
def run_rag(question: str) -> str:
docs = retrieve_docs(question)
return generate_answer(docs, question)
@mlflow.trace(span_type="RETRIEVER")
def retrieve_docs(query: str) -> list[str]:
# vector search
return results
@mlflow.trace(span_type="LLM")
def generate_answer(docs: list[str], question: str) -> str:
# LLM call
return answer
Log a fine-tuned model checkpoint as a registered model:
with mlflow.start_run():
mlflow.log_artifact("./fine-tuned-llama-3.1-8b/", artifact_path="model")
mlflow.register_model(
model_uri=f"runs:/{mlflow.active_run().info.run_id}/model",
name="llama-3.1-8b-customer-support",
)
Promote versions through Staging → Production using the UI or API:
client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
name="llama-3.1-8b-customer-support",
version=3,
stage="Production",
)
pip install mlflow langchain-openai
mlflow.langchain.autolog()
# All LangChain chain/agent calls are automatically traced
Full documentation at mlflow.org/docs/latest/llms and the GitHub repo.
Chat, meetings, project management, and time tracking — all wired together with AI. Stop context-switching between five tools.
Try Zlyqor freePractical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Alibaba is reportedly banning Claude Code from its workplace due to alleged backdoor risks. This post explains the incident, the technical concerns, and the broader implications for AI coding assistants in regulated environments.
