AgentOps: Monitor, Debug, and Optimize Your AI Agents in Production

The Observability Gap in AI Agents

When a traditional API fails, you look at logs. When an AI agent produces wrong output or fails mid-task, the failure is harder to diagnose: was it the LLM's reasoning? A tool call that returned bad data? An error in how results were parsed? AgentOps fills this gap by recording everything an agent does in a structured, replayable session.

Quick Setup

import agentops
from openai import OpenAI

agentops.init(api_key="YOUR_AGENTOPS_API_KEY")

client = OpenAI()

# All subsequent OpenAI calls are automatically tracked
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Plan a 3-day trip to Tokyo"}]
)

agentops.end_session("Success")

Every LLM call in the session is logged with: model name, prompt, completion, latency, token counts, and cost. Sessions appear in the AgentOps dashboard immediately.

Session Recording

A session captures the complete execution trace of an agent run:

• LLM calls - input messages, output, model, latency, tokens, estimated cost
• Tool calls - which tool was called, what arguments were passed, what it returned
• Agent decisions - the chain of reasoning that led to each action
• Errors - exceptions, API failures, parsing errors with full stack traces
• End state - Success, Fail, or Indeterminate

CrewAI Integration

from crewai import Agent, Task, Crew
import agentops

agentops.init(api_key="YOUR_KEY")

researcher = Agent(
    role="Researcher",
    goal="Find the latest AI developments",
    backstory="Expert research analyst",
)

# AgentOps automatically tracks all LLM calls made by CrewAI agents
crew = Crew(agents=[researcher], tasks=[research_task])
result = crew.kickoff()

agentops.end_session("Success")

Cost Attribution

AgentOps tracks per-session cost broken down by model and call type. For multi-agent workflows running at scale, this lets you identify which agents or tasks are responsible for the majority of your LLM spend - often the first step toward optimization.

Error Replay

When an agent run fails, AgentOps records the exact state at failure: what was in context, what tool was being called, what the error was. You can replay the failed session in the dashboard and step through the execution to identify the root cause.

AgentOps vs LangSmith

LangSmith is deeply integrated with the LangChain ecosystem and excels at tracing LangChain chains and agents. AgentOps is framework-agnostic (works equally well with CrewAI, AutoGen, direct OpenAI calls) and has stronger session-level cost tracking. If you're using LangChain heavily, LangSmith is the natural choice. For multi-framework or framework-free agent work, AgentOps provides better coverage.

Resources

• AgentOps GitHub
• Official docs