How to Test AI Agents with Robust Mocks and Integration Suites

Testing traditional software relies on a simple assumption: given identical inputs, a function will always produce identical outputs. A unit test asserts expect(calculateDiscount(100)).toBe(80), and if the assertion fails, an engineer immediately knows there is a regression in deterministic business logic.

When engineering autonomous AI agents, that assumption collapses.

Language models are fundamentally probabilistic. The exact same user prompt executed three times can produce three distinct reasoning paths, alternative tool call arguments, and varying text formulations. Furthermore, testing agents against live commercial LLM APIs inside continuous integration (CI) pipelines quickly leads to flaky test suites, rate-limit throttling, and thousands of dollars in wasted cloud credits.

Shipping reliable agents requires fundamentally rethinking your quality assurance framework. You cannot test an agent simply by asserting exact text matches, nor can you treat the model as a completely opaque black box.

This guide provides a comprehensive systems blueprint for testing AI agents. We examine the three-tier agent testing pyramid, walk through contract-based mocking in TypeScript, explore trajectory evaluation, and demonstrate how to manage non-determinism using statistical benchmarks.

The Core Dilemma of Testing Non-Deterministic Software

Software engineering teams attempting to test AI agents for the first time usually make one of two critical mistakes:

1. The Brittle String Assertion: The test asserts that the model's final response contains an exact phrase like "Your ticket #1042 has been refunded". When the underlying model is updated or temperature variations introduce minor stylistic shifts, the test suite breaks even though the agent performed the refund correctly.
2. The Live-API CI Anti-Pattern: The test suite fires real HTTP requests to remote model endpoints on every pull request. Builds take 15 minutes to run, fail intermittently due to remote 503 errors, and burn through API budgets without isolating local code changes.

To build a resilient testing pipeline, teams must decouple deterministic system code (tool execution, payload routing, state machines, and schema parsing) from probabilistic model behavior (intent classification and semantic reasoning).

Traditional Unit Test:
Input ──> Deterministic Function ──> Single Expected Output (Binary Pass/Fail)

AI Agent Test:
Input ──> Probabilistic Reasoner ──> Trajectory of Tool Calls ──> State Mutation & Schema Validation

The Three-Tier AI Agent Testing Pyramid

Just as standard web applications rely on unit, integration, and end-to-end tests, a production agent architecture requires a three-tier testing hierarchy tailored to probabilistic systems.

                    ┌────────────────────────┐
                    │        TIER 3          │
                    │   End-to-End Trajectory │
                    │   & Statistical Pass^k │
                    │   (Nightly Staging)    │
                    └───────────┬────────────┘
                                │
                    ┌───────────┴────────────┐
                    │        TIER 2          │
                    │  Contract-Based Mocks  │
                    │  & Replay Fixtures     │
                    │  (PR Validation CI)    │
                    └───────────┬────────────┘
                                │
                    ┌───────────┴────────────┐
                    │        TIER 1          │
                    │ Deterministic Tools,   │
                    │ Schemas & State FSM    │
                    │ (Fast Local Unit Tests)│
                    └────────────────────────┘

Tier 1: Deterministic Tools and Schema Validation (100% Mock-Free)

Before evaluating an agent's reasoning, you must verify that the tools it invokes are rock-solid. Tools are standard deterministic software: database queries, REST API wrappers, calculations, and state mutations.

• Scope: Validate input schema parsers, error boundary wrappers, and output formatters.
• Execution Speed: Milliseconds.
• Cost: $0. Run locally on every file save.

Tier 2: Contract-Based Mocking and Replayable Fixtures

In this tier, you test how the agent's orchestrator handles external systems without making real network requests. You mock third-party services and isolate the agent's control loop.

• Scope: Test how the agent reacts when a database times out, when an external API returns a 429 rate limit, or when a tool returns an empty list.
• Execution Speed: Seconds.
• Cost: $0. Run on every Git push and pull request.

Tier 3: Trajectory Evaluation and Statistical Benchmarks

In this tier, the agent interacts with sandboxed environments or real model endpoints to evaluate complex multi-step reasoning.

• Scope: Evaluate whether the agent takes an optimal path to accomplish a goal, tests resilience against adversarial inputs, and calculates statistical pass rates.
• Execution Speed: Minutes.
• Cost: Measured token budget. Run on merge to main or as nightly scheduled jobs.

Building Contract-Based Mocks for LLM Tool Calls

The most effective way to test agent control loops in CI/CD is Contract-Based Mocking. Instead of mocking raw HTTP requests with brittle URL matchers, you mock the tool provider interface.

Below is an enterprise TypeScript implementation demonstrating how to build an isolated, replayable mock environment for an AI agent's tool execution harness.

import { z } from "zod";

// 1. Strict Contract Definition for an Agent Tool
export const SearchCustomerSchema = z.object({
  email: z.string().email(),
});

export interface ToolDefinition<TInput, TOutput> {
  name: string;
  schema: z.ZodSchema<TInput>;
  execute: (input: TInput) => Promise<TOutput>;
}

export interface CustomerRecord {
  id: string;
  email: string;
  tier: "standard" | "enterprise";
  openTickets: number;
}

// 2. Concrete Production Implementation
export const searchCustomerTool: ToolDefinition<{ email: string }, CustomerRecord> = {
  name: "search_customer",
  schema: SearchCustomerSchema,
  execute: async ({ email }) => {
    // In production, queries a real PostgreSQL database
    const response = await fetch(`https://api.internal.net/customers?email=${encodeURIComponent(email)}`);
    if (!response.ok) throw new Error("Customer database unreachable");
    return response.json();
  },
};

// 3. Mock Test Harness with Recorded Fixtures
export class MockToolRegistry {
  private handlers = new Map<string, (input: any) => Promise<any>>();

  registerMock<TInput, TOutput>(
    toolName: string,
    mockHandler: (input: TInput) => Promise<TOutput>
  ) {
    this.handlers.set(toolName, mockHandler);
  }

  async execute(toolName: string, rawInput: unknown): Promise<any> {
    const handler = this.handlers.get(toolName);
    if (!handler) {
      throw new Error(`No mock handler registered for tool: ${toolName}`);
    }
    return handler(rawInput);
  }
}

Testing Failure Boundaries with Replay Fixtures

With the mock registry in place, you can now author deterministic unit tests verifying how your agent's control loop recovers from API failures without ever making a network call:

import { describe, it, expect } from "vitest";

describe("Agent Tool Execution Harness", () => {
  it("gracefully handles tool timeouts without crashing the agent state", async () => {
    const registry = new MockToolRegistry();

    // Simulate an external API gateway timeout (HTTP 504)
    registry.registerMock("search_customer", async () => {
      throw new Error("Gateway Timeout: Upstream service did not respond in 5000ms");
    });

    // Execute through the agent's safe dispatcher
    const result = await safeDispatchTool(registry, "search_customer", {
      email: "client@enterprise.com",
    });

    // Assert that the agent wrapped the failure in a clean error envelope
    expect(result.status).toBe("TOOL_ERROR");
    expect(result.retryable).toBe(true);
    expect(result.message).toContain("Gateway Timeout");
  });

  it("rejects invalid input arguments before invoking the tool body", async () => {
    const registry = new MockToolRegistry();

    const dispatchAttempt = safeDispatchTool(registry, "search_customer", {
      email: "invalid-email-address", // Invalid schema
    });

    await expect(dispatchAttempt).rejects.toThrow("Invalid email format");
  });
});

async function safeDispatchTool(
  registry: MockToolRegistry,
  name: string,
  input: unknown
) {
  try {
    SearchCustomerSchema.parse(input);
    const data = await registry.execute(name, input);
    return { status: "SUCCESS", data, retryable: false };
  } catch (err: any) {
    if (err instanceof z.ZodError) throw new Error("Invalid email format");
    return { status: "TOOL_ERROR", message: err.message, retryable: true };
  }
}

Trajectory Evaluation: Testing the Path, Not Just the Output

In autonomous systems, the final output represents only a fraction of system quality. Consider a customer support agent given the prompt: "Check if order #8129 has shipped and send a tracking link to the customer."

Suppose Agent A and Agent B both return the identical string: "Tracking link sent to customer."

Agent B achieved the correct end outcome, but its trajectory reveals severe architectural regressions: redundant queries, hallucinated tool invocations, 4x latency, and 5x compute costs.

Implementing Trajectory Assertions

When testing complex workflows, record the ordered sequence of tool invocations into an execution log and assert on the trajectory graph:

export interface TrajectoryStep {
  tool: string;
  args: Record<string, unknown>;
}

export function assertValidTrajectory(
  actualTrajectory: TrajectoryStep[],
  expectedToolSequence: string[]
) {
  const actualSequence = actualTrajectory.map((step) => step.tool);

  // 1. Verify no banned or dangerous tools were called
  expect(actualSequence).not.toContain("delete_database_record");

  // 2. Verify all necessary tools executed in correct logical order
  expect(actualSequence).toEqual(expectedToolSequence);

  // 3. Verify maximum step ceiling to detect loop degradation
  expect(actualTrajectory.length).toBeLessThanOrEqual(expectedToolSequence.length + 1);
}

By enforcing trajectory bounds, your integration test suite flags inefficient reasoning paths before they lead to runaway production costs.

Managing Non-Determinism with Statistical Pass^k Metrics

Because foundation models generate responses probabilistically, a test that passes once might fail on the second attempt. Running a test single-shot and celebrating a green checkmark gives a false sense of security.

In modern agent engineering, reliability is measured using Pass^k ($Pass@k$).

$$ ext{Pass}^k = rac{ ext{Successful Trials}}{k}$$

If you run an agent against a complex benchmark scenario $k = 10$ times:

• If it completes the goal successfully 10 out of 10 times, its reliability score is 1.0 (100%).
• If it completes the goal 7 out of 10 times, its reliability score is 0.7 (70%).

export async function evaluatePassK(
  taskFn: () => Promise<boolean>,
  k: number = 5,
  minAcceptablePassRate: number = 0.8
): Promise<{ passRate: number; passed: boolean }> {
  let successes = 0;

  for (let i = 0; i < k; i++) {
    try {
      const isSuccess = await taskFn();
      if (isSuccess) successes++;
    } catch {
      // Failed trial
    }
  }

  const passRate = successes / k;
  return {
    passRate,
    passed: passRate >= minAcceptablePassRate,
  };
}

Setting Production Thresholds

• Critical Business Actions (e.g., automated refunds, database mutations): Require a minimum $Pass^5 = 1.0$ in pre-production staging.
• General Conversational Workflows (e.g., FAQ summarization, doc search): An $Pass^5 ge 0.8$ is typically acceptable for deployment when coupled with human escalation fallbacks.

If a code change drops your checkout agent's $Pass^5$ score from 0.95 to 0.70, your CI pipeline blocks the deployment automatically.

Deterministic Schema Assertions vs. LLM-as-a-Judge

When should you use deterministic code assertions, and when does it make sense to use an independent LLM to evaluate your agent?

Rule of Thumb: Exhaust deterministic code assertions first. Only invoke an evaluator LLM when validating subjective linguistic criteria that cannot be expressed as a schema or boolean condition.

Continuous Integration Architecture for Agentic Systems

To maintain high development velocity without inflating cloud expenditure, structure your CI/CD deployment pipeline into two separate validation tracks:

Track 1: Fast PR Gate (Every Git Push)
[Commit Code] ──> [Lint & Typecheck] ──> [Tier 1 Tool Unit Tests] ──> [Tier 2 Contract Mocks] ──> [Merge PR]
Duration: ~90 seconds | Cloud Cost: $0.00

Track 2: Deep Regression Suite (Nightly Scheduled Job)
[Nightly Trigger] ──> [Sandbox Spin-up] ──> [Tier 3 Trajectory Eval] ──> [Pass^k Benchmarks] ──> [Metric Report]
Duration: ~15 minutes | Cloud Cost: Controlled Token Budget

1. Pull Request Gate: Executes Tier 1 and Tier 2 suites exclusively. Every tool is mocked with deterministic fixtures. No external LLM APIs are invoked. This guarantees that your team gets instant, zero-cost feedback on code regressions, broken schemas, and state logic.
2. Nightly Evaluation Benchmarks: Runs against actual model endpoints inside an isolated sandbox environment. It executes 50 to 100 golden benchmark scenarios across $k=5$ iterations to measure trajectory drift, latency distribution, and $Pass^k$ stability.

For teams building stateful, long-running agent workflows, integrating this testing framework alongside a durable state management architecture ensures that agentic software remains predictable, cost-effective, and safe for production scale.

Engineering Checklist for AI Agent Test Suites

Before deploying updates to an agentic system, ensure your test pipeline satisfies these standards:

• Contract Mocks in CI: Are all external tool dependencies mocked with realistic, replayable golden fixtures in PR builds?
• Error Path Coverage: Does your test suite explicitly simulate API timeouts (504), rate limits (429), and malformed tool responses?
• Trajectory Assertions: Are you verifying the sequence of tool invocations rather than relying solely on final text assertions?
• Statistical $Pass^k$ Thresholds: Do high-stakes agent workflows undergo multi-trial statistical evaluation before promotion to production?
• Zero Live-API Dependency in PRs: Is your primary CI test suite fully isolated from external commercial model API calls?
• Step Ceilings: Does your test harness enforce a strict cap on maximum execution steps to catch recursive loops?