Testing LLMs Like Software

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

When you ship a software change, tests catch regressions. When you change a prompt or swap a model, there is typically nothing to catch quality regressions. DeepEval fixes this by turning LLM quality checks into pytest tests that run in CI exactly like unit tests.

Installation

bash

pip install deepeval
deepeval login  # Optional: connect to Confident AI dashboard

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Writing Your First LLM Test

python

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, HallucinationMetric

def test_rag_answer_quality():
    test_case = LLMTestCase(
        input="What causes hallucinations in LLMs?",
        actual_output="LLMs hallucinate because they predict likely tokens without a fact-checking mechanism.",
        retrieval_context=[
            "Hallucinations in LLMs occur due to the probabilistic nature of token prediction...",
            "Unlike databases, LLMs have no retrieval fallback when they lack information...",
        ],
    )

    relevancy_metric = AnswerRelevancyMetric(threshold=0.8, model="gpt-4o-mini")
    hallucination_metric = HallucinationMetric(threshold=0.5, model="gpt-4o-mini")

    assert_test(test_case, [relevancy_metric, hallucination_metric])

Run with:

bash

deepeval test run test_llm.py

This integrates seamlessly with pytest - you get pass/fail output and the full pytest ecosystem (fixtures, parametrize, markers).

G-Eval: Custom LLM-as-Judge Metric

G-Eval lets you define a custom evaluation criterion in natural language:

python

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

conciseness = GEval(
    name="Conciseness",
    criteria="The response answers the question without unnecessary padding or repetition.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

G-Eval uses chain-of-thought prompting to produce a calibrated 0 - 1 score for any custom criterion.

Built-In Safety Metrics

python

from deepeval.metrics import BiasMetric, ToxicityMetric

bias = BiasMetric(threshold=0.5)
toxicity = ToxicityMetric(threshold=0.5)

These detect race/gender/political bias and toxic language in model outputs - essential for customer-facing deployments.

Red-Teaming Attacks

bash

deepeval red-team --model gpt-4o-mini --attacks prompt-injection,jailbreak,pii-leakage

DeepEval generates adversarial inputs, runs them against your model, and reports which attacks succeeded. Use this before launching a new model version.

GitHub Actions CI Example

bash

name: LLM Quality Gate
on: [push, pull_request]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install deepeval
      - run: deepeval test run tests/llm/
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

If any metric drops below its threshold, the CI step fails and the PR is blocked - the same discipline that prevents software regressions applied to LLM quality.

Confident AI Dashboard

Sync results to Confident AI cloud for historical metric trending, regression alerts, and team-level dashboards without running your own infrastructure.

#deepeval#llm-testing#pytest#hallucination#ci/cd

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.