PristrenPromptfoo runs your prompts against multiple models, checks outputs with assertion functions, and red-teams for jailbreaks and PII leakage - all from a YAML config.
Mahmudul Haque Qudrati
CEO & ML Engineer
// tags
Promptfoo is a CLI tool for testing LLM prompts systematically. You write test cases in YAML, define assertions, and Promptfoo runs every prompt against every model in your config - giving you a comparison matrix and a pass/fail report. It also ships a red-teaming engine that automatically probes for safety vulnerabilities.
npm install -g promptfoo
# or
npx promptfoo@latest
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
# promptfooconfig.yaml
providers:
- openai:gpt-4o-mini
- openai:gpt-4o
- ollama:llama3.1:8b
prompts:
- "Summarize the following text in one sentence: {{text}}"
tests:
- vars:
text: "PagedAttention is a memory management technique for LLM inference that treats the KV cache like virtual memory pages."
assert:
- type: contains
value: "KV cache"
- type: javascript
value: "output.length < 200"
- type: llm-rubric
value: "The summary is factually accurate and concise"
Run with:
promptfoo eval
promptfoo view # Open browser UI with comparison table
Add --ci flag to output JSON results consumable by GitHub Actions:
promptfoo eval --ci --output results.json
Parse results.json in a workflow step to comment score diffs on the PR - your reviewers see exactly which model and which test case regressed.
For complex validation logic, write a JS function:
// assertions/check-structured.js
module.exports = (output) => {
try {
const parsed = JSON.parse(output);
return {
pass: parsed.name && parsed.age > 0,
score: 1,
reason: "Valid structured output",
};
} catch {
return { pass: false, score: 0, reason: "Not valid JSON" };
}
};
Reference in YAML:
assert:
- type: javascript
value: file://assertions/check-structured.js
promptfoo redteam init # Generates redteam config from your system prompt
promptfoo redteam run # Runs attack probes
promptfoo redteam report # View results
Red-team plugins include:
name: Prompt Quality Check
on: [pull_request]
jobs:
promptfoo:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npx promptfoo@latest eval --ci
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Full documentation at promptfoo.dev.
Chat, meetings, project management, and time tracking — all wired together with AI. Stop context-switching between five tools.
Try Zlyqor freePractical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
SWE-Bench tests LLMs on 2,294 real GitHub issues from popular Python repositories, evaluating whether the model can write code that passes the existing test suite - a far harder and more realistic evaluation than HumanEval.
