Benchmarking AI Agents: How to Measure Latency, Token Cost, and Pass Rates with SWE-bench & Custom Evals

The Death of 'Vibes-Based' Testing in Production AI

In the early days of generative AI, developers evaluated prompts through "vibes": running three test queries in a playground, seeing plausible answers, and pushing straight to production.

In enterprise software engineering, this approach is disastrous:

• A prompt tweak that improves edge case A might degrade edge case B by 35%.
• Upgrading from one model checkpoint to another can trigger subtle JSON formatting regressions.
• Tool-calling agents can suddenly enter infinite loops, burning hundreds of dollars in API tokens.

To build software people trust and pay for, you must treat AI evaluation as automated unit testing.

The 3 Pillars of AI Evals: Accuracy, Cost, and Latency

Every evaluation run should yield a three-dimensional scorecard:

codeCopy
                     ▲ Task Accuracy (%)
                     │
                     │      ★ Golden Target (> 95% pass, < $0.05, < 5s)
                     │
                     │
─────────────────────┼─────────────────────► Cost per Task ($)
                    /
                   /
                  ▼ Latency P95 (seconds)

1. Deterministic Accuracy: The percentage of test fixtures where the system achieved its objective according to verifiable unit test assertions (e.g. tests passed, exact JSON match, valid SQL executed).
2. Mean Token Spend: The dollar cost of model tokens consumed per task. A system with 98% accuracy that costs $4.00 per query is commercially unviable for high-volume workflows.
3. P95 Latency: The 95th percentile wall-clock time from initial prompt to final verified output.

Constructing Golden Test Datasets: Synthetic vs Real Traces

A Golden Dataset consists of 50 to 200 carefully curated test fixtures representing realistic production scenarios:

jsonCopy
// evals/fixtures/repo_bug_042.json
{
  "test_id": "fixture_repo_bug_042",
  "input": {
    "issue_description": "TypeError: Cannot read properties of undefined (reading 'split') in parseHeaders",
    "git_commit": "e81f72a"
  },
  "expected_outcome": {
    "modified_files": ["src/http/headers.ts"],
    "unit_test_command": "npm test test/headers.spec.ts",
    "expected_test_status": "pass"
  },
  "budget_constraints": {
    "max_cost_usd": 0.08,
    "max_iterations": 6,
    "timeout_ms": 30000
  }
}

Sources for Golden Test Cases:

• Captured Production Incidents: Every time an agent fails in the wild, turn the sanitised trace into an evaluation fixture.
• Edge Case Synthesis: Use an independent frontier model to generate 20 adversarial inputs (e.g. malformed CSVs, ambiguous instructions, non-standard date formats).
• Golden Reference Annotations: Have domain experts verify expected outputs once to create permanent ground truth.

Running Evals Against SWE-bench & Custom Assertions

For coding agents, SWE-bench is the gold standard benchmark. It evaluates whether an agent can read a real GitHub issue description, locate the correct file, write a patch, and pass the existing repo unit tests.

Here is a lightweight Python runner for executing automated assertions against agent outputs:

pythonCopy
# test_eval_runner.py
import pytest
import asyncio
from dataclasses import dataclass
from my_agent import AutonomousBugAgent

@dataclass
class EvalResult:
    test_id: str
    passed: bool
    cost_usd: float
    duration_s: float

async def run_single_fixture(fixture: dict) -> EvalResult:
    agent = AutonomousBugAgent(model="anthropic/claude-3-7-sonnet")
    start_time = asyncio.get_event_loop().time()
    
    output = await agent.solve_issue(fixture["input"])
    duration = asyncio.get_event_loop().time() - start_time
    
    # Assertive verification
    passed = (
        output.test_suite_status == "passed" and
        output.total_cost <= fixture["budget_constraints"]["max_cost_usd"]
    )
    
    return EvalResult(
        test_id=fixture["test_id"],
        passed=passed,
        cost_usd=output.total_cost,
        duration_s=duration
    )

Integrating Agent Evals into GitHub Actions & CI/CD

Treat agent prompts and tool definitions like application code. Run evals on every pull request:

yamlCopy
# .github/workflows/agent-evals.yml
name: Agent Evals & Regression Suite

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/agent/**'
      - 'system.yaml'

jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - name: Execute 50-Fixture Golden Eval Suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          npm run evals -- --output benchmark-report.json
      - name: Enforce Regression Thresholds
        run: |
          node -e "
            const report = require('./benchmark-report.json');
            if (report.accuracy < 0.85) throw new Error('Regression: Accuracy below 85%');
            if (report.avgCostUsd > 0.10) throw new Error('Cost spike: Average cost exceeded $0.10');
            console.log('Eval passed! Accuracy: ' + report.accuracy * 100 + '%');
          "

Generating Verified Benchmark Cards on AIStoreHub

When publishing your system on AIStoreHub, include your benchmark results in system.yaml:

yamlCopy
benchmarks:
  dataset: "custom-finance-eval-v1"
  total_evals: 120
  success_rate: 0.942
  avg_latency_ms: 4200
  avg_token_cost_usd: 0.038

AIStoreHub's verification scanner validates these claims against reproducible runner environments, awarding your listing the Verified Benchmark Badge to maximize buyer trust and marketplace sales.