AI Evaluation That Measures What Public Benchmarks Cannot

Public Benchmarks Tell You Almost Nothing About Your Deployment

Frontier models are clustered above 88% on MMLU. GPT-5.3 Codex scores 99%. Vellum's 2025 LLM Leaderboard dropped MMLU entirely because it no longer differentiates models in any meaningful way. MMLU-Pro, designed to fix this, is already approaching 90% for frontier models. The industry's most-cited benchmarks have become vanity metrics.

The deeper problem is relevance. A benchmark score predicts production performance only when it tests tasks similar to yours, the test set is free of data contamination (some benchmarks show leakage ratios as high as 100%), and score differences are statistically significant. For most enterprise deployments, none of those conditions hold. We have evaluated models where the #3 on public leaderboards outperformed the #1 by 40% on the client's actual extraction task. This is why we build custom evaluation harnesses: "which model performs best on my data, my edge cases, at my cost constraints?" requires evaluation infrastructure designed for your deployment.

What a Rigorous Evaluation Program Actually Looks Like

We structure evaluation around three layers, each answering a different question about your AI system.

Capability evaluation answers: does this system do what we need it to do? We build task-specific test suites from your production data and realistic edge cases, not convenience samples. For an underwriting model, that means testing on actual declined applications, borderline cases, and the specific document formats your pipeline encounters. Every test case is documented with collection methodology and annotation quality metrics. We measure with statistical rigor: multiple runs with different seeds, bootstrap confidence intervals, paired significance tests. A 2% improvement that falls within the confidence interval is not an improvement.

Safety evaluation answers: where does this system fail, and how badly? We test behavioral boundaries using structured probes: minimum functionality tests, invariance tests (does the output change when it should not?), and directional expectation tests. Disaggregated evaluation reports performance across every operationally relevant data slice, because a model that works on average but fails on a critical subpopulation is not safe to deploy.

Adversarial evaluation answers: can someone make this system do something it should not? This is where red teaming lives.

Red Teaming as Structured Capability Assessment

Red teaming is not penetration testing with a different name. Security assessment asks "can an attacker compromise this system?" Red teaming in the evaluation context asks "what are the boundaries of this system's behavior, and where do those boundaries break?" The methodologies overlap, but the questions are different, the reporting is different, and the audience is different.

We operate under a structured methodology built on the NIST AI 100-2 E2025 taxonomy, which expanded significantly in March 2025 to cover autonomous AI agent vulnerabilities and GenAI-specific attack categories. Our red team programs follow a defined sequence: threat model definition scoped to your deployment context, attack taxonomy enumeration covering the OWASP LLM Top 10 v2 categories (prompt injection, jailbreaking, data poisoning, indirect injection through retrieved content, multimodal attacks, encoding-based evasion), systematic attack execution with documented procedures, and severity-rated findings with reproduction steps.

We complement human red teaming with automated adversarial pipelines. Haize Labs' Cascade achieves 44% attack success rates on frontier models, 4x higher than single-turn baselines. Promptfoo runs 50+ vulnerability types in CI/CD across 300,000+ developer installations. Automated tools catch known patterns at scale. Human red teamers find novel vulnerabilities that automated systems have never seen, which matters most in domains where the consequences of a missed failure mode are severe.

Every red team engagement produces three deliverables: a findings report with severity ratings and reproduction procedures, remediation recommendations mapped to your architecture, and an automated regression test suite derived from discovered vulnerabilities that integrates into your deployment pipeline so discovered weaknesses stay fixed.

When Automated Evaluation Works and When It Does Not

LLM-as-judge evaluation, using a frontier model to score another model's outputs, has become the default for teams that cannot afford human evaluation at scale. It is useful. It is also unreliable in specific, documented ways. Research has identified 12+ distinct bias types in LLM judges. GPT-4 exhibits significant self-preference bias, rating outputs with lower perplexity higher regardless of whether it generated them. LLM judges consistently prefer verbose, formal responses over concise correct ones. Position bias causes judges to favor whichever response appears first.

These biases are manageable for general quality comparison with debiasing techniques (randomized position, multi-judge panels). They are disqualifying when domain-specific correctness matters. An LLM judge cannot reliably assess whether a clinical system correctly identifies drug interactions or whether legal research accurately cites case law. We design evaluation protocols that use automated scoring where bias is manageable and human expert review where correctness requires domain knowledge.

Evaluating Agentic AI Systems

Static model benchmarking does not work for agents that plan, use tools, and execute multi-step workflows. A single accuracy number cannot capture whether the agent selected the right tool, called it with correct parameters, recovered gracefully when a step failed, or produced a coherent result across 15 chained operations. An agent can execute every individual step correctly and still produce a wrong result because the reasoning connecting those steps was flawed.

We evaluate agentic systems across five dimensions drawn from the CLEAR framework: cost efficiency of tool and token usage, latency across full task completion, efficacy of end-to-end task success, assurance that safety constraints held throughout execution, and reliability across repeated runs. For tool-using agents, we test tool selection accuracy, parameter correctness (agents fabricate parameter names at meaningful rates), scope adherence, and error recovery. For multi-agent systems, we test inter-agent communication fidelity, cascading failure propagation, and whether supervisor controls actually intervene when subordinate agents diverge.

The benchmarks are catching up. SWE-bench tests real software engineering tasks. Terminal-Bench evaluates command-line agent workflows. UpBench uses real Upwork job postings refreshed continuously. But off-the-shelf agentic benchmarks rarely match your specific agent architecture, tool set, and domain. We build custom agentic evaluation harnesses because your agent's failure modes are specific to its design.

Evaluation for Regulatory Compliance

The EU AI Act's high-risk provisions take full effect August 2, 2026. Article 9 requires a risk management system with documented evaluation methodology, testing under intended-use and reasonably foreseeable misuse conditions, and ongoing post-market monitoring. Conformity assessment must be completed before placing a high-risk system on the EU market. Non-compliance carries fines up to 7% of global annual turnover or EUR 35 million.

NIST AI 100-2 E2025 provides the authoritative adversarial evaluation taxonomy, now covering autonomous agent vulnerabilities absent from the 2023 edition. These frameworks are appearing in procurement requirements and board-level risk reviews. The practical challenge: no harmonised standard yet defines "adequate evaluation" for EU AI Act compliance. CEN/CENELEC JTC 21 missed its August 2025 deadline and is targeting Q4 2026. We design evaluation programs that produce defensible evidence now while remaining adaptable to standards still being finalized.

Continuous Evaluation in Production

A pre-deployment evaluation tells you the system worked on a specific date against a specific test set. It tells you nothing about next month. Production models drift. Input distributions shift. Retrieved content changes. Tool APIs update. A 2025 LLMOps report found that models left unchanged for six months saw error rates increase by 35% on new data. Gartner estimates that only 18% of software engineering teams had adopted AI evaluation and observability platforms as of 2025, though they project 60% adoption by 2028.

We build evaluation pipelines that run continuously. Production monitoring scores live traffic using the same evaluators from development. Failed evaluations become CI/CD regression tests. Drift detection alerts when input distributions diverge from baselines. Adversarial suites run nightly against production endpoints. This is operational infrastructure that keeps evaluation current as your system evolves.

The Evaluation Tooling Landscape

The market is fragmented. Stanford's HELM evaluates across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The UK AI Security Institute's Inspect offers 100+ pre-built evaluations with ControlArena for agent testing. Promptfoo (now OpenAI-owned, 300k+ developers) integrates eval and red teaming into CI/CD. Patronus AI generates adversarial test cases at scale. Each has blind spots: HELM is heavyweight for fast iteration, Inspect is oriented toward frontier model safety, Promptfoo is shallow on domain-specific methodology, Patronus does not replace human red teaming. We use each where it fits and build custom harnesses where none of them reach.