Eval-Framework Template

AI Sales Engineering · v1.0 · Last verified 31 August 2026

An evaluation framework for a buyer is a test set built from their own data, metrics defined by layer, acceptance bands agreed before any results exist, and a results pack that includes the failures. This is the template for all four.

A buyer-facing evaluation design. It replaces "trust the demo" with structured measurement on the buyer's own data.

Every eval tool in existence is built for engineers. This is the version you hand a buyer. Use it with the POC Scorecard: the scorecard agrees what success means, this measures whether you hit it.

0. What move does this document make?

When a buyer says "how do we know it works", most sellers offer another demo. A demo is a sample of one, chosen by you. Offering to measure instead, on their data, against criteria they set, with failures included, changes the frame from persuasion to evidence, and it is very hard for a competitor to follow you there.

Say this out loud in the room: "Let's measure it together rather than me showing you again." Then hand over this document.

1. What we are measuring and why

Plain English, for the executive who will read only this section.

We are testing whether [PRODUCT] can [SPECIFIC TASK] on [BUYER]'s own [data type], well enough to [BUSINESS OUTCOME], and we are testing where it fails so you know the boundary before you deploy it rather than after.

This is not a benchmark. Published benchmark scores tell you how a model performs on someone else's problem. The only number that matters is performance on yours.

2. How do you build the test set?

From the buyer's own data, sized to the difference you care about, split along the axis your product degrades on, held out before any tuning, with a hand-labelled golden set inside it. This is the most important section and the one most evaluations get wrong.

Source. The buyer's own data. Transcripts, tickets, documents, queries, repositories, historical cases. Never your sample data, never a public dataset. If they cannot give you data, that is a finding worth surfacing early, because it usually means the deployment will hit the same wall.

Size. Enough for the difference you care about to be visible. As a rule of thumb, 100 cases will show you a gross failure, 300 to 500 will let you compare two configurations with any confidence, and below 50 you are reading noise. State the number and its limits rather than implying more precision than you have.

Cohort splits, the part that earns trust. Split along the axis your product actually degrades on, and name that axis out loud before you run anything:

Table: The axis each product type degrades along, for cohort splitting. Source: AI Sales Engineering practitioner template, v1.0, August 2026.

Every AI product degrades on some axis. Name yours before the buyer finds it. Volunteering it is the single highest-trust move available in an evaluation, and it costs nothing because they will find it anyway.

Holdout rules. The test set is built before any tuning and is not looked at during it. If you tune against the test set you have measured nothing. Say this in the room. Buyers with a data science function will check, and buyers without one will be reassured that you offered.

Golden set. Carve out 30 to 50 cases the buyer's own experts have labelled by hand, and treat that as the reference. Disagreement between your automated scoring and their golden set is itself a finding, and usually a more interesting one than the headline score.

3. What should you measure?

Nine metrics across six layers: input fidelity, retrieval, understanding, reasoning, safety, operations and cost. Layered by function, so the same framework serves any product. Delete the layers that do not apply.

Table: Evaluation metrics by layer, with definitions. Source: AI Sales Engineering practitioner template, v1.0, August 2026.

Two metrics people leave out and should not. Abstention correctness, because a confident wrong answer costs more than a refusal and most evaluations only measure the cases where the system answered. And cost per completed task, because it is the number that decides the expansion conversation six months later.

4. What counts as passing?

A target with a tolerance, a failure budget, agreed severity classes and a stated confidence interval. Not a single percentage. This is the section that separates a 2026 evaluation from a 2024 one.

A probabilistic system does not pass or fail. It performs within a band, and the buyer needs to agree the band before results exist.

• Target with a tolerance. "At or above 92%, and we treat 90 to 92% as a pass with a documented remediation plan" is a real acceptance criterion. "92%" alone is a coin toss that both sides will argue about afterwards.
• A failure budget. How many failures of each severity class are acceptable in the period. Not all errors cost the same, and treating them as if they do is how evaluations produce a number nobody trusts.
• Severity classes, agreed up front. Typically cosmetic, recoverable, costly, unacceptable. One unacceptable failure should fail the evaluation regardless of the aggregate score, and saying so first makes you the serious vendor in the room.
• Confidence, stated honestly. With 200 cases, a measured 92% carries roughly a four point interval either side. Quote the interval. A vendor who hands a bare percentage to a buyer with a data science team has told them something about their rigour that they did not intend.

5. What does the system do below the confidence threshold?

Decline, escalate, ask a clarifying question, or answer with a caveat. Document which, then show the curve. This section wins deals and is usually missing entirely.

Document what the system does below threshold. Then show the threshold curve, how accuracy and coverage trade off as you move it.

Letting the buyer choose their own operating point on that curve is the most persuasive thing in the entire evaluation. It converts an argument about whether the product is good enough into a joint decision about where to set a dial.

6. What do you compare it against?

What they do today. Measure the same test set against the human team, the incumbent tool, the existing search, the incumbent vendor, or someone doing it by hand.

Almost nobody has measured their own baseline. Frequently it is worse than everyone assumes, and establishing it is genuine value you deliver before they have paid you anything. It also reframes every subsequent conversation, because the question stops being "is it accurate enough" and becomes "is it better than what we do now, and by how much".

7. How do you present the results?

In six parts, and one of them is not optional. Always include the failure table. Never omit it.

Present in this order: what we measured and on what data, the headline numbers with confidence intervals, per-cohort breakdown, the failures with real examples, what we would change, and what we recommend the operating threshold should be.

Showing real failures is what makes the passing numbers believable. An evaluation that reports only successes reads as marketing, and a buyer's technical reviewer will discount the whole document. Disclosed weakness closes deals that flawless demos do not.

8. How often do you re-test after deployment?

On every model version change, on a fixed quarterly cadence, and on any material change to their data distribution. An evaluation is not a one-off. Agree it in writing, and name who owns the re-run and where the results go.

This is also the natural bridge into the Model-Change brief. The eval suite you build here is the instrument that makes a version-change commitment enforceable.

Before you use this

Delete the layers that do not apply, replace every bracketed field, and agree the acceptance bands in section 4 before you generate a single number. An evaluation whose thresholds are set after the results exist convinces nobody, least of all the person who noticed.

Sources, and what is not verified here

Last verified: 31 August 2026.

What this is: a practitioner template written by Lewis Crook from enterprise AI evaluations, not a research paper.

Not verified here: the sample-size rules of thumb in section 2 and the confidence interval example in section 4 are illustrative rather than derived from a stated power calculation. If your buyer has a data science function, do the arithmetic properly with them.

Corrections welcome, and they get credited.