AI evaluation should test the task, not the demo

A compelling demonstration proves that a system can succeed. An evaluation asks how often it succeeds, where it fails, and whether those failures are acceptable in the intended workflow.

Define the task boundary

Specify inputs, expected outputs, users, tools, data access and downstream actions. A model score without operational context rarely answers the business question.

Build representative cases

Include normal work, difficult edge cases, ambiguous requests and known failure conditions. Preserve a held-out set where practical.

Measure more than accuracy

Latency, cost, refusal behavior, consistency, traceability and human-review burden can matter as much as correctness.

Set thresholds before deployment

Decide which failures require review, which require blocking, and which can be tolerated. Evaluation becomes governance when results map to operating decisions.