How do you test an AI model on data it has never seen?

An AI model can answer an impressive question in a demonstration and still fail a similar question in a different setting. To understand the difference, we need more than a single score: we need to know what was tested, whether the model could have encountered the answers before, and whether the test resembles the work we care about.

A benchmark is a defined set of tasks and a way to score results. It is useful because different models can be compared on a common basis. It is also a sample of work, not a complete description of intelligence or a guarantee of trustworthy behavior.

Why keep test data separate?

Imagine teaching a student with a set of practice questions, then grading them on those exact questions. A high mark might show understanding, memorization, or a mixture of both. A separate test helps distinguish those possibilities, although it still cannot measure every skill the student will need later.

AI evaluation has a similar concern called data contamination: information from a test can enter training or development, making a score less informative about performance on genuinely unseen material. Keeping evaluation data inaccessible to model developers can reduce this risk. It does not prove that every related example or piece of knowledge is absent from training.

In July 2026, the US National Institute of Standards and Technology announced its Artificial Intelligence Technology Evaluation, or AITE. Its proposed test environment uses blind data and common scoring. The initial tasks involve image analysis with vision-language models in quantum science, genomics and public safety. This is a description of the announced program, not a result for a particular model.

A score needs a question

Before comparing numbers, ask what the model was asked to do. Identifying an object in an image, describing evidence in a scientific figure and answering a safety question are different tasks. A model may be useful on one and unreliable on another.

1. Define the task. Decide what the model receives, what a correct answer means and which errors matter.
2. Separate the data. Keep the evaluation set apart from training and tuning where possible, and document what is known about its origin.
3. Measure consistently. Apply the same rules to competing systems and inspect failures, not only the average.
4. Check the intended setting. Test on representative inputs and use additional safeguards before relying on the system.

The steps are a simplified reading guide. An actual evaluation needs documented protocols, appropriate controls and a way to handle uncertainty.

What does an average hide?

Suppose a model answers 80 out of 100 test questions correctly. The result describes that set and the chosen scoring rule. It does not mean that the model will be correct on exactly 80% of every future task, or that the remaining errors will all be harmless.

The average may also hide differences between groups of examples. A system could do well on clear images and poorly on unusual lighting, or perform differently across question types. The size and composition of the test set, repeated runs and scoring choices can change how much confidence we place in a result.

NIST’s statistical evaluation work distinguishes performance on a fixed benchmark from estimates intended to generalize to similar future items. That distinction is useful even when a reader never calculates a confidence interval: a score is always tied to a measurement design.

Can one benchmark prove an AI is safe?

No. A hidden test can make a comparison more informative, but it cannot include every situation, user or failure mode. Real deployments can encounter changed data, unexpected combinations of inputs and consequences that a benchmark was not designed to measure.

NIST’s international evaluation discussion identifies practices as well as open questions. For a practical decision, ask for the task, test conditions, known failures and limits, then consider what human oversight or additional testing the actual use requires.

The useful question is not just “Which model scored highest?” It is “Did the test measure the job I need done, and what would happen when the model gets it wrong?”