AI Through Glass

Learn / Lesson 3 · Intermediate

How do we know a model got better?

A new model gets more answers right. Then a librarian spots a reply that gives a borrower the wrong deadline. Would you put the new model on the help desk?

Start with the job

A benchmark is a standardised test. To understand an improvement claim, ask what the model was asked to do, how the answers were marked, and which cases made it into the score. An evaluation for your own service can answer a more specific question.

For this library, the job is to answer correctly from published policy and refer the borrower to staff when the policy leaves their deadline uncertain. Before seeing either model’s replies, the library sets its rule: an invented deadline blocks an unsupervised trial.

Illustrative worked example

Two models answer the same library questions

The library sets aside 20 questions it has never used to tune the models. Both receive the same instructions, policy excerpts and time limit. Librarians mark the answers against the same rubric with model identities hidden.

Fictional help-desk trial · One response per question and model
ObservationExisting modelNew model
Routine questions answered correctly9 of 1212 of 12
Policy exceptions handled correctly7 of 86 of 8
All questions handled correctly16 of 2018 of 20
Invented deadlines on exception cases0 of 82 of 8
Replies delivered within 3 seconds19 of 2014 of 20
Model usage cost for the 20 repliesS$0.20S$0.34
All figures are illustrative. The deadline errors are among the incorrectly handled exceptions. Timing and cost are separate measures. Neither model stands for a real product.

Read the denominator. Twelve routine questions and eight exceptions make up each model’s 20 replies. The new model handled every routine question correctly. Both of its missed exceptions contained an invented deadline. The existing model’s missed exception involved an unnecessary referral to staff.

Imagine the desk mostly receives questions about exceptions. That changes how useful the combined score is. It also matters whether a slower reply misses the service target, and whether staff time spent checking replies costs more than the model usage shown here.

Which decision follows from the trial?

The library’s rule still applies. Choose the action you could defend to a borrower who relied on a deadline.

Read the worked answer

Keep staff review on deadline answers. The new model broke a rule the library set before the test. Its routine answers may justify a separate supervised trial. Eight exception cases also give the library too little evidence for a guarantee about future answers from the existing model.

What would make the next comparison fair?

  1. Choose cases from the work. Include ordinary requests, rare costly mistakes and requests the service should hand to a person. Keep fresh cases out of prompt tuning.
  2. Record the conditions. Name the model version, prompts, documents, tools, retries and time budget. If the new setup uses a better search tool, you can report that the whole setup improved. Attributing the gain to the model alone would need another comparison.
  3. Read the mistakes. Agree in advance what counts as correct, what an abstention means and which failure would stop deployment. Score the same cases, inspect disagreements, and repeat variable cases before treating a small difference as durable.
  4. Measure the experience. Compare total cost per completed task and latency under comparable load. A model that needs extra retries or human review can change both.

Independent practice

A different decision at the museum

A museum is choosing a model to draft public captions from catalogue records. A vendor reports a higher score for the new model on a general writing test. Before opening the explanation, write the test you would run and the result that would change your decision.

Notes stay in this page while it is open.

Compare your plan with one possible answer

Use unseen catalogue records, including objects whose maker is unknown. Give both models the same fields and caption limit. Have a curator check factual details and whether uncertainty survives the rewrite. An invented attribution would stop automatic publication even if readers preferred the prose. A supervised trial could be justified by better captions, acceptable review time and fewer corrections on a fresh sample.