Open evals: how to know your images are actually getting better

You swap in a faster model, lower the step count, or turn on fp8. The first few images look fine. Two weeks later a customer asks why every product shot now has a slightly melted label. Nobody noticed, because nobody was measuring.

Generative media is hard to test. There is no single right answer to "a ceramic cup on a linen tablecloth". But "hard to test" is not "impossible to test". With a small, open eval set and a few honest metrics you can tell, with confidence, whether a change made things better or worse.

This guide covers how to build that eval set, which metrics are worth trusting, and how to keep an eye on quality once you are live.

What makes an eval "open"

An open eval is one anyone on your team can read, rerun, and argue with. The prompts are written down. The settings are written down. The scoring rules are written down. If a number moves, you can open the exact images behind it.

• The prompt set lives in version control, next to your code.
• Every run records the model, version, seed, steps, resolution, and GPU type.
• Every score links back to the image or clip it came from.
• The scoring method is public, so two people get the same result.

Build a small eval set you actually care about

Public benchmarks are useful for comparing models in general. They tell you very little about your product. The most valuable eval set is the one built from your own traffic.

1. Start with 100 to 300 prompts. That is enough to see real changes and small enough to review by eye.
2. Pull them from real requests, with anything personal removed. Real prompts are messier than the ones you would write yourself, and that is the point.
3. Group them by what they test: text in the image, hands and faces, product accuracy, counting, color, composition, motion for video.
4. Add the prompts that have broken before. Every bug report becomes a permanent test.
5. Freeze the set for each release. Add new prompts in a new version so old scores stay comparable.

Pick metrics that match the question

No single metric captures quality. Use a few, each answering one clear question, and never let one number stand in for all of them.

Automated scores are a filter, not a judge

Preference models and CLIP-based scores are fast and cheap. They are also easy to fool, and they share blind spots with the models they grade. Use them to flag which images to look at, then have people make the final call on anything that ships.

Make comparisons fair

Most "quality regressions" are really broken experiments. Two runs that differ in more than one way tell you nothing.

• Fix the seeds. Run every prompt with the same 2 to 4 seeds on both sides, so you compare the change, not the luck.
• Change one thing at a time: the model, the steps, the precision, or the GPU. Not all four.
• Run blind reviews. Show A and B side by side in random order, without labels.
• Report ties. "No visible difference" is a real and useful answer, especially when the change made things faster or cheaper.
• Look at the worst cases, not just the average. An average can improve while your ten most important prompts get worse.

If you only remember one rule: same prompts, same seeds, one change, blind review.

Measure speed and cost next to quality

Most optimizations trade a little quality for a lot of speed. That can be a great trade, but only if you see both sides. Record time per image, time to first frame for video, and cost per output for every eval run, in the same table as the quality scores.

Keep watching after launch

An eval set catches problems before release. Monitoring catches the ones that only show up with real traffic: new kinds of prompts, a weight file that changed upstream, a driver update, a GPU type you have not tested.

• Sample a small share of live outputs every day, for example 1 percent, and score them with the same automated metrics as your eval set.
• Watch trends, not single scores. Alert when the daily average moves outside its normal range for two days in a row.
• Track failure signals users give you for free: retries, regenerations, downloads, and deletions.
• Rerun the full eval set on a schedule, even when nothing changed. Silent changes are the ones that hurt.
• Keep a short human review every week. Twenty minutes looking at random outputs finds things no metric was built to find.

On ClockingCloud, sessions make this easier to organize: run each eval pass in its own session, tag outputs by prompt group and model version, and pull the full set back later to compare runs side by side.

Common mistakes to avoid

• Tuning against the eval set until it stops meaning anything. Keep a held-out set you only check before a release.
• Trusting FID on small samples. It needs thousands of images to be stable.
• Grading only by automated scores. They reward what they were trained to like, which is not always what your customers like.
• Changing the eval set and the model in the same release, then comparing the scores.
• Forgetting video is more than frames. A clip can have perfect frames and still flicker.

A starter checklist

1. Write 100 to 300 prompts from real traffic, grouped by what they test.
2. Write down what "good" means for each group.
3. Choose two or three automated metrics and one blind human review.
4. Fix seeds and settings, and record them with every run.
5. Record speed and cost next to quality.
6. Set a quality floor before you optimize.
7. Sample live outputs daily and alert on trends.
8. Turn every bug report into a new eval prompt.

None of this needs a research team. A spreadsheet, a fixed prompt list, and the discipline to change one thing at a time will catch most problems before your customers do.