Ask an AI model to score the same thumbnail and title twice and you can get two different numbers. Ours once returned 71, then 68, for an identical request. This is not a bug in one tool. It is how large models run on shared hardware, and setting the temperature to zero does not remove it. What matters is knowing how big the wobble is, and reading every score with that margin in mind.
Key takeaways
- The model behind our score gave different answers to identical requests: about 3 points apart on average when we asked twice, and up to 16 points on the worst case we measured.
- Temperature 0 does not make a large model deterministic. The variation comes from how the computation is batched on the servers, not from random sampling.
- In our study of real videos from the same channel, score gaps of 1 to 3 points could not be told from chance. From 4 points up, the higher score belonged to the better-performing video about two times in three.
- Our analyzer now stores each answer and returns it when you submit the exact same image and title again, so a repeat shows the same score instead of a fresh draw.
- Any change, even re-saving the image, is a new question and gets a new reading. Compare versions by gaps of several points, never by one or two.
Why does an AI thumbnail score change when nothing changed?
Because the model's arithmetic is not perfectly repeatable. Large models run on servers that process many requests together, and the order in which millions of small calculations are added up varies from one batch to the next. Those tiny rounding differences rarely matter, but a score is built from many words the model writes one at a time, and one word that tips the other way near a threshold can move a sub-score by several points.
We found this the ordinary way: the same thumbnail and the same title, analysed twice, came back as 71 and then 68. Nothing in our own code was random. The prompt is built by fixed rules, the arithmetic after the model is plain arithmetic, and the model was already called at temperature 0, the setting usually described as "no randomness". The difference came entirely from the model's side of the call.
Temperature 0 is not the same as deterministic
Temperature controls sampling: how often a model picks a word other than its top choice. At 0 it always takes the top choice. But "top choice" is decided by numbers computed on hardware where the order of additions shifts with the load, so two runs can disagree about which word is on top when two candidates are nearly tied. That is why the same request can produce slightly different text, and slightly different scores, even with sampling switched off.
This applies to any tool that asks a large model for a number, including general chat assistants you might ask to "rate this thumbnail out of 10". If a tool never shows you the same answer twice, that is not proof it is unreliable. If it shows you a precise number with no margin, that number is claiming more than the model can deliver.
How big the difference actually is
We measure this as part of our evaluation runs, which send a fixed set of test cases through the real model and record what comes back. Each case is asked twice, and we compare the two global scores.
What the repeat runs showed
On a run of 26 test cases in early October 2026, the two answers to the same request differed by about 3 points on average, and the worst single case moved by 16 points. A second run on a smaller set of cases two days later gave the same picture: an average gap near 3 points and one case that moved by 10.
Three points sounds small on a 100-point scale. It stops sounding small once you compare it with the differences people use scores to decide. Two thumbnail drafts that score 64 and 66 are, on this evidence, the same score asked twice.
Which score gaps mean something
Repeatability is one half of the question. The other half is whether a gap predicts anything about real performance. Our study of 321 pairs of real videos compared two uploads from the same channel, published close together, and checked whether the higher-scoring one was the one that beat the channel's own baseline. Split by the size of the gap between the two scores, the results look like this.
| Gap between the two scores | Pairs | Higher score belonged to the better video | 95% interval |
|---|---|---|---|
| Tie | 17 | never, ties count as misses | |
| 1 to 3 points | 91 | 56.0% | 45.8% to 65.8% |
| 4 to 7 points | 92 | 64.1% | 53.9% to 73.2% |
| 8 to 14 points | 87 | 63.2% | 52.7% to 72.6% |
| 15 points or more | 34 | 76.5% | 60.0% to 87.6% |
A gap of 1 to 3 points was right 56% of the time, and its interval includes 50%: it cannot be told from a coin flip. From 4 points up, the higher score belonged to the better video about two times in three, and the largest gaps did better still. The median gap in the study was 6 points. These are exploratory breakdowns of one study on established English-language channels, so read them as a guide to scale rather than as exact thresholds.
The two numbers point to the same rule
The repeat runs say a score can move by about 3 points on its own. The study says gaps under 4 points do not predict which video does better. Both point to the same rule of thumb: treat two scores within 3 points of each other as equal, and only act on a difference of 4 points or more.
What our analyzer does about it
Every answer is stored against a fingerprint of the exact request: the image file, the title, the model and the instructions sent to it. When you submit the same image and title again, the analyzer returns the stored answer instead of asking the model a second time. You see the same score you saw before, and a repeat like this normally does not use up one of your analyses.
What still produces a new score
- A different image file, even one that looks identical. Re-exporting or re-compressing a thumbnail changes its bytes, and the analyzer sees a new image.
- Any change to the title, including punctuation or capitalisation.
- A change on our side to the model or to its instructions. When we change them, earlier answers no longer match, and the next request gets a fresh reading.
The last point means a score from a month ago and a score from today may not be directly comparable if the analysis changed in between. We record which version of the analysis produced each result, so we can tell, but the practical advice is simpler: when you compare two versions of your packaging, score both of them on the same day.
How to compare thumbnail scores without fooling yourself
- Score all the versions you want to compare in the same session, so they are read by the same version of the analysis.
- Change one thing at a time. If the title and the image both change, a gap cannot tell you which change caused it.
- Ignore gaps of 1 to 3 points. Treat those versions as equal and choose on other grounds, such as how clearly each one states what the video delivers.
- Act on gaps of 4 points or more, and give more weight to large ones. You can score two versions of your thumbnail side by side and read the gap against this rule.
- Confirm with real data. A score orders options before you publish; your own channel's results after publishing decide. Our guide on telling whether a thumbnail test result is real covers how.
Limits of these figures
The repeat runs used a small, fixed set of test cases, each asked twice, so the average and the worst case will move as more runs are added. The gap table comes from one study of established, English-language channels, in which no smaller segment is validated on its own. A score orders two options; it does not forecast views or click-through rate, and nothing here changes that.
Frequently asked questions
Why did my thumbnail get a different score the second time?
If anything about the request changed, such as the image file, the title or the version of our analysis, it was a new question and the model read it again. Large models do not return exactly the same answer to a repeated request, even at temperature 0. An identical image and title submitted again return the stored answer instead.
How much can an AI score vary for the same image?
In our repeat runs, two answers to the same request differed by about 3 points on average, and by as much as 16 points on the worst case. The average difference was small, but a single comparison can land on a large one, which is why small gaps should not decide anything.
Does setting temperature to 0 make AI scoring consistent?
No. Temperature 0 removes sampling, but the model's internal arithmetic still varies with how servers batch requests. When two possible words are nearly tied, that is enough to change which one the model writes, and with it a score.
How many points apart should two thumbnails be before I pick one?
At least 4. In our study of real videos, gaps of 1 to 3 points could not be told from chance, and the model itself varies by about 3 points between repeats. From 4 points up the higher score was right about two times in three, more for large gaps.
Can I compare a score from last month with one from today?
Only with caution. If we changed the model or its instructions in between, the two scores come from different versions of the analysis. Re-score both versions on the same day when the comparison matters.