How we tested whether the ThumbSignal score means anything

The question, and why it is shaped this way

The obvious test is to score a batch of thumbnails and correlate the scores with view counts. We did not run it, because it would have measured channel size. Views are driven by subscribers, topic, timing, outside traffic and the age of the video far more than by packaging, and the bias points the flattering way: big channels can afford designers and also have big audiences, so the correlation would be positive and meaningless.

Comparing two videos from the same channel, published close together, cancels channel size, audience, budget and design skill, because all of them are constant within the pair. What is left is a fair question with a known chance rate. The score was never built to predict a number of views; it only has to put two inputs in order, so that is the task we tested.

What was fixed before any result existed

The method was written down and frozen on 2026-09-04, before a single video was scored. It names the hypothesis, the sample, the exclusions, the outcome, the test and the controls, and it commits in advance to publishing the result whatever it was, a null included. Writing it first is the only reliable defence against trying six ways of slicing the data and reporting the one that worked. The honest expectation recorded in that document was a small effect: a result near 60% would be a real finding, one near 85% would be treated as a bug in the design before it was believed.

The sample

• 60 channels, chosen by hand before collecting anything, ten in each of six categories (tech, education, cooking, gaming, fitness, lifestyle) so that no single visual convention dominates. The list was frozen and no channel was added, removed or swapped afterwards.
• From each channel, the 300 most recent uploads were considered. A video was left out if it was a Short or under three minutes, published less than 45 days earlier (views not yet settled) or more than two years earlier (another algorithm, another channel), a live stream or premiere, without public view counts, or without a usable thumbnail, or if it had fewer than eight neighbours to build a baseline from.
• For each video we compared its views with the median of the ten uploads before it and the ten after it: a rolling baseline, so a channel that grew fivefold is not read as a list of early flops and recent hits, and the median, so one viral video does not redefine its neighbours' baseline.
• In each channel with at least 20 usable videos, the top fifth were winners and the bottom fifth losers. A winner and a loser formed a pair only if they were published within 120 days of each other and the winner had at least twice the loser's view multiple. No video is in more than one pair, and no channel contributes more than eight.

That produced 321 pairs from 47 of the 60 channels. We had set 400 pairs as the number at which the test could reliably detect a true accuracy of 57%. We did not top the sample up from new channels to reach it, because adding channels after seeing the yield is selecting the sample; we report the study as it came out. At 321 pairs it reliably detects a true accuracy of 58% or better, and is underpowered for anything between 55% and 57%.

The result

Each video was scored by the product's own pipeline as it stood on the run date, the same code path that runs when you analyse a thumbnail, with its real title, and the order of the scoring was shuffled so nothing downstream knew which side of a pair a video was on. A pair counts as a success when the score ranks the winner above the loser. A tie counts as a failure, which is the conservative choice.

Two figures were added after the result was seen, and are labelled as post hoc wherever they appear. Because ties count as failures, a score with no discriminating power at all would reach 47.35%, not 50%. And among the 304 pairs the score did separate, it ranked the winner higher 62.8% of the time. We left the pre-registered test exactly as written: it is harder to pass than it needed to be, and that is not a reason to re-specify it after the fact.

Read it as this: the score carries some signal about which of two comparable videos did better, a few points above a coin flip after the ties are accounted for. It is a modest effect, as the pre-registration expected. It is not a forecast of views.

What the controls showed

A study that cannot fail is not evidence, so four checks were declared in advance and run alongside the main test. All four are published whatever they showed.

• Label shuffle (C1). We scrambled which video was the winner, 1,000 times. Accuracy centred on 47.1%, where the tie rule says a score with no signal should land (47.35%). Our first threshold for this check expected 50%, which was wrong for the reason above; we corrected the threshold, not the rule, and recorded it in the decision log. The measured accuracy of 59.5% is far outside the shuffled range.
• Template channels (C2). On the 5 channels that reuse one thumbnail template for every upload (39 pairs), where the packaging barely differs, accuracy fell to 48.7%, close to chance. On the other 282 pairs it was 61.0%. A score that is confident where the thumbnails barely differ would be reading something other than the packaging; this one is not.
• Trivial baselines (C3). One-line rules scored the same pairs, and none matched the score:

• Title swap (C4). We re-scored every thumbnail with the title of its pair partner, the image byte for byte unchanged. See the next section.

The finding that cuts against a thumbnail tool: the title carries more of the signal

Swapping the title moved the score by 11.3 points on average (median 8), and only 3.9% of thumbnails scored the same under a different title, so the title is plainly part of the score. With the titles swapped, the score ranked the winner higher in 42.7% of pairs (95% interval 37.2% to 48.3%, p = 0.0101 against 50%). That is below chance: the ranking did not degrade, it inverted.

The two readings make different predictions. If the score were driven by the thumbnail alone, swapping titles would leave accuracy at 59.5%. If it were driven by the title alone, accuracy would fall to 35.2%. The observed 42.7% sits much nearer the second. Random noise cannot explain it, because noise applied to both sides of every pair pushes accuracy toward 50%, never below it. So on this evidence the title contributes more of the signal that separates a better-performing video from a worse one than the thumbnail does, though not all of it.

Two cautions. Swapping a title also breaks the match between title and thumbnail, which is itself a scored factor, so the swap is not a pure test of the title. And the experiment changed only the title: it measures the title's share directly and the thumbnail's only by subtraction. The mirror-image experiment, swapping thumbnails and holding titles, has not been run.

Which parts of the score separated winners from losers

For each factor we measured the standardised difference between winners and losers (Cohen's d). This is exploratory, descriptive of this sample, and the easiest result here to over-read: every effect is small, and none of these figures is presented as a confirmation of anything.

Promise clarity separated the two groups most. It is the factor the study supports saying something about; it does not follow that fixing it will lift a particular video.

The score was also more often right when it was more decisive: 56.0% when the two videos were within 3 points, 76.5% when they were 15 points or more apart (34 pairs).

What this study does not show

• It tests the score, not the advice. Whether applying a suggested title or a fix improves a real video is a different experiment, and it has not been run.
• It is observational. Nothing here shows that changing a thumbnail changes views. It shows that the score is associated with a difference in outcome that already existed.
• The pairs were chosen because they differ a lot. The figure applies to clear over- and under-performers, not to any two videos. Two videos that performed alike would be a coin flip for any judge.
• Established channels only. Every channel had a track record to be measured against. Nothing here says how the score behaves on a new channel.
• Views were read once, on the collection date (2026-09-04), and keep accruing.
• Categories cannot be compared with each other. Between 30 and 75 pairs per category is too few. We report each category's result, from 51.6% in gaming to 70.5% in lifestyle, and do not claim that any of the differences between them is real.
• One model. The run used qwen.qwen3-vl-235b-a22b. The model may have seen some of these public videos in training; that cannot be known from outside.

What you can check, and what you cannot yet

The protocol (hypothesis, sample rules, exclusions, outcome, test, controls, limits) was frozen on 2026-09-04, the sample was frozen the same day, and a decision log records every protocol decision taken afterwards, including those we considered and declined: for example, not topping the sample up from 321 pairs to 400, and the correction to the label-shuffle threshold described above. The run is identified by the fingerprint of the model, prompt, formula and schema it executed against (1d2ab116494c); the data were collected on 2026-09-04 and analysed on 2026-09-09.

What is public. Since 7 October 2026 the pre-registration is in a public repository, byte for byte the file we first committed on 2026-09-04, with its SHA-256 fingerprint (b3910ab92111a29ece46d2fcda61f9b41205b2a6db3b14c3cea9c2dae5ac1171) and the fingerprint of the frozen channel list, which we do not publish because it names channels. Anyone can confirm that the text has not changed since GitHub recorded it.

The score of every pair. The same repository holds a table of the 321 pairs: the category and four whole-number scores for each (winner and loser, with the original titles and with the titles swapped), and no channel, video, title, link or date. A short script recomputes the 59.5%, its interval, the exact test and the title-swap result from that table (SHA-256 of the table: 217a0e2e055768632e99ca0f594d6b2e9503cb86f51efec759a3c0bda3bb8ba0). That lets anyone confirm the arithmetic. It does not let anyone confirm that the scores are the ones the product produced, or that the pairs were chosen as the protocol says, because the identifiers that would allow it are not published.

The analysis output. The figures on this page are those of the analysis file (SHA-256: a824ba700c9cbb1123e3fbb8dec4d5ac223209a5120e8196d8afb72cb5b5c65f), published in the same repository: aggregates only, with the figures added after the first analysis named as post hoc in the file itself. It was written on 9 September 2026, the first analysis having been run on 4 September; the repository shows it from 7 October 2026, and the earlier dates are from our own records.

Three limits on that. That repository proves the text from 7 October 2026 on, not before: the repository where the rest of this history is kept is private, so the earlier dates are from our own records and a reader cannot verify them independently. The earliest record that anyone else holds of the text is GitHub's log of a push to that private repository on 5 September 2026 at 00:10 (Paris time), which carried the pre-registration and the first analysis together, so nothing outside our own records shows which came first. And we do not republish the collected data (public videos, third-party channels); we publish the aggregates above and the scores of the pairs. If you want the frozen channel list or the decision log to review, write to contact@thumbsignal.com.

Corrections to earlier articles

Until 6 October 2026, five blog articles presented datasets that were never collected: a sample of 100 thumbnails, a series of 100 makeovers, a study of 500 titles, 15,000 scored thumbnails and 12,487 scored videos. The articles were written from topic briefs and our review did not catch it before publication. Nothing on this page, and nothing in the rewritten articles, comes from those datasets: every figure traces to the study above. A sixth article, on this study, contained one overstatement about its dates. Each is listed here with what it said and what it says now.

If you cited one of the earlier figures, please correct the citation. If you find another claim in our articles that this study does not support, write to contact@thumbsignal.com.

Try the score on your own thumbnail

The score you get is produced by the pipeline this page describes. It orders packaging; it does not predict views. Read the result for the weakest factor and what to change, not as a verdict. Analyse a thumbnail or read the article on the 321 pairs.