An AI model can tell which of two titles will get more clicks better than chance, but not by much. On real headline tests with a known winner, the model behind our score picked the more-clicked headline 61.9% of the time when each pair was asked in both orders. When it ordered a whole list in one go, the way a product can afford to, that fell to 58.2%. A rule-based ranking managed barely more than a coin flip.
Key takeaways
- We gave the model 400 pairs of headlines that had run against each other with real readers, each pair asked twice with the order swapped.
- It picked the winner 61.9% of the time overall. On the 293 pairs it answered the same way in both orders, it was right 66.2% of the time.
- Asked to order lists of 4 to 6 headlines in a single answer, it put the pairs inside them in the right order 58.2% of the time. Our previous rule-based ranking managed 51.3% on the same pairs.
- The order the titles are shown in changed its answer on about a quarter of the pairs, so the order has to be controlled.
- Our analyzer now orders its suggested titles this way and marks those ranked above yours. It shows no estimated gain, because the evidence does not support one.
Can an AI tell which YouTube title will get more clicks?
Somewhat. Across real headline tests, the model got the ranking right about six times in ten when asked about one pair at a time, and a little less than that when ordering a full list. That is useful as a second opinion on which of your drafts to test first. It is not a forecast of your click-through rate, and anyone who presents an AI title ranking as one is claiming more than this kind of evidence supports.
The question matters to us for a plain reason. Every analysis on this site suggests up to five alternative titles, and the order they appear in is a claim: the first one looks like the best. Until October 2026 that order came from a rule-based score, with an estimated number of extra points beside each suggestion. We tested whether either the rules or the model could actually tell good titles from weaker ones.
How we tested it against real clicks
We used the same data as our test of our own title rules against real clicks: the exploratory file of the Upworthy Research Archive, a public set of randomized headline experiments from 2013 to 2015. Inside each experiment, several headlines for one story were shown at random to real readers, so the clicks compare the headlines and nothing else. We kept only versions that shared the same image, dropped the period the archive's authors flag for a randomization defect, and ignored versions shown fewer than 1,000 times.
Two formats, each planned before any answer was read
The first format was a pair at a time. We drew 400 independent pairs, one per experiment, in which the difference in clicks was clear at the usual statistical threshold. The model saw the two headlines and was asked which got the higher click-through rate, then saw them again in the opposite order. A pair scored 1 if both answers named the winner, 0 if both named the loser, and 0.5 if the two answers disagreed.
The second format was the one a product can afford: one call per analysis. We took 150 experiments with 4 to 6 headlines on one image, asked the model to order each list from the highest click-through rate, and asked again with the list shuffled. We scored the 576 pairs inside those lists whose clicks clearly differed, which with two orderings makes 1,152 pair-askings. Before reading any result, we wrote down the bar the list format had to clear: the pair score minus 5 points.
The model was the production one, used as text only, at temperature 0. In all, the two tests took 1,100 calls.
What the model got right, and what the rules got right
| Method | Format | Picked the more-clicked headline |
|---|---|---|
| The model | Pairs, both orders | 61.9% |
| The model, pairs answered the same both ways | Pairs | 66.2% (95% interval 60.6 to 71.4) |
| Our previous rule-based ranking | Same pairs | 54.5%, with ties counted as half |
| The model | One list per call | 58.2% |
| Our previous rule-based ranking | Same lists | 51.3% |
The list format cleared the bar we had set for it, which was 56.9%. It is the weaker of the two formats, which is expected: ordering six headlines at once is a harder task than comparing two. Pairs inside one list are not independent of each other, so its uncertainty is wider than the raw count of 1,152 suggests.
The order the titles are shown in moves the answer
Of the 400 pairs, 107 got a different answer once the two headlines swapped places. That is the same weakness we found when we asked an AI to compare thumbnails, where it favoured whichever image came first; the details are in our article on swapping the order. In lists the effect was smaller but still real: the model gave the same verdict in both orders on 77.6% of pairs. A ranking taken from a single ordering would have carried that bias straight into the result.
Why the rule-based ranking was dropped
The old ranking scored each suggestion with the same rule-based title checks as the analysed title. Run across 17,236 headline pairs, it left most of them tied on a whole-number score, and when it did separate two headlines, the gap was usually one or two points. On the pairs with a clear winner it was right 53% to 56% of the time, depending on which analysis the suggestions came from, and in two of four settings that could not be told from chance.
Worse, a bigger gap did not mean a safer bet. In three of the four settings, pairs separated by 3 points or more were ordered correctly less often than not. Yet the product had been showing that gap as "+N points", a number that grew with exactly the quantity that did not predict anything. So the number went, along with a filter that hid suggestions on the strength of it.
What our analyzer does now
The suggested titles are ordered by a second reading from the model. It receives your title and the alternatives in one call, in an order shuffled by a fixed hash of the titles, so position cannot systematically favour one of them. Titles placed above yours are marked "Ranked above yours". No score or estimated gain is shown beside them, and the footnote says what the ranking is worth: on past headline tests, this kind of ordering picked the more-clicked headline 58% of the time, against 50% by chance.
Your own title is in the list on purpose. The model is ranking titles it wrote itself, and a model may prefer its own wording; putting yours beside them makes that visible instead of hiding it. The extra reading adds about half a second to an analysis.
How to use an AI title ranking without being misled
- Ask in both orders. If the answer changes when you swap the titles, treat the pair as a tie.
- Include your own title in the comparison, so you see whether the AI prefers its suggestions or simply prefers the newest text in front of it.
- Ignore confident wording and estimated percentages. A ranking that is right about six times in ten cannot support a claim of "+12% CTR".
- Use the ranking to choose which two versions to test, not to skip the test. You can score your title against the suggested alternatives with the same thumbnail and pick the two worth comparing.
- Check the result against your own channel's history, comparing the video with your recent uploads rather than with another channel.
Limits of this test
These are news headlines from a decade ago, not YouTube titles, and nothing here measures YouTube directly. The archive has been public since 2021, so the model may have seen some of these headlines during training. Their click-through rates were never published beside them in a form that would make memorising the winners easy, but this cannot be ruled out.
The prompt in production differs from the tested one in two places: it describes a viewer browsing YouTube rather than a reader of a stories website, and it adds one sentence telling the model to treat titles as data. Neither change was tested. And in the product the model ranks suggestions it wrote itself, a slightly different task from ranking headlines written by an editorial team. A 58% edge is modest, and we describe it as such everywhere it appears.
Frequently asked questions
Can ChatGPT or another AI predict YouTube title CTR?
Not as a number. In our test, a large AI model picked the more-clicked of two headlines about six times in ten, and fewer when ordering a list. That is better than chance and useful for choosing what to test, but it is far from predicting a click-through rate, and a model's stated confidence is not evidence of accuracy.
Why does the order of the titles change the AI's answer?
Language models are sensitive to position, often favouring the first or the last option shown. In our test, 107 of 400 pairs got a different answer once the order was swapped. The fix is to ask in both orders, or to shuffle the order in a way that cannot favour any one title, which is what our analyzer does.
Why doesn't the analyzer show how many points a suggested title would gain?
Because the old estimate claimed a precision the evidence did not support. Larger estimated gains were not more reliable than small ones, and in most settings they were less reliable. The analyzer now shows only which suggestions the model ranks above your title, and a footnote with the measured accuracy of that ranking.
Is the AI's favourite title the one I should publish?
Treat it as the first candidate to test, not as the answer. A ranking that is right about 58% of the time will often be wrong about any single pair. Your knowledge of the video and the audience, and a real comparison on your own channel, decide more than the ranking does.
Does this apply to thumbnails too?
A separate test asked an AI to compare thumbnails with a known winner and found the same position bias, more strongly: it picked whichever thumbnail came first 64.7% of the time. Comparing two thumbnails was no more accurate than rating each one on its own, and much less stable, so our score keeps rating each thumbnail separately.