Why one creative execution may be too little evidence
A failed ad does not necessarily disprove its underlying idea. In our creative reviews, closely related executions sometimes produce very different results. Testing a few purposeful variations helps distinguish a weak execution from a concept that deserves more work.
By NewForm · Updated
Key takeaways
- Separate the concept you want to test from the particular hook, edit or opening frame used to express it.
- Our video review reports that about 25% of variations within winning concepts met its winning criteria. That is a dataset observation, not a universal success rate.
- Additional variations provide more evidence, but their results are not independent and no fixed count guarantees a reliable conclusion.
One concept can contain both winners and losers
In a review of more than 10,000 ads, we grouped executions by concept. The video reports that only about a quarter of the variations within winning concepts were themselves winners. It does not disclose the full account mix or the threshold used to define a win, so this should not be used as a benchmark for every advertiser.
A separate video shows Ad Atlas clustering 30,000 creatives by similarity. Within tightly grouped examples, some ads attracted spend and others did not. These observations are a reason to test more than one execution; they do not isolate whether the differences came from the creative, delivery conditions or another factor.
What the probability example actually tells you
Consider a deliberately simplified model: each execution has a 25% chance of success, independently of the others. Under those assumptions, the chance that every tested execution fails is 0.75 raised to the number of executions. One gives 75%; two give 56.25%; eight give about 10%; ten give about 5.6%.
This corrects the rough calculation in the video, which associates ten to eleven variations with a 10% miss rate. More importantly, real variations share messages, footage, audiences and delivery conditions. Their outcomes need not be independent, and the observed win rate need not carry over to the next concept. The calculation illustrates the danger of a single attempt; it does not establish statistical significance or prescribe a testing quota.
Give each variation a purpose
Start with a clear concept and change an execution detail you can name: the opening hook, a visual lead-in or the way the same message is presented. Record those differences so a promising result tells the team what to investigate next.
The workflow described in our Ad Atlas video aims to find an efficient variation before concentrating scaling spend. Its preference for choosing among close variants is a testing strategy, not proof that similar ads always compete with each other or that only one can work. Let your available budget and the evidence determine how far to take a concept.
Adapted from NewForm’s original videos on creative strategy and paid social.
Explore NewForm Framework →