Why one creative execution may be too little evidence
A failed ad does not necessarily disprove its underlying idea. In our creative reviews, closely related executions sometimes produce very different results. Testing a few purposeful variations helps distinguish a weak execution from a concept that deserves more work.
By NewForm · Updated
Key takeaways
- Separate the concept you want to test from the particular hook, edit or opening frame used to express it.
- Our video review reports that about 25% of variations within winning concepts met its winning criteria. That is a dataset observation, not a universal success rate.
- Additional variations provide more evidence, but their results are not independent and no fixed count guarantees a reliable conclusion.
One concept can contain both winners and losers
In a review of more than 10,000 ads, we grouped executions by concept. The video reports that only about a quarter of the variations within winning concepts were themselves winners. It does not disclose the full account mix or the threshold used to define a win, so this should not be used as a benchmark for every advertiser.
A separate video shows Ad Atlas clustering 30,000 creatives by similarity. Within tightly grouped examples, some ads attracted spend and others did not. These observations are a reason to test more than one execution; they do not isolate whether the differences came from the creative, delivery conditions or another factor.
What the probability example actually tells you
Consider a deliberately simplified model: each execution has a 25% chance of success, independently of the others. Under those assumptions, the chance that every tested execution fails is 0.75 raised to the number of executions. One gives 75%; two give 56.25%; eight give about 10%; ten give about 5.6%.
This corrects the rough calculation in the video, which associates ten to eleven variations with a 10% miss rate. More importantly, real variations share messages, footage, audiences and delivery conditions. Their outcomes need not be independent, and the observed win rate need not carry over to the next concept. The calculation illustrates the danger of a single attempt; it does not establish statistical significance or prescribe a testing quota.
Give each variation a purpose
Start with a clear concept and change an execution detail you can name: the opening hook, a visual lead-in or the way the same message is presented. Record those differences so a promising result tells the team what to investigate next.
The workflow described in our Ad Atlas video aims to find an efficient variation before concentrating scaling spend. Its preference for choosing among close variants is a testing strategy, not proof that similar ads always compete with each other or that only one can work. Let your available budget and the evidence determine how far to take a concept.
A variation matrix that keeps the concept recognizable
Suppose a hypothetical budgeting app wants to show that users can understand where their money went without building a spreadsheet. Keep that product promise and the demonstration stable while testing how the execution introduces it. Each variation should have a named purpose that the team can inspect after delivery.
| Execution | Opening | Question for the review |
|---|---|---|
| A: screen-led demonstration | Show the spending breakdown first | Does immediate product evidence help the intended user understand the benefit? |
| B: first-person problem | A creator describes losing track of small purchases | Does the recognizable problem attract customers who use the product? |
| C: contrasting workflow | Show manual spreadsheet work before the app | Does the comparison explain the same benefit more clearly? |
| D: direct explanation | State the product benefit before the walkthrough | Does a plain introduction improve understanding and customer quality? |
Log the execution and the conditions around it
Keep a record of the concept, hook, opening frame, edit, offer, destination, audience, spend, conversion window, and outcome. If execution B performs better, that is evidence to investigate the first-person framing. It does not automatically prove that the hook alone caused the difference when footage, delivery, or other conditions also changed.
Use the result to choose the next purposeful variation. If no execution receives useful delivery, first examine the test structure and available budget. If several adequately observed executions miss the business goal, document why further work on this concept would be justified before producing another batch. There is no universal number of variations that replaces that judgment.
The Meta creative testing framework describes the wider process. If you are evaluating an agency, ask it to count concepts and variations separately, so a large deliverable count tells you what new ideas you will actually receive.
Adapted from NewForm’s original videos on creative strategy and paid social. Worked examples, where included, are NewForm editorial illustrations, not measured client results.
Explore NewForm Framework →