Why comparing ads in an ad set is not a controlled A/B test
Comparing ads in a standard ad set helps you choose what to run, but it doesn't isolate the creative's effect. Delivery, audiences and creative differences all affect results. Structured concept testing makes those comparisons easier to interpret.
By NewForm · Updated
Key takeaways
- Ordinary ad-set results do not establish that ads received comparable, randomly assigned audiences.
- Comparing entirely distinct ads introduces multiple confounding variables, making it difficult to isolate what drove performance differences.
- A structured testing model clusters variations around a single concept or persona, testing specific combinations such as modular hooks and bodies.
- Repeated patterns can generate better creative hypotheses; they do not by themselves prove which element caused the result.
Dynamic delivery confounds standard ad set comparisons
A controlled A/B test uses random assignment to comparable groups and defines the difference being tested. Simply placing two ads in the same ad set does not establish those conditions. The videos highlight that different creatives may receive different delivery and reach different audience segments.
An observed CPA difference is still useful for an operating decision. What it does not establish on its own is whether the creative caused the difference, or whether one ad would be better for every audience. The critique here concerns casual ad-set comparisons, not a claim that controlled advertising experiments are impossible.
The problem with changing several creative variables
Ads in the same ad set often differ in pacing, hooks, value propositions and calls to action. Audience delivery is uneven too. When those variables change together, you can't attribute a performance difference to any single element.
An opening frame might attract more engagement even when the value proposition is weaker. Calling that ad the winner tells you little about what drove the result unless you control the individual creative variables.
Structuring tests around concepts and modular components
We structure creative tests around distinct concepts or personas. Hold that frame constant, then work through variations of the components instead of treating unrelated ads as a split test.
This structure organizes learning; it does not remove delivery bias or turn the workflow into a randomized experiment.
- Dedicate each ad set to a single defined concept or customer persona to maintain messaging consistency.
- Use modular hook-and-body variations: pair distinct hooks with consistent body segments to explore the combinations systematically.
- Compare hook and body combinations in context, then carry promising patterns into the next round of testing.
Look for patterns across test cycles
Review reports across several concepts and personas before treating an isolated result as a general creative rule. A recurring pattern is a stronger reason to investigate an idea than one unusually successful asset.
Our videos describe using language models to summarize those reports. They can help propose patterns for a strategist to check against the actual results. A model summary is not independent evidence, and repeated correlations still do not establish causation.
Adapted from NewForm’s original videos on creative strategy and paid social. Worked examples, where included, are NewForm editorial illustrations, not measured client results.
Explore NewForm Framework →