THE CURRICULUM / CH 3 — Marketing Mastery

3.5Testing Strategy

One variable, a budget that can produce an answer, a fixed window, and kill criteria written before launch.

5 min read

Day four. Two ads, same audience, same budget, one variable between them. One has three purchases and the other has none, and the one with none is the one you wrote. Your finger is over the button that turns it off, and the window you agreed to on Sunday still has three days left in it.

A test against a vibe check

A test

  • The question is written in one sentence, first.
  • One variable differs. Metric, window and kill criteria fixed first.
  • Nothing is edited inside the window.

A vibe check

  • Several ads go live because they were finished.
  • Everything differs, including budget and audience.
  • The metric is whichever looks best in the report.

A CONFOUNDED TEST LOOKS EXACTLY LIKE A REAL ONE.

which is what makes it worse than no test

BUDGET IS ARITHMETIC, NOT AMBITION

The learning phase drawn as its own mechanism. Fifty marks stand for the roughly fifty optimisation events the delivery system counts inside a seven day window before it treats a set as settled. Below them, delivery is drawn jagged while the count is running and level once it has finished, on an axis with no scale. Below that, the edits that put the count back to nothing: a new audience or placement set, a large budget change, a different optimisation event, swapping the creative set, or changing the offer or price. Count events, not days.
Roughly fifty optimisation events in seven days, before delivery settles.
  • Purchases are rare. A handful against a handful is chance dressed as a finding.
  • A budget that cannot generate fifty of the event you optimise for leaves delivery unstable.
  • Three options: raise it, consolidate rather than fragment, or optimise further up the funnel.
A worked example of a seven day product test at sixty dollars a day, drawn as a vertical track with the cumulative spend running beside it. Day one sixty dollars, day two one hundred and twenty, day three one hundred and eighty and the first decision point: is anyone reaching the cart at all, because if not it is the page or the creative. Day four two hundred and forty. Day five three hundred and the second decision point: what is one order costing now, held against forty dollars of contribution. Day six three hundred and sixty. Day seven four hundred and twenty and the test ends: kill it, change one thing, or scale, with the reason written down first. Four hundred and twenty dollars divided by forty dollars of contribution is ten and a half orders to break even on the test. It is a worked example, not a quote and not a result.
A test laid out day by day, with the decision points marked.
The three layers of an ad account drawn as nested blocks, with the decisions each one owns. The campaign owns the objective, the bid or buying strategy, the budget if set at that level, and the attribution setting for the report. The ad set owns who it can be shown to, where it is allowed to appear, the optimisation event it learns on, the schedule, and the budget if set there instead. The ad owns the creative itself, the destination it sends people to, and the post the comments accumulate on. No budget, bid or result figure is shown.
The same spend split across more ad sets starves every one of them.

DECISION WINDOWS

A vertical timeline of one product test with three decision points. Days one to three: leave it alone while delivery settles. Days three to five: read where the funnel drops rather than the profit line. Days five to seven: kill it, change one variable or scale, and write the reason down. No spend or target figures appear.
One to three, three to five, five to seven, and what each settles.

What you may conclude, and when

PointYou can concludeStill premature
Day oneDelivery started, spend pacing, ads approved, link works.Everything else.
Day threeThe upper funnel begins to separate.Purchase verdicts.
Day sevenA full weekly cycle, and learning has had its window.Nothing.

every edit inside the window restarts the clock.

  • What restarts learning: large budget jumps, a changed optimisation event, edited targeting, swapped creative, a long pause.
  • Fragmenting across small ad sets starves all of them without resetting anything.
Two routes to the same height. The first is a ladder of small, equal steps taken often, and the line holds its footing all the way up. The second is a single large jump: the line is flat, rises once, and then moves about unsteadily before settling, because a change that size sends delivery back to settling. Both lines finish level on purpose. The step heights are drawn equal and no amounts, currency or percentages appear.
Small steps the delivery survives, beside the one jump that resets it.
  1. Check the winner was not a single-day artefact.
  2. Confirm it against a second audience before building on it.
  3. Scale in steps, or duplicate higher and leave the original up.
  4. Log the losers. The cheaper half of the education.
A decision tree for what to do with a running ad, ending in kill, hold or scale. The first question is whether it has run the window you agreed to; if it has not, hold, because the window was the decision you already made. If it has, the second question is whether any part of it is still improving; if it is, hold, because changing it now teaches you nothing. If it is not, the third question is whether it pays for itself on your own numbers; if it does not, kill it, because waiting longer will not make it pay. If it does, scale it, one step at a time, on the part that is already paying. Every branch here is a question rather than a number, because the threshold is the reader's own and is written down before the window starts.
Kill, hold or scale. The branch was decided before you opened it.
Statement poster: "We do not lose money. We purchase information."

THE TEST YOU CAN RUN TOMORROW

A curve of break-even return on ad spend against contribution margin. Break-even ROAS is one divided by contribution margin, so the curve falls steeply as margin rises: a thin margin needs a high multiple to break even, a fat margin needs a low one. This is arithmetic, not a target.
One divided by your contribution margin. Arithmetic, not a target.
  1. Write the question in one sentence, and name the variable that moves.
  2. Write success and kill criteria, with metric and window. Date it.
  3. Set the spend that window needs, from your break-even cost per purchase.
  4. Build variants so exactly one thing differs, named to explain themselves.
  5. Launch identically: same audience, event, placements, start time.
  6. Edit nothing inside the window.
  7. Read the upper funnel first, then downstream, then cost per purchase.
  8. Record the verdict and the reason in one line.
The five funnel stages from impressions to purchases, each with a narrowing bar, and between them what a drop at that step points at: the hook or audience, a page that does not match the ad, the offer or price or proof, and checkout or shipping or trust. No conversion rates are shown.
Impressions to purchase, and what a drop at each stage points at.

TAKEAWAYS

  • A test is a question with one variable that moves and a pre-agreed answer.
  • Budgets too small to produce enough events inside the window buy noise, not answers.
  • Day one shows faults, day three the upper funnel, day seven a verdict.
  • Kill criteria are only useful if they were written before you saw the numbers.

CHECK YOURSELF

Which of these restarts the learning period?

On day one of a test, what can you legitimately conclude?

DO THIS BEFORE THE NEXT MODULE

  1. 01Write one test question in a single sentence, naming the variable that moves.
  2. 02Write the kill criteria for it, with window and spend ceiling, and date them before you build anything.
  3. 03Run that test without editing it inside the window, then log the verdict in one line.
Enrol free