A product page can be changed in many ways at once, a new photo, a shorter description, a different button colour, but changing several things together and then comparing the result to the old page leaves no way to tell which change actually caused a better or worse outcome. A test only proves something if everything except the one thing being tested stays the same.
This lesson gives you the method a factory or retailer uses to set up a fair, one-variable test before drawing any conclusion from it. You will apply it to Delta Garments' own factory-outlet online store, testing a new product photo for an Order A-style second-quality T-shirt against the existing photo, because a test with more than one change mixed in cannot honestly tell anyone which change worked.
You need: the idea of reading a product page's own baseline conversion rate correctly, from Lesson 2.1, is useful background but not required.
By the end of this lesson you can:
Not covered here: judging whether a completed test's result is real or ordinary noise (Lesson 2.3), or diagnosing a page that already gets plenty of traffic but converts poorly (Lesson 2.4). This lesson only teaches how to set a test up correctly before it runs.
| Term | Meaning |
|---|---|
| One-variable test | A test in which exactly one element of a page changes between the control and the test variant, so any difference in result can be attributed to that element alone. |
| Test variant | One of the two or more competing versions of a page being compared in a test. |
| Control condition | Keeping every factor except the one variable being tested the same between the two versions, so the comparison isolates that single variable. |
| Confounding factor | Any other change happening at the same time as a test that could also explain a change in the result. |
| Traffic split | The way incoming visitors are divided between the control version and the test variant, usually as close to equal as possible. |
| Live page | The version of a page a visitor is actually shown during a test, whether the control or the test variant, decided by the traffic split. |
| Test period | The fixed length of time a test runs before its result is read, agreed in advance and not extended or shortened partway through. |
Changing the photo and the price at the same time, then seeing a higher conversion rate on the new version, leaves no way to know whether the photo, the price, or some combination of the two produced the result, because two explanations now compete for the same single number.
| What can differ | The shapes you meet | Why it matters to you |
|---|---|---|
| Business model | A Make to Order factory outlet like Delta Garments' own store tests a single style at a time, since stock of any one style is limited and does not repeat indefinitely. A Make to Stock retailer running a standing style can run a longer test, since the same style stays available throughout. | The test period and traffic split should fit how long the product itself will realistically stay on sale. |
| Market | An export brand's online store serving more than one country at once should split traffic within each country separately if buying behaviour differs by country, rather than pooling all countries into one combined result. | Pooling different markets into one test can hide a result that only holds true in one of them. |
| Traffic volume | A page with heavy daily traffic can reach a trustworthy split within days. A lower-traffic outlet page, like Delta Garments' own second-quality store, needs a longer test period to collect enough visits in each variant to mean anything. | Lower traffic does not make a one-variable test wrong; it makes the test period that produces a trustworthy result longer. |
| Number of variants tested | Testing two versions splits traffic in two; testing three or four photo options at once splits the same traffic further, so each variant collects fewer visits in the same period. | More variants tested at once need proportionally more total traffic, or a longer period, to keep each comparison reliable. |
Delta Garments runs a small factory-outlet online store selling second-quality Order A-style T-shirts that do not meet Northline Apparel's export standard but are still sound enough to sell locally at a lower price. The digital team wants to know whether a new on-model product photo converts better than the existing plain studio photo.
Everything else on the page stays exactly the same between the two versions: the same price, the same product description, the same page layout, and the same two-week period, with incoming visitors split roughly 50/50 between the existing studio photo (the control) and the new on-model photo (the test variant).
The worked numberAn even split, held for the full two weeks without adjusting the ratio partway through, is what lets each variant's own conversion rate later be compared on equal footing; changing the split mid-test would itself become a confounding factor.
A team correcting a spelling mistake it happens to notice on the test variant's page, without making the same correction on the control page, has quietly introduced a second variable into what was meant to be a one-variable test, even though the change felt small and entirely unrelated to the photo actually being tested. The cost of this mistake is not the correction itself, which was harmless on its own; it is that the entire test's result becomes unusable, since no one can now honestly say whether the photo, the correction, or some mix of the two produced the outcome, wasting the full two weeks of traffic the test consumed and forcing the whole comparison to be run again from the start.
If you want to know whether a new photo sells better than an old one, change only the photo and leave everything else on the page exactly the same. Split your visitors evenly between the old version and the new one, and keep that split steady for a fixed period agreed before the test starts. Do not fix, tweak, or improve anything else on either version while the test is running, even something small and unrelated, because any second change ruins your ability to say the photo alone caused the result.
Partway through the two-week test of the new on-model photo against the existing studio photo, a team member notices the new photo file is smaller in size and loads noticeably faster on a slow mobile connection, purely because of how it was exported, not because anyone intended a speed change.
Should Delta Garments treat the test's final result as proof that the on-model photo itself converts better? Decide before opening the answer.
No. The unintended difference in file size and load speed is a confounding factor; the test can no longer isolate the photo's own visual effect from its faster loading time.
1. A one-variable test requires every other factor to be held equal, and load speed, especially on slow mobile connections, is itself a factor that can change conversion independent of what the photo shows. 2. A third of visitors are on connections where this difference is large enough to matter, so the effect is not a trivial rounding error to ignore. 3. The honest response is to note the confound, re-export both images to a matching file size, and re-run the test, rather than reporting a result that mixes two causes as if it proved one.
Order A's T-shirt carries a shop price of USD 19.90 per piece, though Delta Garments' own second-quality outlet sells the same style discounted to roughly 60% of that, USD 11.94 per piece (docs/02 section 7 anchors the full-price figure). If a confounded test wrongly credited the on-model photo with a result that was really the faster load speed, and Delta Garments rolled the photo out without fixing the underlying speed gap on other pages:
A small weekly figure compounds silently across every week a wrong lesson keeps being applied, a far larger true cost than the one extra week a clean re-test would have taken.
Use the T23-C02 Digital analytics and testing worksheet, "Test setup log" sheet. For each test, record the one variable changed, what was held fixed, the traffic split, the test period, and any confounding factor noticed while the test was running. Start with Delta Garments' example above, then add your own test.
[Download the Digital analytics and testing worksheet](../../templates/T23-C02-digital-analytics-and-testing-worksheet.xlsx)
Download the Digital analytics and testing worksheet (Excel)
1. Why does changing the photo and the price at the same time make a test's result impossible to interpret cleanly?
2. Delta Garments' test variant photo happens to load faster than the control photo on slow connections, for reasons unrelated to the test. What should the team do?
3. Why should a test's traffic split and test period be fixed in advance, rather than adjusted once early results start coming in?
| EN | AR | FR | TR | Meaning |
|---|---|---|---|---|
| One-variable test | اختبار متغيّر واحد | Test à une seule variable | Tek değişkenli test | A test in which exactly one element of a page changes between the control and the test variant, so any difference in result can be attributed to that element alone. |
| Test variant | متغيّر الاختبار | Variante testée | Test varyantı | One of the two or more competing versions of a page being compared in a test. |
| Control condition | شرط الضبط | Condition de contrôle | Kontrol koşulu | Keeping every factor except the one variable being tested the same between the two versions, so the comparison isolates that single variable. |
| Confounding factor | عامل مربك | Facteur confondant | Karıştırıcı etken | Any other change happening at the same time as a test that could also explain a change in the result. |
| Traffic split | تقسيم الزيارات | Répartition du trafic | Trafik bölünmesi | The way incoming visitors are divided between the control version and the test variant, usually as close to equal as possible. |
| Live page | الصفحة المباشرة | Page en direct | Canlı sayfa | The version of a page a visitor is actually shown during a test, whether the control or the test variant, decided by the traffic split. |
| Test period | فترة الاختبار | Période de test | Test dönemi | The fixed length of time a test runs before its result is read, agreed in advance and not extended or shortened partway through. |
Next you will take a completed test's own two results and decide whether the gap between them is large enough to trust, or small enough to be ordinary noise. Back to the course map