A self-directed case study applying formal experimental design, power analysis, and significance testing to a public e-commerce A/B test dataset, built to sharpen the statistical rigor that sits alongside my product analytics work.
The scenario: an e-commerce site tested a redesigned landing page against its existing one, with the question being whether the new page converts visitors at a higher rate than the old one. This is one of the most common tests any product or growth team runs, and it's also one of the easiest to get wrong in a way that looks fine on the surface.
I took this on as a self-directed exercise using a public, widely-used e-commerce A/B testing dataset, not a proprietary company result, specifically to close a gap I'd identified in my own skill set: I've A/B tested creative and content in past roles, but I hadn't formalized the statistical rigor, power analysis, sample size planning, and significance testing, that sits underneath a properly designed experiment. This case study is that rigor, applied end to end.
Before any analysis, I checked the raw data for integrity. This dataset had a real, common problem: a subset of users assigned to the control group had somehow been shown the new page, and a subset assigned to treatment had been shown the old page, a classic randomization leak. Trusting group labels without checking them would have quietly corrupted every result downstream.
After removing the mismatched rows and one duplicate user record, the clean dataset held at 290,584 users, split almost exactly evenly between control and treatment.
This is the step that's easiest to skip and most costly to skip. Before trusting any result, I calculated the sample size this test would need to reliably detect a modest, business-relevant lift, a 2% relative improvement over the baseline conversion rate, at a standard 80% statistical power and 5% significance threshold.
That's the number that should have been the target before the test launched. The actual dataset held roughly 145,300 users per group, about half of what would be needed to reliably detect an effect this small. That single number changes how every downstream result should be read.
With the data validated and the power question on the table, I ran a two-proportion z-test comparing conversion rates between groups.
Control converted at 12.04%, treatment at 11.88%, a small, statistically insignificant difference (p = 0.19, well above the 0.05 threshold). Taken alone, that reads as "the new page didn't help." Taken alongside the power analysis, the more accurate read is: this test wasn't designed to reliably detect an effect this small in the first place.
The overlapping confidence intervals are the visual version of the same story the p-value tells: there isn't enough separation between these two groups to call this a real, confident difference.
A non-significant result doesn't mean "no effect." It means "this test couldn't distinguish a real effect from noise at this sample size." Those are different claims, and conflating them is one of the more common mistakes in how experiment results get reported up to stakeholders.
The honest recommendation here isn't "ship it" or "kill it." It's: this test needs roughly double the sample size to answer the question it was actually designed to answer. Extending the test, or accepting a larger minimum detectable effect if the business can't wait, are the two legitimate paths forward, not a coin flip on the current numbers.
I've run A/B tests before, mostly on marketing creative and content, and made calls based on the results. What I hadn't formalized was the layer underneath that: calculating the sample size a test actually needs before trusting its outcome, and being precise about the difference between "not significant" and "no effect." This case study was a deliberate, self-directed way to close that gap using real methodology rather than just reading about it.
Note: this analysis uses a public, widely-referenced e-commerce A/B testing dataset (not a proprietary employer's data or results) and was built independently as a demonstration of experimental design and statistical analysis methodology.