Sample data
Step 1: State hypotheses
We want to know if B performs better than A.
\(H_0: p_A = p_B\) (null hypothesis, which we don’t expect/want to disprove)
\(H_1: p_B > p_A\) (alternative hypothesis)
Step 2: Pooled proportion
Because under \(H_0\), both groups have the same conversion rate.
Total clicks = 3 + 6 = 9 Total users = 20
\(p_{\text{pooled}} = \frac{9}{20}=0.45\)
Step 3: Standard error
\(SE = \sqrt{p(1-p)\left(\frac{1}{n_A}+\frac{1}{n_B}\right)}\)
\(SE = \sqrt{0.45 \times 0.55 \times \left(\frac{1}{10}+\frac{1}{10}\right)}\)
\(= \sqrt{0.2475 \times 0.2} = \sqrt{0.0495} \approx 0.2225\)
Step 4: z-statistic
\(z = \frac{p_B - p_A}{SE} = \frac{0.60 - 0.30}{0.2225} = \frac{0.30}{0.2225} \approx 1.35\)
Step 5: p-value
For a one-sided test, z = 1.35 → p ≈ 0.088.
Interpretation
- p-value = 0.088 > 0.05
- We fail to reject (H_0).
- Evidence is suggestive but not strong enough to conclude that the red button performs better.
Business interpretation:
Even though conversion increased from 30% → 60%, the sample is very small (only 10 per group), so the difference is not statistically significant at α = 0.05.
What we can say
- B looks promising: a +100% relative lift in conversions.
- But the sample size is too small to be confident.
- A larger A/B test should be run.