Pathology · 05
Sample-ratio mismatch
A 50/50 split that arrives as 48/52 with 20k users is not noise. Any lift estimated from such a test is suspect. A χ² check at p < 10⁻³ catches assignment bugs before they reach analysis.
All 20 tests in this set
| Test | Control N | Treatment N | SRM p | Flagged | Truly broken |
|---|