At small-scale, intuition and research trumps statistical analysis.
|
Scooped by
Guillaume Decugis
onto Ideas for entrepreneurs May 17, 2016 3:49 PM
|

Get Started for FREE
Sign up with Facebook Sign up with X
I don't have a Facebook or a X account
A/B tests are statistically significant only with large numbers. You can play with Optimizely's tools (picture) or use this one by RJ Metrics or even this other one. Doing that, you'll rapidly convince yourself of a simple mathematic truth: it takes a lot of events to achieve 90% confidence - something early stage startups usually don't have.
tom Tunguz makes some great points but the other mistake I've made (and see other do) is to believe you can A/B test a whole UX. A/B tests are good to determine the impact on a direct conversion: typically whether or not landing page A converts better than landing page B. But when you start to look at indirect events like retention or activity level, it becomes very hard to tell. What's worse: it even gets polluted when you have a lot of activity.
So while A/B tests are great and should be used, they're not the way to find product / market fit - interviews and qualitative research are - and they're not the magic answer to everything even at a later stage when you're trying to optimize revenue, usage and retention.