Test and Learn - Statistical Comparison of Two Groups in Business Applications

Test and learn also known as A/B testing in business applications is the comparison of two groups that aims for the exploration and assessment of new business ideas and digital approaches before rolling them out across the enterprise. In essence, it is a practice of hypothesis testing of one change at a time; leveraging such sampling procedures as independent sample design or matched sample design and inferential statistics to determine statistical significance. Independent sample design (also known as parallel-group study) involves subjects (experimental units) that will be randomly assigned to one of the two study groups at the start of the study. Example is the comparison of a control group to a treatment group. Randomization provides the basis for unbiased comparisons. Matched pairs consist of one random sample of subjects. Each subject first uses one method and then uses the other method. That is, the same subjects are observed twice. The order of the two methods is assigned randomly to the subjects, with some subjects starting with one method and others starting with the other method. One advantage of using the same subjects is that experimental variability is less than for the independent sample design case.

Considerations For Test and Learn Design: It’s not about testing – It’s about learning

1. Frame the business problem: Define a specific problem, and generate ideas for testing. Designing a good A/B test and collecting and analyzing trustworthy data starts with asking a SMART question. Use the SMART-approach in framing a problem for test and learn.
• Specific
• Measurable
• Actionable
• Relevant to the business, and
• Timely problem
2. Prioritize issues/ideas and identify your goal.
3. Pick one variable to test at a time. Knowing what you can control normally relates to the specific business question under consideration. Please note that accounting for confounding and nuisance factors along with randomization are essential aspects of test and learn design. Subject matter experts and data analytics professionals may have information on covariates. This information can be used to balance study group assignment with respect to covariates prior to randomization.
4. Develop a hypothesis to test and define the primary test metrics and/or secondary test metrics. The test should have a defined outcome that is meaningful, reliable, and measurable over a relatively short duration. Decide on an outcome that is an indicator of what you want to know about in your long-term goal.
5. Decide on the sampling design and compute sample size. It is of vital importance to use the most appropriate comparison test for making unbiased decisions. Prospective determination of the appropriate sample size is the very basic up-front activity in any test and learn. Computing sample size is not only a statistical requirement, but it's also a cost issue. Experimental data using a small sample size would not provide convincing evidence due to inadequate statistical power, while a sample size that is too large can lead to wasted time, resources and additional cost.
6. Launch the study with representative samples and collect data. For effective test and learn, it is important that the study is appropriate and relevant to the customer population to which evidence-based business decisions will be applied. Give ample time for the data flow.
7. Analyze the data and synthesize findings. Statistical analysis with respect to A/B testing can typically be categorized into two basic groups: descriptive statistics and statistical inference. In descriptive statistics, basic measures such as mean, median and distributions are used to explore and summarize the data for a treatment vs a control group, as an example. The logical next step is a statistical inference whether the observed differences are true or just a chance difference.
8. Socialize insights and recommendations in a way that demonstrates value. Test and learn would only be complete, successful and impactful when insights can be understood and used by business lines and stakeholders.
9. Consider MMR in your presentation. MMR being
• Meaningful: Analysts need to clearly state why the study is important once more to business leaders and stakeholders. A task that should be ironed out during the pre-planning stages of the test and learn.
• Measurable: A metrics that is relevant to the business and was clearly defined and agreed upon by the analytics team. This too is something that needed to have been decided during the sample design process.
• Reproducible: It is not credible and trustworthy analytics, if findings of an A/B test can’t be repeated. There needs to be confidence that if the study is repeated, a very similar set of conclusions will be achieved.

Related Articles:

Do-it-yourself web-apps for:

1. Sample size calculation,
2. Descriptive statistics, creating Pearson correlation matrix,
3. Inferential statistics of A/B test designs,
4. Supervised machine learning: feature selection, and
5. Supervised machine learning: classifier predictive model.

In-memory processing. No input data file is saved!

Sign up, indicate the location of your input file and click Run!