Maven

Accelerating Innovation with AB Testing

Intermediate

Learn from a world-leading expert how to design and analyze trustworthy A/B tests to evaluate ideas, integrate AI/ML, and grow your business. This course will empower your organization to be data-driven and innovative, providing you with the skills to design and interpret experiments effectively.

A/B testing
Experiment design
Data analysis
AI/ML integration
Causal inference

Overview

This course covers the design and analysis of trustworthy A/B tests, integrating AI/ML, and growing your business through data-driven decision-making. You will learn from real examples and stories from Microsoft, Amazon, and Airbnb, understanding the challenges and pitfalls of experimentation. The course also covers cultural challenges, institutional memory, observational causal studies, and more technical topics like Bayesian vs. Frequentist methods.

English
course language
Professional Certification
upon course completion

Who is this course for?

Data science managers and scientists

Will be able to design and interpret the experiment results in a trustworthy manner.

Program managers

Focused on growth, revenue, conversions, and prioritization will understand how to provide the organization with robust clear metrics.

Engineering leaders

Will be able to make the organizations more data-driven and efficient with fewer severe incidents through A/B tests.

This course will teach you how to design and analyze trustworthy A/B tests, integrate AI/ML, and grow your business. It is ideal for data science managers, program managers, and engineering leaders who want to make their organizations more data-driven and efficient. By taking this course, you will gain practical knowledge and hands-on experience in A/B testing, helping you achieve your goals and advance your career.

Pre-Requisites

1 / 3

Basic understanding of data science concepts
Familiarity with statistical methods
Experience with A/B testing or experimentation

What will you learn?

Session 1: Introduction, interesting examples, organizational tenets

Introduction, interesting experiments (3 polls + 3 without), organizational tenets, advantages and limitations of controlled experiments.

Session 2: End-to-end example, metrics, and the OEC

End-to-end example of running and analyzing an experiment (discount code), metrics, the OEC.

Session 3: Statistics, E2E ex 2, Twyman's law, Ideas, prioritization, ethics

P-values and statistical power, end-to-end experiment: speed matters, Twyman’s Law and trustworthy experimentation, generating ideas, prioritization, ethics.

Session 4: Cultural challenges, maturity, observational causal studies, pitfalls

Cultural challenge, institutional memory, maturity model, observational causal studies, pitfalls in observational studies.

Session 5: AI/ML, triggering, leakage/interference, scaling, requested topics

AI/Machine Learning model, and triggering, leakage and interference, experimentation platform and scaling experimentation, build vs. buy, challenges, requested topics.

Optional: Alumni event for Accelerating innovation with A/B Testing

Open to all alumni from all cohorts.

What learners say about this course

“Dr Kohavi answered every single question that was asked throughout the course. I definitely left feeling more confident in my AB Testing approach. Dr Kohavi also provided us with a lot of reference material we can refer to as needed (course recordings, slides, and ancillary references). I think this course was more than worth it's price.”
A.J
Data Science Lead, Salesforce
“The A/B Online Controlled Experiments course was excellent. Kohavi explained everything clearly, from designing to analyzing A/B tests. The hands-on projects were great for applying what we learned. Having so much documentation will be very helpful for future reference. Highly recommend this course!”
Anahit
Data Scientist, Picsart
“Ronny is a phenomenal instructor. His knowledge goes deep, wide and applied with real world examples. No question is left unanswered! Highly recommend”
Gil
Senior Data Scientist, StateFarm

Meet your instructor

Dr. Ronny Kohavi
Consultant, Instructor, Ron Kohavi
Dr. Ronny Kohavi is a consultant and instructor specializing in A/B testing, controlled experiments, and data-driven organizations. He is a recognized thought leader and innovator in the field, with over 42,000 citations to his published work and three machine learning papers in the top 1,000 most-cited papers in Computer Science.

New cohorts coming soon