Course Curriculum

6 Sections
00

Introduction

3 Lessons 16 min
  • A Warm Welcome 3 min
  • Guess the Test: Which Variant Won? 7 min
  • The 8-Step Process for Successful A/B Tests 6 min

Resources Provided

  • 0.1_CheatSheet_AB-Testing-A-Complete-Guide (PDF)
01

Avoid the Silent Killers of Good A/B Tests

9 Lessons 61 min
  • Is A/B Testing What I Need? 4 min
  • You’re Asking Why Instead of What Happened? 6 min
  • You Have Low Traffic 2 min
  • You’re Comparing More Than Two Variables 5 min
  • Your Audience Definitions are Vague 2 min
  • You Don’t Have Concrete and Measurable Outcomes 2 min
  • You Want to Conduct Long-Term Usability Studies or Do Longitudinal Research 2 min
  • You’re Still Prototyping 2 min
  • Eight Reasons to Run an A/B Test 10 min

Practice Activities

  • Deduce Whether A/B Testing Is Appropriate for Your Own Project 26 min
02

Choose Impactful Design Ideas to Test

6 Lessons 72 min
  • How to Discover Opportunities Where A/B Testing Will Create Value 7 min
  • Three Problematic Prioritization Methods for A/B Testing (PIE, ICE, RICE) 5 min
  • The Prioritization Method I Actually Use With Clients 5 min
  • The Best A/B Testing Hypothesis Formula in the World 9 min
  • Define Your Success Metrics: Key Performance Indicators (KPIs) in A/B Testing 16 min
  • Two Examples of Excellent A/B Testing Hypotheses 4 min

Practice Activities

  • Prioritize Six Design Ideas You Could A/B Test (Instructions) 23 min
  • Watch Me Prioritize Six Design Ideas You Could A/B Test (Walkthrough) 3 min
03

Determine Sample Size and Testing Duration

4 Lessons 40 min
  • Visually Design Your Test for Success 2 min
  • Determine a Statistically Rigorous Sample Size 15 min
  • How to Run a Properly Timed A/B Test 9 min
  • Don’t Peek! It’s Far Too Risky 6 min

Practice Activities

  • Calculate an A/B Test Sample Size (Instructions) 2 min
  • Watch Me Calculate and A/B Test Sample Size (Walkthrough) 6 min
04

Select Your Testing Platform Carefully

2 Lessons 28 min
  • The Six Questions to Consider When Selecting an A/B Testing Platform 12 min
  • How to Set Up and Run Your A/B Test: A Glimpse Inside the Convert Platform 16 min
05

Analyze and Present Compelling A/B Test Results

7 Lessons 42 min
  • The Three Possible Test Outcomes 9 min
  • How to Call Your Tests 6 min
  • Statistical Significance Clearly Explained 4 min
  • Twyman’s Law and the Winner’s Curse 3 min
  • Sample Ratio Mismatch (SRM): The Invisible Threat 9 min
  • Know Your Audience and Speak Their Language 7 min
  • Wrap-Up and Conclusion 4 min

Learning Outcomes

  • Discover high-value A/B testing opportunities for your organization
  • Recognize when A/B testing is (or isn’t) the right method for your research question
  • Formulate clear, measurable hypotheses with defined KPIs
  • Set up rigorous and statistically sound A/B tests
  • Interpret test results correctly and confidently communicate them to stakeholders

Course Preview

 to watch NN/g videos

Tools Used in This Course

To complete activities in this course, you will need access to either Google Sheets or Microsoft Excel.

All other resources can be accessed through a standard PDF reader.

Turn Problems Into Solutions

Confidently identify design changes that will increase conversions

  • Uncover testing opportunities hiding in your analytics, heatmaps, and qualitative research
  • Turn existing research insights into specific, testable design changes
  • Prioritize test ideas to target the largest potential impact

Set up statistically sound A/B-testing conditions

  • Choose appropriate sample sizes and test durations
  • Avoid common mistakes that silently undermine test reliability
  • Write hypotheses that define exactly what you're changing, for whom, and what outcome you expect

Stand confidently behind your A/B testing results

  • Determine whether a test is a win, a loss, or inconclusive — and know what to do in each case
  • Spot inflated or suspicious results
  • Reframe “losses” as learning opportunities that still offer value

Frequently Asked Questions

How long will I have access to this course?
Once purchased, your access will never expire. This ensures you can learn at your own pace and return to review content as often as you'd like, whenever you need.
Will I get personalized instructor feedback on course activities?
No, personalized feedback is not provided for activities completed during the course. Most courses provide answer keys to activities that help learners reflect on their progress and understanding of course materials and examples.
What content can I download?
You can download a variety of resources provided with each course, including templates, how-to guides, cheat sheets, posters, reading lists, and more. Video lessons provided in all courses will not be available for download.
Can I access this course offline or on a mobile device?
Offline access isn't available yet, although we plan to add it. Most courses offer downloadable resources for offline reference.

For the best experience, mobile devices are not recommended.
Is this course eligible for the NNGroup UX Certification Program?
Unfortunately, self-paced courses do not count toward UX Certification. However, learners who complete the final exam will receive a “Recognition of Completion” to demonstrate their achievement.

See our Live Online courses for options that count toward UX Certification.
What can I share with my employer to show I’ve completed the training?
Upon completing a course’s final assessment, you will receive a “Recognition of Completion” showcasing your dedication to learning the topic. This document can be shared with your employer or professional network to demonstrate your new skills and commitment to professional development.