Chapter 10 Project 8: A/B Testing Framework

A/B testing is the gold standard for causal inference in product development. But running tests incorrectly — stopping too early, using underpowered samples, or ignoring guardrail metrics — leads to bad decisions. In this chapter, we build a comprehensive A/B testing framework with sample size calculation, sequential testing, and Bayesian analysis.

10.1 Business Context

Product teams need to:

• Determine if a new feature improves conversion rates
• Calculate required sample sizes before launching tests
• Monitor experiments without inflating false positive rates
• Make ship/no-ship decisions with confidence intervals

10.2 Sample Size Calculation

Before running a test, calculate the required sample size to detect a meaningful effect.

library(pwr)

calculate_sample_size_r <- function(
    baseline_rate,
    mde,              # Minimum detectable effect (relative)
    alpha = 0.05,
    power = 0.80,
    ratio = 1.0       # Treatment:Control allocation ratio
) {
  p1 <- baseline_rate
  p2 <- baseline_rate * (1 + mde)

  # Cohen's h effect size for proportions
  h <- ES.h(p1, p2)

  result <- pwr.2p.test(
    h = h,
    sig.level = alpha,
    power = power
  )

  # Adjust for unequal allocation
  n_per_group <- ceiling(result$n * (1 + ratio)^2 / (4 * ratio))

  list(
    n_per_variant = n_per_group,
    total_n = n_per_group * 2,
    effect_size = h,
    baseline = p1,
    expected_treatment = p2
  )
}

# Example: Detect 15% relative lift from 12% baseline
calculate_sample_size_r(0.12, 0.15)

Note: The pwr package uses Cohen’s h for proportion tests. For continuous outcomes, use pwr.t.test() with Cohen’s d.

10.3 Sequential Testing

Traditional fixed-horizon tests require waiting for the full sample. Group sequential testing allows early stopping while controlling the family-wise error rate.

library(gsDesign)

design_sequential_test <- function(
    alpha = 0.05,
    beta = 0.20,
    k = 5              # Number of interim analyses
) {
  gsDesign(
    k = k,
    test.type = 2,     # Two-sided test
    alpha = alpha,
    beta = beta,
    sfu = "OF",        # O'Brien-Fleming spending function
    sfl = "OF"
  )
}

The O’Brien-Fleming spending function is conservative early in the trial, requiring stronger evidence for early stopping. This protects against false positives from peeking.

10.4 Experiment Analysis

library(infer)
library(broom)

analyze_ab_test <- function(results_df) {
  control <- results_df %>% filter(variant == "control")
  treatment <- results_df %>% filter(variant == "treatment")

  # Conversion rates
  control_conv <- mean(control$converted)
  treatment_conv <- mean(treatment$converted)
  relative_lift <- (treatment_conv - control_conv) / control_conv

  # Chi-square test for independence
  contingency <- table(results_df$variant, results_df$converted)
  chi_test <- chisq.test(contingency)

  # Confidence interval for lift
  se <- sqrt(
    control_conv * (1 - control_conv) / nrow(control) +
      treatment_conv * (1 - treatment_conv) / nrow(treatment)
  )
  ci_lower <- relative_lift - 1.96 * se / control_conv
  ci_upper <- relative_lift + 1.96 * se / control_conv

  # Revenue analysis (continuous outcome)
  revenue_test <- t.test(control$revenue, treatment$revenue)
  revenue_lift <- (mean(treatment$revenue) - mean(control$revenue)) / 
    mean(control$revenue)

  # Guardrail metrics: ensure no harm to other KPIs
  list(
    control_conversion = scales::percent(control_conv, accuracy = 0.01),
    treatment_conversion = scales::percent(treatment_conv, accuracy = 0.01),
    relative_lift = scales::percent(relative_lift, accuracy = 0.01),
    ci_95 = paste0("[", scales::percent(ci_lower, accuracy = 0.01), ", ", 
                   scales::percent(ci_upper, accuracy = 0.01), "]"),
    p_value = format(chi_test$p.value, digits = 4),
    significant = chi_test$p.value < 0.05,
    revenue_lift = scales::percent(revenue_lift, accuracy = 0.01),
    revenue_p_value = format(revenue_test$p.value, digits = 4),
    recommendation = ifelse(
      chi_test$p.value < 0.05 && relative_lift > 0 && revenue_test$p.value > 0.05,
      "SHIP",
      "DO NOT SHIP"
    )
  )
}

Important: The revenue_test$p.value > 0.05 guardrail ensures we don’t ship features that improve conversion but hurt revenue (e.g., by discounting too aggressively).

10.5 Bayesian A/B Testing

Bayesian methods provide intuitive probability statements: “There is a 94% probability that the treatment is better than control.”

library(bayesAB)

bayesian_ab_test <- function(control_conversions, control_trials,
                             treatment_conversions, treatment_trials,
                             n_samples = 1e5) {

  AB1 <- bayesTest(
    c(control_conversions, control_trials - control_conversions),
    c(treatment_conversions, treatment_trials - treatment_conversions),
    priors = c('alpha' = 1, 'beta' = 1),  # Uniform prior
    n_samp = n_samples,
    distribution = 'bernoulliC'
  )

  list(
    prob_treatment_better = AB1$posteriors$Probability$prob,
    expected_lift = AB1$posteriors$Probability$lift,
    credible_interval = quantile(
      AB1$posteriors$Probability$posterior,
      c(0.025, 0.975)
    )
  )
}

10.6 Experiment Dashboard

# Shiny app for real-time experiment monitoring
experiment_ui <- dashboardPage(
  dashboardHeader(title = "A/B Test Monitor"),
  dashboardSidebar(
    selectInput("experiment_id", "Experiment", choices = get_experiments()),
    actionButton("refresh", "Refresh Data", icon = icon("sync"))
  ),
  dashboardBody(
    fluidRow(
      valueBoxOutput("control_rate"),
      valueBoxOutput("treatment_rate"),
      valueBoxOutput("lift")
    ),
    fluidRow(
      box(plotlyOutput("conversion_trend"), width = 8),
      box(tableOutput("metrics_table"), width = 4)
    )
  )
)

10.7 Exercises

1. Implement a multi-armed bandit algorithm (Thompson Sampling) for dynamic traffic allocation.
2. Add stratification variables (device type, geo) to the analysis and check for interaction effects.
3. Build a function that calculates the “probability of being best” for each variant in a multi-variant test.
4. Implement a peeking-adjusted p-value using the gsDesign package for a test with weekly looks.

---

In the next chapter, we’ll tie everything together with production deployment patterns, monitoring, and MLOps best practices.