Chapter 1 Introduction to Production Data Science

Production data science is fundamentally different from exploratory data analysis. When you move a model from your laptop to a production environment, you inherit a new set of constraints, responsibilities, and engineering challenges. This chapter establishes the principles and patterns that underpin every project in this book.

1.1 The Production Gap

Many data science projects fail not because the model is inaccurate, but because the system surrounding the model is fragile. The “production gap” manifests in several ways:

Problem Symptom Solution
Data drift Model performance degrades over time Monitoring pipelines, retraining triggers
Latency Predictions take too long Feature stores, caching, model optimization
Scalability System crashes under load Connection pooling, horizontal scaling
Reproducibility Results vary between runs Version control, containerization, renv
Observability You only know it’s broken when users complain Logging, metrics, alerting

1.2 The R Production Stack

R has evolved from a statistical scripting language into a robust ecosystem for production data science. The modern R production stack consists of several layers:

1.2.1 Data Layer

  • DBI + RPostgres: Database connectivity with parameterized queries
  • dbplyr: Lazy evaluation and SQL translation
  • pool: Connection pooling for concurrent requests
  • bigrquery: BigQuery integration for cloud data warehouses

1.2.2 Modeling Layer

  • tidymodels: Unified framework for preprocessing, resampling, and tuning
  • xgboost: High-performance gradient boosting
  • prophet: Automated time series forecasting
  • DALEX + vip: Model explainability and variable importance

1.2.3 Application Layer

  • shiny + shinydashboard: Interactive web applications
  • plumber: REST API generation from R functions
  • plotly + DT: Interactive visualizations and tables

1.2.4 Orchestration Layer

  • targets: Reproducible pipeline orchestration
  • logger: Structured logging
  • blastula: Automated email reporting

1.3 Design Principles

Throughout this book, we adhere to six core design principles:

1.3.1 1. Functions Over Scripts

Every analysis is encapsulated in functions with explicit inputs and outputs. This enables testing, reuse, and composition.

# Good: Pure function with clear contract
build_churn_features <- function(df) {
  df %>% mutate(engagement_velocity = ...)
}

# Bad: Script that mutates global state
df$engagement_velocity <- ...  # Don't do this

1.3.2 2. Lazy Evaluation for Scale

When working with databases, collect data only after filtering and aggregation.

# Good: SQL pushed to database
customer_features <- tbl(con, "customers") %>%
  filter(segment == "enterprise") %>%
  group_by(region) %>%
  summarise(avg_mrr = mean(mrr)) %>%
  collect()  # Only now data moves to R

1.3.3 3. Defensive Programming

Assume inputs are malformed, databases are down, and memory is limited.

safe_execute <- function(expr, fallback = NULL) {
  tryCatch(
    expr,
    error = function(e) {
      logger::log_error("Error: {e$message}")
      fallback
    }
  )
}

1.3.4 4. Version Control Everything

Use renv to pin package versions, Git for code, and DVC or S3 for data artifacts.

1.3.5 5. Monitor Everything

Log inputs, outputs, predictions, and errors. Set up alerts for drift and failure.

1.3.6 6. Design for Failure

Build graceful degradation into every system. If the model API is down, return a cached prediction or a business rule fallback.

1.4 Project Structure

A consistent directory structure makes projects navigable and maintainable:

project-name/
├── R/                  # R source files (functions only)
├── data-raw/           # Raw data ingestion scripts
├── data/               # Processed data (gitignored)
├── models/             # Serialized model artifacts
├── reports/            # R Markdown reports
├── plumber/            # API definitions
├── shiny/              # Shiny app code
├── tests/              # Unit tests with testthat
├── _targets.R          # Pipeline definition
├── renv.lock           # Reproducible package environment
└── README.md

1.5 Getting Help

The R community is exceptionally supportive. Key resources:

  • RStudio Community: https://community.rstudio.com
  • Stack Overflow: Tag questions with r and relevant package names
  • Package documentation: Most modern R packages have excellent vignettes
  • This book’s GitHub: Open an issue for errata or questions

1.6 Exercises

  1. Set up a new R project with renv and install the core packages listed in the Preface.
  2. Create a safe_execute() wrapper that also logs warnings, not just errors.
  3. Write a function that connects to a PostgreSQL database using environment variables for credentials, with a fallback to SQLite if the connection fails.

In the next chapter, we’ll set up our development environment and establish the shared infrastructure used across all projects.