Discrimination: ROC, Precision-Recall, Gains, and Lift

Discrimination evaluates how well a model separates positive instances (events) from negative instances (non-events). rtichoke groups discrimination visualizations into one coherent conceptual family:

1. Receiver Operating Characteristic (ROC) Curve: Sensitivity vs. 1 - Specificity.
2. Precision-Recall (PR) Curve: Precision (PPV) vs. Recall (Sensitivity).
3. Gains Curve: Cumulative percentage of positive outcomes vs. percentage of population targeted.
4. Lift Curve: Factor by which model predictions improve detection over random targeting.

For methodological comparisons and statistical theory on when to prefer PR over ROC curves, see the rtichoke blog.

---

ROC Curves

ROC curves plot Sensitivity (True Positive Rate) against $1 - \text{Specificity}$ (False Positive Rate) across all probability thresholds.

library(rtichoke)

# Directly create an interactive ROC curve
create_roc_curve(
  probs = list(
    "Good Model"   = example_dat$estimated_probabilities,
    "Bad Model"    = example_dat$bad_model,
    "Random Guess" = example_dat$random_guess
  ),
  reals = list(example_dat$outcome)
)

Reference Lines

The diagonal identity line represents a uninformative classifier ($AUC = 0.50$). Models bowing toward the top-left indicate stronger discrimination.

---

Precision-Recall (PR) Curves

Precision-Recall curves are particularly informative in imbalanced datasets where the positive outcome prevalence is low.

create_precision_recall_curve(
  probs = list(
    "Good Model"   = example_dat$estimated_probabilities,
    "Bad Model"    = example_dat$bad_model,
    "Random Guess" = example_dat$random_guess
  ),
  reals = list(example_dat$outcome)
)

Baseline Reference Line

Unlike ROC curves, the random baseline for a Precision-Recall curve is a horizontal line equal to the event prevalence in the population.

---

Gains Curves

Gains curves display the cumulative percentage of true positive cases captured as an increasingly larger fraction of the highest-risk population is targeted or screened.

create_gains_curve(
  probs = list(
    "Good Model" = example_dat$estimated_probabilities,
    "Bad Model"  = example_dat$bad_model
  ),
  reals = list(example_dat$outcome)
)

• Random Baseline: Diagonal line from $(0,0)$ to $(1,1)$.
• Perfect Reference: Reaches $100\%$ true positives at a population percentage equal to the prevalence.

---

Lift Curves

Lift curves measure the ratio of model performance relative to a random selection baseline at any given population quantile.

create_lift_curve(
  probs = list(
    "Good Model" = example_dat$estimated_probabilities,
    "Bad Model"  = example_dat$bad_model
  ),
  reals = list(example_dat$outcome)
)

• Baseline Reference: Horizontal line at $\text{Lift} = 1.0$.

---

Prepared Data Workflow for Discrimination

If you plan to inspect multiple discrimination curves on the same models, prepare performance data once to avoid repeated calculations:

perf_data <- prepare_performance_data(
  probs = list(
    "Good Model" = example_dat$estimated_probabilities,
    "Bad Model"  = example_dat$bad_model
  ),
  reals = list(example_dat$outcome)
)

# Render individual curves
plot_roc_curve(perf_data)
plot_precision_recall_curve(perf_data)
plot_gains_curve(perf_data)
plot_lift_curve(perf_data)