Indeed.com

Simulation: When Does the Correction Become Unstable?

The correction depends on how well the classifier separates positive from negative records. We simulated three classifiers with the same true prevalence, Pₜ = 30%, but different values of D = Se + Sp − 1: Strong: Se = Sp = 0.90, giving D = 0.80 Moderate: Se = Sp = 0.75, giving D = 0.50 Weak: Se = Sp = 0.55, giving D = 0.10 Each run classified 50,000 records.

Use a normal approximation instead?

Miller (2024) recommends analytical confidence intervals using a normal approximation: “we regard bootstrapping as unnecessary unless a complicated sampling scheme or estimator is being used”. We agree with this quote, to some extent. We think both solutions are useful and recommend both, but if a single solution is desired, across diverse estimation tasks, we strongly recommend the bootstrap.

How much does this help?

Offline model quality On an offline apply-prediction benchmark, we compared a strong baseline model with traditional features against the same model augmented with UBM embeddings. Adding more behavior streams improved ROC AUC consistently in the multi-sequence version: Apply sequence only: roughly +3.0% relative ROC AUC. Apply + click sequences: roughly +3.6%. Apply + click + impression sequences: roughly +4.9%.