My offline ablation said -0.19pp. The production retrain said +1.11pp. [D](reddit.com)

Last month I stopped trusting feature importance and started trusting offline ablations instead: retrain with and without the change on a held-out split, measure the delta directly. That worked exactly once. Then four changes that looked positive offline either regressed or vanished in production. We forecast pre-owned watch prices with LightGBM quantile regression (p10/p50/p90). The four: Experiment Offline Prediction Production Result Root Cause Best Offer feature Slight improvement +0.12pp regression Train/serve skew Auction data backfill Roughly neutral +0.37pp regression Unmeasured distribution shift Outlier trimming −0.19pp improvement +1.11pp regression Training population shift CatBoost encoder −0.199pp improvement ~0 (noise) Baseline instability Best Offer is the cleanest one. A lot of our eBay sold comps close via an accepted offer below the asking price, and those carry a systematic premium, so a flag for it looked like free signal sitting in the data. Problem is the sold rows have the flag and the live listings can't: you don't know a listing's offer status until it actually sells, so at inference time it's hardcoded to zero. I trained the model on something it never sees in production. The ablation can't catch that, because its train and val splits both come from the historical data where the flag is everywhere. Outlier trimming is the one that concerned me. I dropped training rows where |log(sold_price / family_median)| > 0.8, the usual move for damaged pieces and mistitled lots in the tails. Four seeds, clean −0.19pp, won every seed. Shipped it to the retrain and got +1.11pp. I burned an afternoon looking for a harness bug before I accepted there wasn't one. Here's the part I had wrong at first: the validation rows were never trimmed, so this isn't "you deleted the hard cases." The harness graded the change on a held-out window that had drifted out of sync with the cohort production actually scores against. A model that had stopped learning from the tails looked better on that window and worse on the live stream, which still has the tails. Pinning the harness split to production's split date was the fix. So the claim I'd actually defend, narrower than I first wrote it: a standard held-out ablation can systematically overestimate a change when that change alters the training population, not just the features. Drop a column and the rows are the same, so the held-out slice tells you how it generalizes. Drop or reweight rows and you've moved the training distribution while still grading on a slice that may not represent where the model now fails. The bias runs toward optimism, exactly where the production downside is biggest. What actually catches it is comparing against production, not against your own offline number. Every retrain trains a candidate and grades it against the live incumbent on a verified-sold cohort; more than 0.30pp worse and it doesn't get promoted (that's 3 to 6 sigma against our ~0.05 to 0.10pp seed noise). It's stratified by confidence band rather than the raw headline, because we'd already been bitten by a Simpson's-paradox version where a shift in band mix dragged the headline while every individual band was improving. The outlier change is exactly what it's for: offline said ship, the gate measured +1.11pp against production and refused it on its own. Full writeup with the per-experiment detail, the thread-pinning fix behind the CatBoost "improvement," and the gate code: https://flyback.ai/engineering/ablation-said-ship submitted by /u/Nj-yeti [link] [Kommentare]