ABS Challenge Policy
A break-even card by count and game state on the state-dependent cost of a failed challenge, with role success rates against it and a first-half → second-half forward test.
What it predicts
Break-even = cost of a failed challenge ÷ (gain of an overturn + that cost). Gain: the overturn’s run value in that count. Cost: the challenge a failure burns, which depends on challenges left and innings left.
Inputs
- Every 2026 MLB challenge (Baseball Savant): count, inning, challenger, outcome, the overturn’s run value and Savant’s state-dependent cost of each failure; ordered within each game from Statcast to count the challenges left
- Team-games with no challenges too, for the option-value check
- Triple-A challenges 2025–26 for the catcher repeatability test
Method
- Failure cost by challenges left (two or one) × inning, from Savant’s per-failure cost; checked against an independent option value from team-games (what clubs with two left went on to win vs. clubs with one)
- Three game states: two left before the 7th, the last one before the 7th, the 7th inning on; the costliest inning in each row is flagged where it changes the call
- Success rates by role × count × state, shrunk toward the role’s count rate (empirical Bayes), shifted by the club’s own accuracy regressed by how much club accuracy is real across the 30 clubs
- A cell’s 80% range comes from refitting every step on resamples of whole club-games (the prior’s own uncertainty included), plus the club shift’s. A cell is a call when that range clears the break-even, and a toss-up otherwise; the same test runs inning by inning before the 7th. No individual green lists: batter and catcher repeatability (odd vs. even challenges, catchers pooled with Triple-A) can’t be told from zero
- Forward test: every input (ranges included) fit before July 1, and the card scored as published (early-inning holds, toss-ups kept, pitchers’ challenges left as made) on the challenges made after it, at the state cost and at the flat 0.21 for comparison. The old flat-cost grid is scored the same way with its late-game exception (“late in a close game with challenges unused”, read as no hold from the 7th with no challenge lost and the score within 2), and under other readings of it
- Roles and players from one pitch-level log, by the club that made the challenge, so player tables add to role rows
- Where a club won and lost (zone edge, pitch group): each split against the league rate for the same role, count and state, and against the rest of the club’s own challenges; a split stands out only on the second test, past a six-test line
How well it works
Computed by the pipeline from the data on this site
- Challenges modeled (2026 MLB)
- 10,557
- Forward test: rules fit on the 5,330 challenges before Jul 1, scored on the 5,227 after it (runs per club over a season, 80% range)
- This card +0.01 runs (−0.04 to +0.05); it held 20 of the 5,227 The first count grid −9.4 runs (−9.9 to −8.9); it held 2,777 Against how clubs already challenged, the card’s value is inside the noise; it beats the first grid by 9.4 runs a club-season. Its value is the lost runs it avoids.
- Pittsburgh retro-score (in-sample, whole 2026 season)
- This card would have held 0 of 355 challenges. The first grid, with its late-game exception, would have held 194, worth −6.9 runs at the state cost and +9.3 at a flat 0.21 runs a failed challenge: the verdict flips on the cost.
- Break-even by count
- 0-0 32%0-1 27%0-2 16%1-0 23%1-1 23%1-2 13%2-0 16%2-1 17%2-2 10%3-0 15%3-1 12%3-2 6%
Data: every 2026 MLB challenge (10,557, Baseball Savant), ordered within each game from Statcast; Triple-A challenges 2025–26 for the repeatability test.
What it can't see
- The state cost is calibrated to how clubs challenged in 2026. If clubs challenged far more, a challenge in hand would be worth more and the bars would rise.
- A cell compares the average challenge made in that spot with the break-even. Whether challenging more would pay depends on how sure players were on calls they let go, which isn’t observed.
- Overturns are deterministic given Hawk-Eye location; what’s uncertain is the player’s perception, which isn’t observed.
- One season of MLB challenges; the extra challenge in extra innings is inferred from the log, not from a rulebook.
- The ranges cover the success rates, not the bars: the cost and gain behind each break-even are treated as known.
How we'd measure impact
- Pittsburgh challenge success rate vs. 2026, and accuracy runs vs. an average challenger
- Share of challenges inside the card’s bars (compliance)