Methodology

Whiffs is my attempt to implement Tom Tango's approach to evaluating projection systems.

Key posts I'm drawing from:

• Forecast Evaluations - Tom Tango responds to Nate Silver (remember when he was just a baseball nerd?)
• Testing the 2007-2010 Forecasting Systems: Official Results - Tom Tango's results from testing projection systems with his recommended methodology.
• Who's evaluating the 2011 forecasts this year?

Whiffs follows four principles:

Separate playing time from rate stats

Suppose a player is projected for 300 PA and 30 HR but actually records 600 PA and 30 HR. Was that a good HR projection?

Looking only at totals, the projection seems perfect. In reality it got lucky—underestimating playing time and overestimating power, with the errors canceling out.

Whiffs evaluates playing time and rate stats separately to avoid misleading conclusions.

Playing time is measured by plate appearances for batters and batters faced for pitchers. Other stats are evaluated as rates using the most appropriate denominator.

For example, strikeouts and walks are per plate appearance or batter faced; batted-ball outcomes are per ball in play. This keeps each stat in its most meaningful context.

Adjust for league context

Here's an example based on one from Tom Tango:

One system projects .340 wOBA for a player in a .330 league; another projects .330 in a .330 league. The player posts .338 wOBA in a .338 league. Which was more accurate?

Because the player was league average, the second projection is more useful—even though its raw number was farther off.

Whiffs subtracts the league average from each projected rate before computing error.

In this example, the projected league-adjusted wOBAs are +.010 and .000. The second better predicts the actual league-adjusted wOBA (.000).

Weight by playing time

An error for a 100-PA player matters less than the same error for a 600-PA player. Systems shouldn't be penalized equally for misses on marginal and regular players.

Whiffs weights each error by the player's actual PA or BF to reflect real-world impact.

Include unprojected players at league average

Limiting evaluation to players projected by every system biases toward the easiest cases. We want to reward systems that attempt harder projections.

If a player is missing from a projection system, Whiffs assigns them 1 PA/BF for playing-time evaluation and league average for rate stats.