Eyes Versus Numbers

Look at these numbers. • The average number of X is down from 15.2 in 2011, versus 14.7 in 2012.

• The average number of Y is 75 this year, 74 last year.

• Z is 23 this year and 21 in 2011.

• ZZ is up from 48 to 51.

So what do you think? Is there a large shift? Or do the numbers look about the same.

To me, they look about the same.

Now I'll tell you what the numbers are: NFL officiating.

• The average number of penalties per game is down from 15.2 to 14.7.

• On player-safety calls, such as roughing the passer; unnecessary roughness, including hitting defenseless players; and face-mask or horse-collar violations, the calls are nearly even: 75 this year, 74 last.

• Instant replay reviews are way up, an increase of 16. But the percentage of reversals is way down: 23 this year out of 62 as opposed to 21 of 46 in 2011.

• Defensive pass interference and illegal contact penalties are up, but only from 48 to 51, surprising because of the hubbub raised on the airwaves about the lack of such calls.

I find this interesting.

NFL referees have been locked out by the league. So this year there are replacement refs.

The narrative among players, coaches, and fans is that the replacement refs are missing a lot of calls. But the replacement refs seem to issue penalties at roughly the same rate.

In fact, on this metric -- how many instant judgements are overturned later by video replay -- the replacement refs seem to be doing better than the regular refs. Their calls are reversed less often.

So who is right? The numbers or the observers?

This is relevant to whether teachers should be measured, in part, based on student gains on tests. The Gates-funded MET studies showed that trained adult observers -- using various scorecards -- were pretty bad at "evaluating" teachers.

Or more specifically, these observers struggle to tell the difference between teachers whose kids do very well on the exams, and teachers whose kids do quite badly. The numbers and the "eyes" don't line up well.