Stats Versus Stories

From the NY Times:

There is a tension between stories and statistics, and one under-appreciated contrast between them is simply the mindset with which we approach them. In listening to stories we tend to suspend disbelief in order to be entertained, whereas in evaluating statistics we generally have an opposite inclination to suspend belief in order not to be beguiled.

Consider how this applies to teachers in a "data-driven school." What if the data challenges your existing narrative about how things are going?

This human tendency (disbelieve stats) combines with another human tendency (resistance to change). Then those two combine with a third human tendency (avoid work when you already feel maxed out).

That is, data invariably shows gaps. To address the gaps, a teacher would either need to change or to do more -- or both.

Guess what the books about using data focus on? Not surprisingly, many of the chapters aren't about data at all. They're about psychology, like the excellent Driven By Data. Authors seek to help teachers and principals defeat this Cerebus (the 3-headed monster) of humanity that makes "using data" hard.

* * *

In 2007 I joined the board of a small pilot school. Since the school was fairly new, there was no MCAS (state level) data yet. At a board meeting, we were shown an internal assessment, done by a Harvard expert. Results were scary. Kids were very poor readers.

Some teachers didn't seem overly worried. Instead, they questioned the data. It did not tell the story they believed. Which was that things were going fairly well.

The flustered headmaster told me when they reviewed the data, some teachers had Googled and found various objections to that particular reading test.

a. Disbelieve the stats? Check.

b. Resistance to change? Check. Not all change, and not all teachers. Some teachers felt the change would likely be the dumb/narrow sort of "test prep" (which I agree accomplishes next to nothing). That's not what the headmaster had in mind so far as I could tell, but anyway.

c. Avoid additional work when you're already maxed out? Check.

When surveyed, teachers had cited student misbehavior as a big issue. I agreed. In my view, this made them frazzled. They agreed.

My own view was that the school would benefit from a warm-but-strict schoolwide discipline system, where they'd all enforce the same rules consistently. They did not agree. I never raised it at the Board level, but in individual conversations with some teachers, they didn't like the idea.

In my view, the totality of the reaction to scary reading data was: Let's not ignore it, but nor should we enter crisis mode.

Two years later, the school was ranked #995 out of 1,000 elementary schools on the MCAS exam. Recently, the Superintendent decided to close it, merging it with another school.