Today's Puzzle

These 2 lines show large groups of teachers in a study. Before I share what the graph shows, what is your gut reaction to what you see?

My reaction is: each line goes up steeply at the beginning, but then levels off.

Okay, new graph below. These are 3 groups of teachers. What is your gut reaction?
My reaction is: those 3 groups of teachers seem very similar.

Now let me explain. Both graphs are from a 2010 paper by Jennifer Rice King.

1. The first graph is teacher success (as measured by value-added student test scores) based on years of experience. It looks like steep teacher growth -- they get better for some reason in Year 2 -- and then levels off in the later years.

This is what it looks like in toto:

An obvious explanation is experience: teachers learn by trial and error. Another possible explanation would be training/coaching. This may be less likely because separate studies of training/coaching tend to show no affect on student achievement (although sometimes it works).

Anyway, we see a decent-size jump from rookie year to second year of teaching, and then some smaller ones.

2. The second graph shows how 3 different types of teachers do. Three bell curves. All look quite similar. The curves are: rookie teachers, second and third year teachers, and more experienced teachers.

But how can this graph be true?

How can these rookie teachers, and second and third year teachers, and veteran teachers -- all do about the same (the "upside u" for each type of teacher overlaps almost perfectly)...when on the first graph, there was a big jump up after rookie year?

The reason is the scale of the y-axis.

In the first graph, it makes a year of experience look like it matters a lot. But it's really 0.04 standard deviations.

That 0.04 is "statistically significant" in that it's been measured well enough to believe this is not due to chance error.

But it's not necessarily "significant" in the sense of "big."

I asked MTR friend and Harvard doctoral student Matt Kraft to translate that very roughly into normal people speak. Which he did reluctantly, because he's a quant. He said in this case, a 0.04 gain might move a teacher from the 50th percentile to the 51st percentile overall.

(If I understood him correctly. Which itself is probably just a 40% proposition).

3. So teachers, on average, do get better with experience. But it seems like: small amounts better. And really small amounts of improvement after the first few years on the job.

The Malcolm Gladwell maxim does not seem to apply. He argues that 10,000 hours of activities often leads to expertise. A teacher usually reaches 10,000 hours of teaching in Year 14 (roughly 700 hours per year).

The evidence is a 14-year-vet does not seem to get kids to learn any more (or, possibly a tiny teensy bit more) than a 3-year-veteran.

It would be nice -- for teachers and kids -- if we could figure out how to make these jumps bigger.

Why are teacher improvements so small?

My man Cormac on the West Coast sent me this article.

You could call it the O.K. plateau.

Psychologists used to think that O.K. plateaus marked the upper bounds of innate ability. In his 1869 book “Hereditary Genius,” Sir Francis Galton argued that a person could improve at mental and physical activities until he hit a wall, which “he cannot by any education or exertion overpass.”

In other words, the best we can do is simply the best we can do.

But Ericsson and his colleagues have found over and over again that with the right kind of effort, that’s rarely the case. They believe that Galton’s wall often has much less to do with our innate limits than with what we consider an acceptable level of performance.

Is that what often happens to teachers?

Is there an OK plateau which is really about our own expectations of acceptable?