This Puzzle Suitable For Children Aged 110 and Over

A few folks have asked me about our teaching coaching work in New Orleans, and particularly this:

We promised the teachers (and our backer, New Schools For New Orleans) that we’ll do everything we can to get them to improve. We’ll be diligent. We’ll collaborate with them and their principals. We’re very confident that most teachers will enjoy their coaching and find our folks absolutely professional.

But we guarantee nothing in terms of measurable change. It’s quite different from our teacher residency, where we know we can deliver, and guarantee it both to the schools which hire them (money back guarantee) and to our residents (they’ll get hired in top charter schools).

I was asked: Lots of professional development (PD) providers DO claim that their stuff works. They say it's "proven." So:

a. Why wouldn't NSNO buy their PD? Why yours instead?

b. And why be so explicit about this "We guarantee nothing" business. Nobody else does it like that. Is that just a classic grumpy goldstein vibe?

My response:

a. Not sure, actually. I think NSNO believed we have a good reputation among Northeast charter schools. And they know many PD success claims are based on weak evidence. Perhaps that made it okay for us to not be in "sell mode."

b. Why emphasize a negative? Maybe it's just my negative nature. But I don't think so. We can't promise results because the puzzle is so hard. Others have been foiled. We need to make that okay -- to try a hard puzzle and possibly fail. If we can't fail, we can't learn.

Cancer researchers don't promise results. You can be very smart, have good ideas grounded in science, execute diligently, and it just turns out that X particular intervention cannot flip enough bad cells into good cells. (I fear my very smart, diligent, and attractive wife will explain to me later that bad cells to good cells is a terribly inaccurate explanation. But just roll with it).

Similarly, smart, well-meaning PD providers can provide PD to smart, hard-working teachers -- and yet fail to cause teachers to flip more low-achieving kids into higher-achievers. Respect the puzzle of teacher change.

For example, consider two recent studies. My friend Todd just sent them along. They're described in a July memo from a top think tank, called MDRC. The studies are rare in that the evaluators did a first rate job.

The impacts of both interventions were substantially less positive than had been hoped....Most critically, students of teachers who received the training scored no higher on subject-matter achievement tests than students of teachers who did not receive the training.

Moreover, in the reading study, professional development that included one-on-one coaching as well as group workshops did not lead to significantly larger impacts than professional development involving just the workshops; in the mathematics study, receiving two years of professional development did not lead to better results than receiving just one year.


The author, Janet Quint, goes on to write:

Nonexperimental analyses that were conducted as part of these two studies, along with other research, suggest that the theory of change underlying the studies is correct: professional development of the type that was delivered is associated with increased teacher knowledge and that teacher knowledge and improved instruction is associated with higher student test scores. But changes in teacher-related variables must be substantial — considerably larger than they were in these studies — to move the needle on student achievement even a small amount.

Maybe. I wasn't as convinced, actually, having read the underlying studies. It seemed like one of those things where perhaps you see what you wanna see in the data. She continues:

By themselves, the findings of the two studies do not mean that professional development efforts cannot work.

I agree. Teachers are not immutable. In fact, it may not be the teachers. It may be that typical PD is just very, very wrongly conceived. This is my hypothesis. I reject the notion that experienced teachers' key shortcoming is knowledge. Since most PD provides knowledge, perhaps that would explain why increased knowledge doesn't lead to better student outcomes. It wasn't the limiting factor.

Of course we'd prefer more teacher knowledge to less. And zero knowledge is a problem. But perhaps the marginal increase of knowledge doesn't help much without other mechanisms.

Instead, our work with MTR boils down to: teachers need practice with good feedback loops. That seems to be how expertise is developed in most fields. But anyway.

Quint goes there:

New thinking emphasizes a broader conception of teacher learning that involves all teachers in a school in a professional learning community that is engaged in a continuous and collegial cycle of learning, practice, reflection, and improvement.

Tony Bryk and Louis Gomez are, I think, the guys behind this "new thinking." They are smart guys. And they're taking ideas from other fields. But I'm not sure even they would say "let's bet on this" yet. An incisive recent guest blog on EdWeek, written by an actual teacher (!), describes how hard it is for he and his colleagues to be productive when they work together.

Next, at a grade-level meeting my colleagues and I take advantage of an opportunity to plan across disciplines using recent benchmark data. We're enthused to share our insights and emboldened by our well-informed plans for student growth.

Yet when we proffer student needs and solutions the conversation feels as if we're treading on shaky ground.

Personal anecdotes with students, experiences not commonly shared, are layered in as justification for certain actions and are difficult if not uncomfortable to question. If I do, I might come across as questioning a colleague's teaching abilities and/or resolve. We're all coming from a place of altruism in that we similarly want the best for students, but we can't objectively decide between whose best intentions are, in fact, best for our own students. Paradoxically, we end the meeting sidelining concrete next steps for cross-discipline teaching.

So let's not bet the farm on "professional learning communities" just yet. Various types of "colleague fear" may be limiting factors. My point isn't about this particular notion of PD. It's broader: just because we know most "normal" PD fails only suggests that something quite DIFFERENT would work better. But we shouldn't rush to choose WHICH type of different and "lock in."

Quint continues her thoughts on PD:

Randomized trials to test professional development that is reinforced within professional learning communities are in order.

Or, said my way, randomized trials to test radically different PD are in order.

* * * Our NOLA work is a randomized trial.

That is, you learn 2 ways. One is -- you ask people, in various ways, "How is it going?" That's useful. I was just perusing some reports from down there. A teacher got 3 days of coaching. She rated them all 10 out of 10. Her principal dropped by each day. He was stoked.

So the vibe is good, for sure. But will the teacher, months from now, really be better? We hope so. We're trying. She's trying.

But good vibe isn't enough. It may be enough to get more customers. But we need an external evaluator (in this case, named Matt Kraft) armed with a randomized trial to really find out.

And given that the "Teacher Change Via PD" Puzzle should probably have a warning label on it that it's not suited for anyone under 110 years old, we have to be open that Matt may one day write an evaluation like the one above about our work.