More Houston

I got some emails and calls yesterday. The New York Times did a story on the Apollo turnaround schools in Houston. Positive story, and very deservedly so, for HISD and EdLabs. However, I thought the story was framed imprecisely. Not inaccurately, but imprecisely.

I've blogged before about the work we did there in Summer 2010: dispatching a delta force named Patti, Cathryn, Tim, Christie, Erica, and Eli. Their job was to recruit, train, and deploy 200+ full-time tutors in just a few weeks. They killed it.

The Times story ascribes Apollo gains to the "copying the charter school model" broadly. But there was, to my knowledge, only one aspect of the experiment thus far which has really cleaned up.

In the Apollo schools, the really big gains were in 6th and 9th grade math only. Those were the 2,500 or so kids who got daily MATCH-style math tutoring.

So far as I'm aware, there were not really big gains in 7th, 8th, 10th grade math, or 6th, 7th, 8th, 9th, 10th grade English. None of those subjects was tutored. (There were some gains, just not biggies).

At the high-performing charter schools, the gains tend to happen roughly across all subjects in all grades, with math gains typically larger. That didn't happen here.

Now this was only Year 1 of the intervention. MATCH wasn't tested in our first year of operation (2001), but if we had been, the results would have been terrible. So we'll learn more this coming year as the Apollo turnaround effort both includes several additional Houston schools, and launches in Denver.

I felt a little bad for our troopers who put in 16-hour days in Houston. Wish they'd gotten recognition. Don't worry peeps. It happens. When I wrote a few articles, back in the day, for New York Magazine, sometimes I'd simplify a story to get a "cleaner" narrative. Certainly editors press for it.

My only regret is readers may take the wrong policy prescription. It's not "copy no excuses charters" per se. To copy the no excuses model, you gotta be ALL IN, 100 one-percent solutions. This is not a recipe that works if you "Sort of" follow it.

The right policy conclusion, I'd submit, is that high dosage, 2-on-1 tutoring by "mere mortal tutors" (i.e., good people where you reject 4 of 5 applicants, but not TFA alums who are Yale grads; working 40 hours a week, not 75) can lead to unusually large gains.

At least in math....

* * *

Part 2: Matt's Study

I've blogged before about this issue: math gains in good charters typically much bigger than English gains.

Commenter Tom pointed out

One problem with our testing regime is that it makes math look like half of the goal of a school. Math as a discipline is constructed and taught differently than all others. If your school is constructed to optimize math instruction, you’re not half way there; you’re more like 1/5th of the way.

I agree. English is typically more important to a kid, whether for college success or the ability to compete in the labor market.

Enter tutoring. Can tutoring help bolster English scores?

Our experience at MATCH high schools was that a high-dosage weekend tutoring program, which we deployed in 2003 and 2004, seemed to help a lot in math, but not in English. That is, our math scores went way up in those years. But not our English scores. Causal? Unclear. An alternative explanation would be our math teachers outperformed our English teachers. But that didn't seem right, based on observation.

In Fall 2004, we changed our tutor delivery model. Most tutoring was no longer provided by capable work-study students who were undergraduates, helping us on weekend. Instead, we hired full-time tutors called MATCH Corps. Our kids math gains were already quite high. But could these tutors drive English performance, too?

Yes. Or at least it seemed that way. Our English scores went up after MATCH Corps got going.

And now Harvard doctoral student Matt Kraft has essentially done the quantitative equivalent of "reconstructing the scene of the crime." (Well it wasn't a crime, it was the opposite of that, but you know what I mean).

Matt went back to 2004 and 2005. Reassembled all the data. Did a lot of fancy calculations. And the result is this paper.

It's called:

How to Make Additional Time Matter: Estimating the Causal Effect of Extending the School Day with Individualized Tutorials. Here is the graph.

Matt writes:

Two large shocks to trends in student achievement at MATCH are evident in this graph. First, there was a dramatic improvement in ELA achievement during the first year of ELT tutorials in 2005. Secondly, there was an ever larger gain in mathematics achievement in 2003, two years prior to the implementation of ELT tutorials.

This earlier increase of 0.60 standard deviations in mathematics achievement raised MATCH into the 81st percentile of schools statewide in term of average mathematics achievement. Comparatively, MATCH students were only at the 50th percentile of average ELA achievement in the same year.

The comparative effect sizes of attending a semester at MATCH in 2004, the first year that lottery data are available, confirms the differential impact of attending MATCH on ELA and mathematics scores. Using an instrumental-variables approach where the offer of enrollment at MATCH serves as an instrument for attendance, I estimate that attending a semester of MATCH in 2004 had no effect on students’ ELA achievement (-0.053, p=0.676) but provided almost a quarter of a standard deviation difference (p<.001) on achievement in mathematics per semester. Thus, given the extremely large effect size of attending MATCH on students’ achievement in mathematics before 2005, it may be that there was a far smaller margin for subsequent improvements in mathematics achievement through the addition of ELT tutorials as compared to ELA.

A review of the changes in academic programing at MATCH reveals that the large gains in mathematics achievement evident in 2003 coincided with the first major expansion of instructional time at the school. In 2003, two years prior to the implementation of ELT tutorials, MATCH expanded and formalized its fledgling weekend tutorial program so that students were required to attend a total of 25 weekend tutoring sessions. The school hired college students through work-study programs to tutor sophomores for four hours in mathematics and English on either Friday afternoons or Saturday or Sunday mornings.

However, the effect of these weekend 25 tutoring sessions on MCAS scores cannot be isolated from other important changes that occurred concurrently at the school. MATCH was still in the very early phases of its development, and like all new schools, was undergoing rapid organizational change from year to year. In addition to adding the weekend tutorials, MATCH moved from its original cramped location at a converted synagogue to its current building and underwent several important curricular changes.

Because of these confounding factors and the absence of lottery data, I cannot attribute these gains definitively to additional time for weekend tutoring sessions, and hence decided not to make them the subject of a more detailed analysis.

A second possible explanation for the differential impact by subject of expanded learning time at MATCH is one of quality. It is possible that the rotating group of undergraduate volunteers who tutored MATCH students on the weekends were able to provide valuable assistance in mathematics, but not in English language arts. These minimally trained college volunteers might have been more successful at teaching rules and concepts from Algebra and Geometry during unstructured and isolated weekend sessions than improving reading comprehension or writing skills.

It is also possible that gains in mathematics achievement are simply easier to produce than those in ELA. Recent evaluations of TFA Corps members (Decker, Mayer, & Glazerman, 2004) and no-excuses charter schools find consistently that students of TFA teachers and no-excuses charter schools are making gains in mathematics that are two to three times as large as gains in ELA (Abdulkadiroglu et al., forthcoming; Dobbie & Fryer, 2011; Angrist et al., 2010). Thus, producing measurable gains in ELA might have only been possible with the high-quality individualized tutoring that full-time Tutor Corps members were able to provide through structured, curriculum-based and sequenced tutorial sessions.

Great work by a great young scholar.