College Persistence: Josh checks in

MIT economist Josh Angrist wrote me a note. He suggested I read four studies of interventions on college persistence.

I've done two randomized evaluations of college retention/grade treatments on community college campuses.

One published (STAR).

One in review (OK).

Both involve a mix of $ aid and services. These look at Toronto campuses w many LEP/immigrants

See also the MDRC randomized evals of various interventions including "Learning Communities", a widely used post-secondary model and the ongoing "Opening Doors" trial for community college campuses with lots of African-American single moms

Here

And here.

The bottom line so far: some short-term gains (for a term or two). Not much in the way of lasting gains. Pretty close to nothing works for men.

Okay, Josh. I'm digging in on Paper 1, the one you published in the American Economic Journal. Just a sec while I take out my slide rule.

[If you're a No Excuses teacher but don't like wonkish experiments, skip all the way to the bottom of the blog. Read what the college students said.]

This paper reports on the Student Achievement and Retention Project (Project STAR), a randomized evaluation of academic services and incentives at one of the satellite campuses of a large Canadian university. In American terms, this institution can be thought of as a large state school, with tuition heavily subsidized. Most students are from the local area and have a common secondary school background.

For the purposes of the study, all first-year students entering in September 2005, except those with a high school grade point average (GPA) in the upper quartile, were randomly assigned to one of three treatment groups or a control group.

One treatment group was offered an array of support services including access to mentoring by upper-class students and supplemental instruction.

A second group was offered substantial cash awards, up to the equivalent of a full year’s tuition, for meeting a target GPA.

A third treatment group was offered a combination of services and incentives, an intervention that has not been looked at previously.

The control group was eligible for standard university support services but received nothing extra.

Okay. Pause. What do you think happened?

Ah, first you want to know more about the details about the treatments. What type of support? How much money?

I thought so. Here are some details:

Group 1: "An array of support services"

250 kids in the first group got "Facilitated Study Groups (FSGs)." Peer advisors were trained upper-class students in the treated students’ program of study. Advisors were meant to offer academic advice and suggestions for coping with the first year of school. Advisors e-mailed participants regularly and were available to meet at the STAR office. FSGs are class-specific sessions designed to improve students’ study habits without focusing on specific course content. The FSG model is widely used in North American colleges and universities.

I wanted a bit more info, and I found this later in the study.

Peer advisors in the STAR program had exceptional social and academic skills. They participated in a three-day training course and received continuous training and feedback from supervisors. Advisors e-mailed advisees at least biweekly as a reminder of the advisors’ availability and to solicit questions about university assimilation, scheduling, studying, and time management.

The advisors complemented existing student services by reminding advisees of the availability of STAR and non-STAR services, and by encouraging advisees to use these services and to attend tutorials and make use of faculty office hours. Advisors were also trained to identify circumstances that called for more professional help and to make appropriate referrals.

That wasn't all. They also got Facilitated Study Groups.

FSGs were voluntary, course-focused, weekly sessions open to all treated students in (Group 1 or Group 3). FSG student facilitators were previously successful in the course they were hired to facilitate. They attended the same course with their assigned STAR students and tried to help students develop reasoning skills useful for the subject they were facilitating. FSGs were designed to complement the regular content-based tutorials taught by graduate students. Rather than walking through sample problems, FSGs focus on critical thinking, note-taking, graphic organization, questioning techniques, vocabulary acquisition, and test prediction and preparation.

Group 2: Money if you get good grades in college

My parents gave us $5 for good report cards. Here the stakes were higher.

250 students (got) the opportunity to win merit scholarships for solid but not necessarily top grades in the first year. Award targets were set based on high school grade quartiles.

Participants from the lowest grade quartile were offered $5,000, roughly a year’s tuition, for a B average (a GPA of 3.0) or $1,000 for a C+ (a GPA of 2.3).

Participants from the second quartile were offered $5,000 for a B+ average or $1,000 for a B-

Award thresholds were raised to A- and B for those in the third quartile.

Upper-quartile students were not eligible to participate in the study.

Group 3: 150 kids who got everything in Group 1 and Group 2.

Control Group: 1,006 college students got nothing.

Results?

The estimated effects of SSP and SFP are consistently zero across most cuts of the data.

Women benefitted from Group 3 (the combo). An average GPA of 1.7 went up to 2.0.

Men did not benefit from anything. C'mon guys! Shape up.

This is what it looks like graphically, men v women. Remember, the goal is to shift the "control" (light color) curve "to the right."

First, the male response to these interventions. SSP = Group 1 (support). SFP = Group 2 ($$$). SFSP = Group 3 (both).

Now the female response to these interventions.

Perhaps the clearest pattern coming out of the subgroup analysis is that the SFSP effects are larger for women whose parents did not attend college.

I know that No Excuses charter schools are increasingly trying to support their alumni. It was interesting to read some of the interviews:

Some interviewees from the fellowship groups suggested that the fellowship reminders through e-mails and advisor contacts mattered quite a bit. No student said there were too many fellowship reminders or too many e-mails from advisors.

Those we talked to about the SSP focused almost exclusively on the advisor program (and not on the study skills classes which were available). Many students were pleased with their advisor interactions, or were simply glad to know that they could communicate with their advisor if needed. One male SSP students noted, “University is something different and I also wanted extra help. The peer advisor, I personally did not meet him, but he was really helpful because at every 15 days he used to e-mail me and ask me how it’s going and how I did on the test.”

Another female student said, “I thought that it was nice to know that there was someone there that you could run to and ask for help. At the beginning I just started to talk to my advisor and she did give me some advice but I found that at the time it’s hard to just all of a sudden change all of your schedules just because she said you should have good time management and you should do this and that. So I guess that you slowly start to see what your peer advisor is talking about then you try to change for the best.”

Another student never met or responded to her advisor, but nevertheless felt the advisor’s regular e-mails were helpful, “Like somebody who cared.”

I think this sums up MATCH's experience to date. Generally, our alums in college are okay with us trying to be in touch; many seem to like it. But we find what Josh finds: gentle, helpful nudges have not seemed to change academic behavior.

Charter schools and others (including a growing number of nonprofits dedicated to college success) making new investments in "alumni coordinators" to provide precisely this sort of "support" should think hard.

Same with charter networks which are fundraising to provide alums with various forms of Pell-like cash donations (see yesterday's blog).

Are there better interventions to drive college persistence? If so, what?