Randomized Trials - for Everything?

A. I enjoyed this essay by Rebecca Tuhus-Dubrow. It was in the Boston Globe "Ideas" section. She writes:

Randomly assign a representative sample of the population — say, 10,000 taxpayers — a lower tax rate, and see what happens. Did these Americans, on average, behave any differently than their counterparts? Did they work longer hours or more jobs, start more businesses, hire more employees?

In other words, test government policies using the same technique — randomized controlled trials — used to test new drugs. A growing chorus of legal scholars, economists, and political scientists believes that such trials should be conducted to evaluate a wide range of laws: gun control, safety and environmental regulations, election reforms, securities rules, and many others. And some believe that we are ethically obligated to do this, because laws affect our lives so pervasively. Understanding the true costs and benefits of legislation, they say, is essential to making good policy — and we may know much less about our own laws than we think.

“The randomized experiment is kind of the gold standard in medicine and social science,” says Ian Ayres, a Yale law professor and economist who advances the idea of the tax experiment in a forthcoming paper. “We should use that same tool to inform us whether laws work.”

Tuhus-Dubrow goes on to cite MIT economist Michael Greenstone.

An aside. Recently I mentioned that math teacher Ryan Holmes has an uncanny jumper from 17 feet. Greenstone is actually even better. He has a ridonkulous jumper from 20 feet (the 3-point line), like: insane, rain man level.

In fact, in the world of people I've mentioned recently on this blog:

Three-point shot (20 feet):

1. Greenstone 2. Ross Trudeau (of Teaching For The Win, now teaches English at KIPP King)

17-foot jumper:

1. Orin (director of our teacher residency) 2. Ryan

Just thought you'd want to know. Now back to the essay:

Proponents of this idea don’t claim it’s a panacea. They do believe randomized controlled trials are, if imperfect, the best way we have of generating empirical data. In his recent essay, Greenstone argued for a “new era in regulatory reform.”

The first era, he writes, was that of the New Deal and the Great Society, when the focus was on well-meaning efforts to remedy social problems. In Greenstone’s view, the effort seemed to count more than the results, with little emphasis on follow-up and evaluation.

Then, in a backlash, came the second era, under the Reagan administration, when government was recast as the problem.

Now, Greenstone believes, we must instigate a third era, in which government is neither demonized nor valorized, when results are measured meticulously and count for more than good intentions.

Totally agree!

B. Politically, however, it will be tough to win Greenstone's Third Way. We react first based on values, not evidence.

For example, what do you think about counseling of children as a general matter?

A good tool, under-deployed in society? Or occasionally useful but often just a crutch, lots of blathering and indulgence, resources that could better be used in another way?

Well, here was an old policy land">study on this topic.

In the 1930s, the Cambridge/Somerville Youth Study, designed to reduce delinquency, randomly assigned over 500 boys to either a treatment group, which received visits from a counselor and other services, or a control group, which received neither. A 30-year follow-up found that the intervention had not diminished criminality and, strangely, seemed to have slightly exacerbated it.

Now if your initial view was the former, your response is easy: they must have done it wrong. And if your initial view was the latter, your response is easy: see? That's what I told you.

Here is a more recent example, from the NY Times.

The New York study involves monitoring 400 households that sought Homebase help between June and August. Two hundred were given the program’s services, and 200 were not. Those denied help by Homebase were given the names of other agencies — among them H.R.A. Job Centers, Housing Court Answers and Eviction Intervention Services — from which they could seek assistance.

Advocates for the homeless said they were puzzled about why the trial was necessary, since the city proclaimed the Homebase program as “highly successful” in the September 2010 Mayor’s Management Report, saying that over 90 percent of families that received help from Homebase did not end up in homeless shelters. One critic of the trial, Councilwoman Annabel Palma, is holding a General Welfare Committee hearing about the program on Thursday.

So you see the obstacles to achieving Greenstone World. They already believe in the values: we should help these people. Don't get all cute and measure carefully.

And when you don't want that to happen, there's ol' reliable. Attack randomized studies.

“I don’t think homeless people in our time, or in any time, should be treated like lab rats,” Ms. Palma said.

Now there is a Greenstone-like guy who makes the counter-argument. Here he is:

But Seth Diamond, commissioner of the Homeless Services Department, said that just because 90 percent of the families helped by Homebase stayed out of shelters did not mean it was Homebase that kept families in their homes. People who sought out Homebase might be resourceful to begin with, he said, and adept at patching together various means of housing help.

The department, Mr. Diamond added, had to cut $20 million from its budget in November, and federal stimulus money for Homebase will end in July 2012.

“This is about putting emotions aside,” he said. “When you’re making decisions about millions of dollars and thousands of people’s lives, you have to do this on data, and that is what this is about.”

The department is paying $577,000 for the study, which is being administered by the City University of New York along with the research firm Abt Associates, based in Cambridge, Mass. The firm’s institutional review board concluded that the study was ethical for several reasons, said Mary Maguire, a spokeswoman for Abt: because it was not an entitlement, meaning it was not available to everyone; because it could not serve all of the people who applied for it; and because the control group had access to other services.

C. So Where Does That Take Us With Research on Teacher Moves? I've written before about research on teacher choices. As in: there's little that's legit. Three million American teachers make scores of decisions each day. No one knows much about those decisions. Generally there are directly competing ideas of "what works," backed by weak research, called "best practices."

The bad news: for a variety of reasons, K-12 is likely to remain that way for a while. Perceived best practices being argued by people driven by philosophy, not practical experience.

The good news: the mathematization of everything. A societal trend is studying more things with numbers. Greenstone and others leading the charge. Again: There are big limits to randomized trials. I will blog soon about some interesting stories out of the medical world, questioning certain trials.

But trials remind me of Churchill's quote about democracy. "The worst form of government...except all the others that have been tried."

Eventually that mathematization-of-everything trend will hit the study of teacher decision-making. And a new generation of researchers will explore teacher moves, not to replace observation and judgment, but to supplement it and challenge conventional wisdom.

Tomorrow I'll tell you about one of those young guns: U-Mass doctoral student Becky Allen.