DR. BORUCH: Okay, brief history Bob just referred to, I’m going, when Lisa asked me a couple weeks back to do this she didn’t define the word implications, so I’m going to take some license here. The history to which Bob referred to includes that as the Social Science Research Council Committee on Social Experimentation in 1974 that covered an awful lot of the problems that have been discussed in broad terms at least this morning and earlier this afternoon. There was a sequel volume in 1975, Experimental Testing of Policy, in which some of the major contentions about randomized experiments were confronted. At the time we weren’t accused of being Nazi’s but we were told that randomized experiments are illegal, immoral, and not much fun. Apparently that character of objection hasn’t changed much.

Following that during the ‘70’s, Evaluating Youth Employment Programs, the Academy contributed considerably in a fine committee and volume on evaluation of youth employment programs in which randomized experiments were considered seriously and encouraged, partly because of the crummy quality of the evidence available on the effectiveness of youth employment programs. Since then, somebody might be able to correct me, I don’t think the Academy has had one major meeting on randomized control trials in any area, including education. So it’s probably about time, better late then never.

I must say I admire the composition of this earlier session, it’s very unusual to find evaluators seated at the same table with the people who are responsible for cooperating, for actually doing a lot of the work that is entailed in generating the better evidence. That is marvelous, it should happen more often. Let me advertise William T. Grant Foundation’s award for such collaborations, which was, the first one was last year, the rules of the game are going to change and I hope we will continue with that award next year.

Implications of what, we were talking about the implications partly of U.S. federal policy on emphasizing science, the law, No Child Left Behind, the program announcements that were proposals that have been issued on teacher quality, for teacher development, on English as a second language and other areas are actually quite well written, recognized that they put in language that is intelligent, thoughtful, well crafted about randomized experiments. They do not ignore other approaches to generating information and they are basically good literature reviews. I’ll mention a little bit more about the What Works Clearinghouse in a minute.

Let me get to another kind of, although most of what drives this conversation is U.S. education policy or contemporary leadership and its emphasis on randomized experiments, and we go to the next slide Jason, it may surprise, it may not surprise some of you to learn that the World Bank is now into the business of trying to understand better how to justify the loan programs, the loans it makes, partly through randomized control trials. The workshops that the bank runs on evaluation generally in Canada in Ottawa last July had a special session on randomized control trials, it was a real revelation. The last biennial conference here in D.C. in July included speakers from the Work Bank, Harvard and other institutions, in the interest of information bank people and people from other countries who get those loans, who control those loans, about the idea of using randomized control trials, including on projects related to education.

How many people know about the progressive trial in Mexico? Alright, some of you who do not might be surprised to learn that in that trial conditional transfer, the conditional transfer program and from the economic jargon in which momma’s are paid, roughly speaking, to encourage kids to stay in schools rather then going into the agricultural fields to work. It’s 500 villages being randomly allocated to different regimens and to control conditions in the interest of understanding whether conditional transfer programs work. That’s coupled with a health component as well as some other bells and whistles.

We can go to the next slide, that might point in raising both the Bank and also the Campbell Collaboration efforts is that, did we miss a slide? Okay, I’ll wait for that other piece of information.

One of the broad implications from the past as well as some of the current discussion this morning and earlier this afternoon has to do with the idea that there are conditions under which one runs these trials. Some of them are broadly ethical, it turns out to be the case that 15 years ago the Federal Judicial Center was interested in social ethics of running randomized trials in the crime and justice sector including with prisoners in court rooms, randomly allocating youth to alternative judges in the interest of understanding judge behavior or alternative court procedures. And wound up with basically five relatively simple questions to understand the ethical propriety, the justifiability, the warrant for running randomized trials in a variety of sectors. They’re in plain English rather then philosophical language. Is the problem severe or were the solutions debated? Well, do RCT’s, randomized control trials, you’ll have appreciably less equivocal estimates then the alternatives, will results be used, will human rights be recognized? If you get yes answers to all of them then you’re well positioned to think seriously about doing control trials, if on the other hand you can’t assure yes’s to all of them, if you get no’s on all of them, you’re in bad shape, don’t even think about it.

On the question of will results be used, this is always a forecasting matter, speculation, my best recent illustration of interesting research not being used has to do with a very good systematic review, high end, of randomized trials of Scared Straight programs in the United States and elsewhere. There are, out of the roughly speaking 200 or 300 things that could be called studies rather then anecdotes, there are about seven or eight real randomized trials that are well run. They show roughly speaking that the program doesn’t have positive effects, that is to say bringing children into prisons and to talk to convicted felons and so on so as to be scared into behaving properly doesn’t have positive effects in the sense of reducing recidivism. It has negative effects in the sense of increasing recidivism. That is to say you have just created role models for these kids. The man is healthy, good physique, teeth are fixed, articulate, otherwise he wouldn’t be speaking, so you get the idea. Those results have been advertised in the Cochran Collaboration website, which Kay Dickersin knows about, the Campbell Collaboration thing on systematic reviews, and lots of other places, several journal articles in crime and delinquency and the like. The state of Illinois just passed legislation recently that would provide substantial monies to Scared Straight programs or Scared Straight like programs. There is no recognition at all of those scientific results. None, zero, zippo, they’re not trying to do a case study on why the governor, his legislative staff, and the legislators were entirely uninformed about the effectiveness of such programs.

Alright that was a little sidebar, why don’t we go to the next slide, implications, entities as units, we talked today about classrooms and schools as units, but there are other people in this game trying to change all the units with some of the same difficulties that we have in education. They include efforts to change hospital practice in which 20 to 30 hospitals at a clip are randomly assigned to different regiments in the interest of understanding how to get them to adopt a particular innovation that will result in fewer people killed, or fewer babies dying prematurely, or being born prematurely, and so on.

Crime hot spots were also targets for randomized trials that are place based. To understand whether alternative police strategies work to reduce crime, getting the cooperation of cops, of police chiefs, just as getting the cooperation of principals, is heavy duty work. We have to develop the relationships and know what the incentives are and so on.

In housing developments, Judy Gueron did not mention one of the trials that NDRC is doing on randomly allocating entire housing developments to alternative residence in order to understand how to raise the social capital, the education, wage rates, jobs, employability, and so on.

Cathy Sicama(?) at Yale has been doing randomized trials using entire housing projects targeting women in them to understand how opinion leader approaches can be used in introducing interventions that reduce the risk of, the probability of high risk sexual behavior, this is all in the interest of reducing incidence of AIDS.

Bank America has actually been tied up in experiments, a whole series of them, to try to understand how to improve services. The problems that have been identified in the Bank America research agenda include trying to assure fidelity of the implementation of the intervention. Does this sound familiar? Cooperation of the people tied up in the various banks and banking units, cooperation in the sense that they have to buy into the experiment despite the fact that their bonuses, their salaries, depend on high end performance in doing another job, not in the experiment. Control conditions and how they contaminated.

The main point here is that in addition, well, Shep and somebody else have focused on the, so prevention arena has one potential resource for learning how to get these trials done, when to get them done and the like. I agree strongly with Kellam’s take on it, where are you Shep? That is to say, well, AERA people ought to be going to meetings of the Society for Prevention Research and vice versa because schools are the units of random allocation analysis, classrooms in a lot of those cases.

I won’t bore you with the cloud seating experiments, they’re a little easier then these social experiments. But they do involve random allocation of places, blocks. Okay, next slide.

Okay, we’ve got a lot to learn how to do, that’s one of the broad implications, understanding how to learn cooperatively, how to share that information, is going to be tough. We don’t yet have a society for controlled trials in education or randomized control trials in the social sector generally. There is a society for clinical trials, for controlled clinical trials. We ought to have one in the social sector. We ought to be able to share information about determining, how to best determine the eligibility of places or entities for trials and readiness to participate. Work Bank has a big problems in this respect, they have to try to understand the stability of countries as well as local jurisdictions before they get into the business of doing controlled trials in those arenas. We have to understand say for the principal turnover rate or teacher turnover rate in school districts if we really want to do a good job forecasting how well we’re going to be able to execute the trial.

Incentive structures people have talked about already, this includes randomized roll-out in which you promise everybody the intervention but you roll the thing out randomly so you can actually test over a year or two period of time. Partnerships have been mentioned, but we do not have handbooks of how to develop partnerships in this arena, how to develop, how to identify the places that are ready to experiment. We don’t have cross cutting essays in handbooks that help us understand the extent to which approaching chief of surgery in various hospitals and about doing controlled trials in which the entire hospital is a unit. Engendered problems they’re similar to the problems we encounter when we try to engage teachers, principals to get their own schools involved in these trials. Next slide.

Okay randomized control trials versus the quasi experiments. A number of very good pieces of research have been carried out on this topic in recent years, British Medical Journal published something a couple years back. Glaseman(?), David Myers at Mathematica, and Dan Levy(?), among others, Will Shaddisk(?), in the business of trying to understand the extent to which estimates based on non-randomized trials, quasi experiments, model based estimates look like the estimates based on the randomized trials. If they look the same why should we bother with the trials? This stuff is hard work. Now the problem is that those models don’t forecast well what people would have done in the absence of the intervention, in the absence of the treatment. It’s a very tough job, some of the work suggests that there are approaches that are bias reducing or maximally bias reducing but you still never know whether the bias is zero.

Now we’re not alone in all this, the World Bank has the problem, I might say that the Columbia episode has a similar problem. Those of you who follow the science section of the Times may recall for real phenomena a full blown mathematical equation was given in one of the issues concerning the Columbia disaster, so called Crater Equation, an equation that predicts the depth and size of the crater on a surface when a projectile hits it --

-- [End of tape.] --

-- it’s impressive in 85 percent, we have to handle, learn how to better handle no difference findings. We have no tradition, no orderly tradition, of doing good post mortems, that is why didn’t we find the effects that we thought we were going to find despite this enormous brain power, the effort, the human resources, the bucks tied up in this thing. We got zero, nada, why? Is it the trial? Is it the power? Is it this, that, and the other thing? Okay, next slide and I’m gone I think.

I was going to advertise the What Work’s Clearinghouse is collecting these trials and going to try to criticize them, understanding what they do, make sure that the good ones are advertised and that the results of those trials are advertised to the publishing community. The big bottom line for me is trying to learn how to build capacity in this arena because it is really choppy, we’re all learning from one another, we ought to be learning from other societies, other professionals, learning how to build organizations that help us build the capacity is absolutely essential.

Final slide and I’m away. Walter Littman in 1937 criticizing Franklin Delano Roosevelt for doing phony experiments where disguising his political initiatives as experiments, announced, had a lovely phrase, unless we are honestly experimental we leave the great questions of society to the ignorant advocates of change on the one hand, and the ignorant opponents of change on the other.