DR. FLODEN: So time for question and answer, start with the committee. Any members? Yes.

PARTICIPANT: Could all of you answer a very simple question, and I know this is way too broad, but if I came to your school with a question that I wanted to do in a randomized field trial that was relevant to your school with sufficient resources to help you and a well trained research staff that could train your staff and they were diverse and appropriate culturally sensitive would you be willing to do a randomized field trial in some part of your school system for which you’re responsible?

PARTICIPANT: My answer would be yes, I would jump at it.

PARTICIPANT: I’m not at a school system now, we work with school systems, I’m at a university, but we would want to partner with you because some of our faculty would be interested in working with you and we would want to have some of our graduate students involved in the research.

PARTICIPANT: I should also say that I can name many school districts who would welcome the opportunity to do this, many school districts.

PARTICIPANT: I think it was said, Rich said it and it was restated a number of times today, it’s the question that you come, the context that we’re in where we’re talking about having to raise achievement, if you’ve got a good question, a good issue to evaluate or to study and it’s the good question in terms of the current context as well as its relevance within the community I think you’ll get that cooperation. It’s that old perception that a lot of research has to do with playing around the edges of important questions and not really getting to the heart of the matter, that is one of those stereotypes left over from years ago that often gets in the way of good policy going forward.

The other question, I’d ask one more question though if you came to us, it’s how soon would we have those results because we’ve got all these other schools, school districts, who need these results.

PARTICIPANT: If Congress were to create some waivers so that if schools participated in randomized field trials of important educational programs they might be exempted or deferred for a couple of years from the NCLB requirement, do you think that would change the motivation to participate?

PARTICIPANT: 100 percent participation is virtually guaranteed. You just can’t, this context issues, and I don’t want to overplay what I was trying to say, but it has been, when you look at trying to make change in educational institutions where you want to have good change, the potential negative sea change from No Child Left Behind is pretty overwhelming just in terms of what you see and while it may be antithetical to even contemplate research about the single trajectory of No Child you’d probably have some willing participants there, but clearly if there were some incentives in terms of the path, this relentless march that schools see before them and states and distracts see before them, I think you’re really --

-- [End of tape.] --

PARTICIPANT: -- you have to send a report of what impact there has been and too soon there are requirements for showing progress and showing impact when it’s not practical and it’s not reasonable and I’m afraid that there are many important and good innovations that probably we have thrown out because of that and so that there are many that we could probably return to, it’s also added to some of the feeling of being burned in school districts where teachers and others have started on things that they know are good or feel very strongly are good and they have to stop and move onto the next thing because it doesn’t receive additional funding after a year and it’s really not an appropriate amount of time.

PARTICIPANT: And also just to add, when Kay posed her question about the conditions for which a researcher would go to a school district or a state and we all said yes we would participate, so there’s no need for a waiver if you do it correctly.

PARTICIPANT: I think what you’d have is schools over willing to participate for all the wrong reasons and you’d be back into a, you really, 20/20 hindsight into a situation where you don’t have the active participation, people are just looking for a way around the system rather then working together to answer an important question.

DR. FLODEN: Jack and then David.

PARTICIPANT: Well, I’m a little bit lost at this point and so I’m going to ask the same question that I asked Dr. Gueron this morning, which is how many of the problem that you guys are identifying with randomized field trials are specific to RFT’s? It seems to me that what you’re providing is really a discussion of some of the issues involved with doing research in schools and I’m just wondering how much, how many of these concerns that you’re expressing really involve RFT’s.

PARTICIPANT: I think that there are a number of them that cross over to research in general and I think that a problem arises in my mind when the focus is such that there is an exclusion of others, there’s not the freedom or resources available to identify an appropriate one and I mentioned some of the specific ones in terms of the huge cost involved when you are dealing with highly mobile districts --

PARTICIPANT: If I could respond, I would think the same issues would be involved if I designed a quasi experiment with comparison groups, for example, I think it would be just as expensive, I’d have the same problems with attrition, I might have the same issues with withholding resources, I don’t see why that’s specific to a randomized field trial.

PARTICIPANT: From my perspective I don’t think there’s really anything unique other then the issue, I think at least the perception that the randomization dehumanizes what may be taking place, the fact that some students are getting it, some aren’t, and it’s a question I think that can be addressed. Most of these issues though from my perspective are issues about research et al, you’ve got a limited, you’re in a difficult financial time for schools, they’ve got a very limited set of resources, very high goals have been set for them in terms of a trajectory, and almost all offers of assistance, however you want to look at that, that come from outside are seen as potentially deflecting them from a direction in which they’re trying to go. So I think it’s a challenge not just for randomized field trials but for all sound research that we’d want to have done in educational settings.

PARTICIPANT: Let me phrase the question a little bit differently and then I’ll be quiet, but isn’t what’s specific and unique about a randomized field trial the randomization? And what Dr. Gueron described this morning as the participation in a lottery as opposed to arbitrary decisions about what intervention, like I might go in and I might give intervention A to a school and intervention B to a school and intervention, and school C would get nothing for example. And I would try and equate all those schools in a quasi experimental sort of fashion as much as possible. As opposed to randomly putting all three interventions in the school and randomly assigning kids to the intervention and teachers to the intervention, you know things of that sort. I mean if one design is stronger then the other because I have to make fewer assumptions about the design because I do randomization, what’s the problem there, what’s the problem with the lottery component?

PARTICIPANT: I wasn’t really speaking to the issue of lottery, I was speaking to the issue of getting underneath beyond what is answered by impact and outcome to the how and the why, the things that would come out of a case study, ethnographic sort of study, some of those sorts of things.

PARTICIPANT: I’d have the same problem if I did my quasi experiment wouldn’t I? I still wouldn’t know what the processes were unless I went in and measured them. I mean what specific, what’s the specific problem with the RFT?

PARTICIPANT: One of the problems is is there’s limited money and higher value is placed on randomization only and people want to get at some questions that are important, educational ones perhaps, that would use some of the other sorts of methods, they would not get done, those were some of the concerns that I heard expressed.

PARTICIPANT: Well, I’ll just stop. Thank you.

PARTICIPANT: My question is more about markets then about lotteries and it seems that, I guess this question is for the panelists and also the other people in the room who have done school based research. Is it a buyer or a seller’s market? It’s not clear whether there’s more people trying to get into schools and do research or school districts or whether there’s more schools who really wish that somebody would come around and address issues that are of importance to them and I can imagine a kind of ebay for educational research where there’s a kind of bidding system that goes on where you wind up, usually the samples we have are samples of convenience, I mean they’re almost always local, and even when they’re big they tend to be with people we already know, people we’ve worked with, and there’s tens of thousands of school districts out there where research is never done and where a lot of researchers don’t even know there’s an issue that they could address in that school. I wonder how your experiences as the recipients of this stuff, if many people are coming to you trying to get into your schools or if it rarely comes along and you hop at the chance, and from the other perspective how do you identify schools in some systematic way, or in some focused way?

PARTICIPANT: Let me, one of the concerns that we have when you’ve got a diverse, Indiana’s not diverse in a lot of senses but you’ve got several large urban centers, several districts that would be recognized anywhere as being urban school districts and lots of rural very homogeneous districts, but you’re diverse in these other ways and the issue of generalizability of the research or of method is a very real concern that doesn’t often get addressed. That scalability issue, how do you take things to scale no matter what the conditions, and the fact that people, because the airport’s nearby and you’ve got a nice ring of people around Indianapolis, lots of people want to come and do stuff in and around Indianapolis. Nobody wants to go to Marengo(?), Indiana and do research, and it’s the sample of convenience that’s often the problem in not looking at that larger population. There’s enough near Bloomington, Indianapolis is less then an hour drive from Bloomington, so you get hit from your research institutions, hour drive from Perdue as well. In trying to look at those under served populations in a sense of places where there has never been any research, I think you’ve got more willing customers, for people who’ve dealt with it and have panels that regularly review, whether it’s graduate students or other, they’re typically not that receptive, it’s a larger task for them and those people who haven’t been involved maybe much more willing participants and more easily brought on board.

PARTICIPANT: If I might answer that as well, I think it’s clearly a buyer’s market, particularly with so many people working on dissertations so you have the researchers and you have people working on their dissertations and so for the urban centers everyone is coming there trying to get work done. But I don’t think that, I think that there appears, from this conversation there appears to be the perception that we’re not willing to work with you on that and that’s not a true perception. I can think of several times when we have, I’ve been called by one of the school districts who said let’s get together and write a proposal so we can do a randomized trial on, the most recent one was one social promotion. And we had Philadelphia, New York City, and Broward County, they all wanted to do this research, we wrote the proposal to the Department of Ed, didn’t get funded, but this is another example of something that was important to us. We are now, we have a council of grade city schools who are now being approached by more and more organizations coming to us asking us to convene some school districts to help do some type of research and some of it is indeed random and we hope to be working with NDRC in the short future to do another study that we consider random around some other issues that are important to us. But again, the issue, you have to make sure that the issues are important to your constituency, some of the conversation, some of things you say you make me think that you think we don’t want this to happen and that’s not true, change that perception.

DR. FLODEN: Comments from the floor? Karen?

DR. KING: Karen King from the National Science Foundation. Speaking about the urban centers and being so inundated, what is the capacity of the urban centers to take four or five different randomized clinical trials going on at the same time in any one school district, say New York, that has a lot of moving pieces and lots of things going on and things changing every day? And how are the researchers who are doing that kind of work in those settings accounting for the fact that there are three other major initiatives going on at the same time and the issue of replicability when you’re talking about some place like New York that has the change in the way the school district is organized going on and the implementation of the district wide curriculum, and all these other things and then you’re going to add a randomized clinical trial or maybe two into that setting, the interaction between the two trials is one issue but then the interaction of the trial with all the other initiatives that are going to come and go given the way that urban school districts, will come and go over the course of a say four year trial period.

DR. FLODEN: Who’d like to address that?

PARTICIPANT: I’ll just start it by saying that I think that that is an issue, it’s certainly an issue within schools that may have so many things going on and in the districts where there’s so many things going on that you can’t even find a person who can name them let alone have them coordinated or really understanding ways in which they are interacting or not. I think that this is something that would need to be considered and looked at, but the important thing, I think Sharon keeps referring to, is seeing to it that in conversations and planning with the districts you are identifying problems and areas that are important to them also, you’ve got to find a vested interest of both in order to be able to move forward. And then there is the willingness to do what it takes, to figure out if it’s the appropriate time to go ahead, but I think too many going on would be an issue.

DR. GOLD: Norman Gold again. It seems to me that there are two issues, one is random assignment, and then one is random field trials, and I think they’re different. One has to have much more control, be much more intrusive, with much more specificity, much more intrusive from the outside it seems to me in terms of what we’ve done in the past when we used quasi experimental designs and that sort of thing, what we were trying to do was to represent some kind of systematic way of observing phenomenon that are there but were not trying to control those phenomenon from the outside. And I’d like, we’ve been using those terms as though they were both the same thing and I don’t think they are --

PARTICIPANT: Could you say what you think is the difference between the two of them?

DR. GOLD: I mean one, when we can take a quasi experimental design and simply say that we’re going to look at the different schools that are involved in a particular program and look at them in a random way, we’ll assign, there are programs that have within a school district some programs have a treatment, some don’t, and we’ll randomly select which ones we’re going to look at, we’re not going to select them on purpose. Or what we might do is to look at them in terms of characteristics, so that we would have, we would do it random within a set of parameters that we were interested in. So what we can do is to use randomness as a way of trying to not deal with bias selection. On the other hand if we were going to do a particular, try and do something that was an experiment then what we would do something that was much more controlled and much more defined. Now am I --

PARTICIPANT: I think the latter was certainly the intent of the panel in putting this together was to look at things where there’s random assignment of units to different sorts of treatments rather then sort of haphazard selection of treatments or random sampling of people in a survey.

DR. GOLD: But I’m saying I think we’ve been using those terms interchangeably and I think that the requirements for the latter, for the more controlled kinds of studies, have to be much more rigorous and therefore have to be much more judicially selected.

DR. KELLAM: Well I learned something very striking and startling from this panel. The problem isn’t any school districts receptivity, the problem is the researchers don’t know how to go into a school districts figure out what the vision is of the school district and come to mutual self interests. Now if we can nail that down you’ve guys have made a major contribution because that’s the excuse that researchers in education and in public health very frequently but don’t tell them, that’s the excuse people use, you know, you can’t get in there they won’t let you. I would just like to point out that this is the least taught subject in any discipline coming anywhere close to doing research, not just randomized field trials. You don’t get taught in graduate schools or medical school how to make partnerships which start with getting into the vision of the institutions and people who run them and people who use them. It’s not taught, so when we’re talking about how to move this field forward we got to start with how do we identify groups of researchers who can quickly learn, and what mechanisms will we use to get them going fast, that will get us up to snuff so we know how to relate to he needs of a school districts, particularly when they’re under a gun. So that’s where we are.

I can tell you this anecdote, when the first multi hospital study of phenothyazine(?) was done, that’s the modern beginning of drug treatment for schizophrenia, nobody knew how to do the research, what they did was to do a multi site multi hospital study, which in fact had a hidden agenda, mobilize the partnerships that would be needed and get young people in there and old characters who had some insight how to start doing it and get it going with a multi hospital study. The thing you couldn’t surrender was rigor, you had to develop it with rigor, and it was a randomized hospital based trial. My first trial. I was a kid supposed to measure ward environments and the impact of drugs interacting with patients in ward environments. And you’ve got the same problems now and the clinical trial network that NIDA has just set up that Bob Boruch talked about earlier is in fact a structure which does that.

A similar thing happened in Holland when they developed community mental health centers, they wanted to move research in, what they did was to make, they insisted that the research element be part of the budget but then they had to go around and make the partnerships and find the researchers and train them to know how to use the research element and the budget.

So there are mechanisms for shoring up this kind of partnership but it has to begin with the researchers realizing that the fundamental task is learning how to make this new institutional partnership, which identifies the self needs of the institutions, and you can then build the research into their vision.

PARTICIPANT: One way of looking at doing that, we started four years ago here in the D.C. area when we formed the Washington Area Education Council, made up of all the universities in the greater D.C. area, the superintendents of each of the districts in the area, and the overall goal was to work long term in partnership to address the challenges in K-12 and do it as a partnership. And through that, I mean we’ve conducted studies, we’ve ended up helping districts look at what their challenges are around No Child Left Behind, and then various universities have faculty with different interests and expertise, we can then bring to bear ones that are responsive to them.

When I came here almost five years ago and suggested it the idea actually came from when I was at UNC Chapel Hill where they started the same sort of thing, there are a number of universities in the research triangle area and school districts and they came together. Long term partnership and commitment relationships are built so it’s not just individual faculty member who’s going to go and let me say this and that’s where over time trust is built up so that as needs arise you’re able to respond into it.

DR. FLODEN: I think from the beginning of the day today people have been emphasizing that if you’re going to do any sort of rigorous research design, randomized field trials or another one, that it has to be done in a way where everybody agrees that it’s an important thing to carry out the research in a way that meets the characteristics of the design and I think what we’re hearing here again is if you want to do that you have to work in partnership with the people in the schools, in the states, in the districts where that’s going to happen because everybody has to see the value in the question that’s being addressed and see the value in carrying out the research in a way that you can use the methods that were appropriate to the question and carry the research out in a way that’s consistent with those methods.

So I hope you’ll join me in thanking this panel and then Kay Dickersin is going to come up and offer some concluding comments.