DR. DICKERSIN: I want to thank all of you for sticking around until the end of the day but I think it really has been worth it, this last panel was particularly exciting to hear because so many ideas were pulled together that we’ve been hearing throughout the day and in such an important way from the idea that we had people who are both training graduate students and people in the field working with students and with teachers. So I’m going to try to pull together everything I’ve heard and if I’ve forgotten something make sure that you add it.

I think the first thing that we heard and what we’ve kept in mind all day is that it’s appropriate to match the question that we’re interested in asking with the research design, and how important qualitative methods and observational or quasi experimental methods are to obtaining the answers to many, many questions. But when it comes to looking at cause and effect, perhaps, the randomized field trials is one that might be implemented more often then it has been in the past to try to address cause and effect.

There are perceived problems with this, however, and there’s a lot of concern out there. Some of the areas that came up repeatedly were the costs, the large costs of randomized field trials and that spending money in this way may take away from other types of important research. There’s also issues about the generalizability of the results from trials to the populations, whether it’s the teachers, the setting, or the students and how important this is to those who are trying to interpret the results of trials. The ethics of possibly withholding what might be a promising intervention at the end of the trial from those who end up in the control group worried some as well.

There’s also worry that we heard about that outsiders are coming in and trying to apply a medical model to education and that this might not be an appropriate fit, and I’ll talk a little bit more about that later.

I also heard that, and I hope I’m interpreting this correctly because this is definitely interpretation, that perhaps this interest and move towards randomized field trials is being associated with the No Child Left Behind legislation and that it feels like a mandate because they’re together rather then looking at the benefits of the randomized field trial as a stand alone design, and is there really a benefit, or is it too intimately tied with some of the other negative feelings that go with legislation and mandates. And that is probably worth all of us talking about and keep talking about it.

Then the next issue, I’m trying to talk about these in a slightly different order so that it makes sense, at least to me. Then big challenges were mentioned, the capacity, that is researchers needing to be trained to do these types of studies and what about the staffing at the schools where the research is taking place, there’s capacity, doesn’t exist now, how are we actually doing to do this, what kinds of resources and time will it take.

What about schools who are already struggling with No Child Left Behind mandates, now this is going to take time, research is going to take time from the instruction goals and needs that they already have.

There’s also a need on the part of researchers to be cognizant of the communities into which they’re coming, they have to be culturally competent, racially and otherwise diverse and not just impose themselves on a community and pose research, but they have to work in partnership with the community in which they’re coming. And more resources are needed by the schools and the school districts if they’re going to undertake and participate in research.

But many of these problems are not unique to randomized trials. The benefits of randomized trials may be unique to randomized trials, that is being able to look at cause and effect in an unbiased way but the problems for the most part may be common across many types or all types of research design. However, I think what I heard about the problems has something to do with the tension between population based research and individual needs, that is the application of the research is looking at the population as a whole and yet the teachers and the school district are concerned about individual student needs. And these are very, I think about this one often because my mother was, she’s a recently retired reading teacher in the Boston Public Schools and she’s always believed in what I do as a randomized trial person but when it comes to, we had a recent conversation about reading recovery and she’s been to reading recovery open court, she done the various methods, and what does this mean, am I going to mean that what she feels she has done and contributed to children’s lives is now nothing, that it means nothing if these approaches end up not effective by a randomized trial, and so I think we have to really talk to one another about this very difficult intersection between population based results and the individual needs and keep that conversation going at all times because one is not better then the other.

So how do we do a good randomized field trial? We have to focus, we heard that we have to focus on internal validity of the study, that it does no good to have a generalizable result, external validity, if we don’t have an internally valid study to begin with. And it’s especially important to focus on that early on in the planning stages, the design stages of the study. We have to ask the correct question and we have to have a lot of planning time up front to give everybody time to do this study properly, we don’t want to throw away a lot of excellent resources on a study that was poorly designed and implemented.

We have to focus on recruitment, which is always difficult, no matter where you are or what you’re doing in a randomized trial, we have to get sufficient sample sizes to be able to draw valid and reliable conclusions. We heard also several times that there are key positions in these randomized trials that are key to success, it may be the project coordinator, it may be the coordinator in individual schools, it may be the people who are talking to the parents about what this study means, but these people who are there on the ground floor doing the study, not the leadership necessarily but the people doing the study are key people to success.

We also heard many times that you can’t do research without the ability to then implement the results of that intervention, if you find out that it works well you have to be able to implement it. You can’t then say to the school system or the community yeah this works but sorry, we don’t have the money to do it, that that’s not fair. So that has to be part of the planning process from the beginning.

And this is part of the agreement that the researchers have with the community, it has to be a true partnership so that the research has to be incorporated into a master plan and that was described how that was done for Baltimore in one particular situation but to me it was a very impressive description of how the research was part of the master plan and then what would happen after the research took place if in fact the intervention was determined to be useful. And that these partnerships are part of building a trust and perhaps the trust is one of the important key elements of making a successful research and randomized trials in the community.

That is a summing up of what I heard and oh, one other thing, the most important thing perhaps. The educators that we heard from at the end and those training educators were very positive about participating in trials if the barriers that have been mentioned can be overcome. And I was really happy to that, I didn’t only hear a yes, I head a lot of enthusiasm and that was from people who also recognize all the potential barriers that are out there and I heard a lot of them and I think they all sounded pretty valid to me. So I thought there was a lot of positive that came out in terms of doing randomized trials but we have to pay attention to where the challenges are.

[Applause.]

PARTICIPANT: Does anyone else want to add anything?

PARTICIPANT: I just want to say one thing about something I didn’t hear us talk much about just so that we’ll have something to talk about next time. And that is back to sort of where Rich started us off about the importance of the question and it has to do kind of with the grain size of the treatment that’s most appropriate for these kinds of trials. In the three examples that we heard they varied a little bit in terms of the scope of the treatment and the scope of outcomes against which the treatment was being evaluated, but you can think about the extremes so I maybe want to test some new great way of teaching kids to solve quadratic equations, that’s kind of narrow and probably wouldn’t get a lot of excitement in terms of subjecting it to a large and rigorous design.

At the other extreme is the problem I’m facing, I’m supposed to evaluate the impact of the high school graduation test in California and there is not an easy prospect short of waivers, I was actually kind of serious about the waiver idea, of waiving some districts out of this requirement and some districts in, randomly assigned, so that I could get some valid comparison, I’m supposed to look at things like graduation, retention, drop out rates and then college attendance and outcomes after, and I can look at that but there’s no great comparison. So this grain size issue I think is something that would be fruitful topic for some further discussion, hopefully not today.

The one other thing I didn’t hear was sort of the blind and double blind part of the research design that when I went through school they were emphasizing. I did hear Sharon talk a little bit about how much everybody distrusts the publishers own results because there’s a high potential for bias and a call for sort of non-involved third party research and evaluation, but the idea of whether you can or can’t have a study where the participants do and don’t know whether they’re getting the placebo or the treatment and whether the researchers collecting the data and analyzing it and scoring it and so on can and can’t know sort of what the treatment condition is other they know that they’re analyzing it. So I just throw that out as maybe some additional topics that might be around the edge of this, not so much at the core, for a future discussion.

PARTICIPANT: Can I follow up a little bit on that? I also thought that this was a wonderful day and a lot on this issue about random assignment but one of the comments that came up are these issues about random selection, and if we think about the grain, how many people, what are we looking at, what kinds of effects, when do we consider issues about random selection and when do we want to use quasi experimental design and when do we want to do something else. So it seems to me that when we have the whole range of opportunities that are available to us as researchers to work on these pressing problems in education that we can become much more deliberate and thoughtful about what were choosing to come to investigate the problems that are out there. So it seems to me that I would hope that we would have the same kind of close look at some of the other kinds of methods that researchers use in their toolkit.

PARTICIPANT: I just wanted to throw something else in. One of the things that Judith Gueron said this morning about randomized field trials and when they’re effective is that they’re useful in situations where you have limited resources and you can’t give the same thing to everybody. But many of the questions of great importance in education today are things about say choice of different curricula where it isn’t the case that there isn’t enough for everybody to have any particular curriculum, people have to choose a curriculum and there are different ones and some people are going to choose some and other people are going to choose a different one. It’s important, people are interested in the causal question of well if I pick this curriculum are my kids going to learn more, but it’s not the case that you can’t pick it for everybody. So I think a question about randomized trials is whether it’s feasible to do things about important causal questions when they are about things where people could, if they thought that one was the good one they could choose it because it’s not a question of scarce resources but a question of which among different things each of which might cost about the same is the best one to choose.

PARTICIPANT: I would just add a perspective as coming to this group as an anthropologist, throw in a little cultural perspective, which is today I think we also learned that these random field trials aren’t just a matter purely of logic or method, there’s also a culture of this kind of world, and so from inside it it sounds like people often who do this kind of work don’t get why there’s resistance to it because like any culture it has its norms and beliefs and the people who are resistant to it are seen as being ignorant of it, if they knew more about it they would embrace it. And I think today what we had a chance to do, especially with the last panel, is to hear how, I think Eamonn also put it, there’s the intended and the achieved and I think for those of us who may recommend something like random field trials we can’t, I think it’s disingenuous to say that all the things that have been done badly are insensitively using this method, that’s not, we don’t mean that, we only mean the good stuff. I think we really have to think about how methods carry with them politics and histories and relationships. So I commend the last panel, particularly the morning one too for bringing us closer to appreciating these very complicated questions.

And I think also finally I would say it’s not just a matter of people who do this kind of work, as Shep said getting better training and knowing how to assess, get access to communities, it’s also a matter of changing who gets the grants and who does the work. In fact there are people who already know how to work in these communities, they’re just not getting the money to do the research. And there are, for instance, in schools have had courses on action research and teacher research and ethnographic research where people spend a lot of time learning about how to serve their community for instance. But the history of the funding of these kinds of grants means that most of the money tends to go to people who aren’t from the communities being studied, so I think it’s also a matter of thinking much more globally about changing funding and power. And then I think these methods would be embraced if they could somehow be put into a different political and cultural context.

PARTICIPANT: At the risk of being a little anecdotal here I’d like to echo that, and also at the risk of bringing in something from public health. I would encourage the groups to really, who are perhaps not so in favor of randomized trials to be put in the situations with the trialist where they constantly are talking. Just from my experience again as a trialist, I work quite a bit also with breast cancer advocacy groups, as a consumer, not as a trialist, I just happen to be a trialist. Then I found out low and behold that all these consumers were not in favor of randomized trials, they wanted the treatment they wanted. And at first that was kind of amazing to me because as you’ve just said you can’t understand where people are coming from if you’re involved in something. And I also was amazed they didn’t want me to hold places on committees as a consumer because they thought I was too invested in trials, well wait a minute, I’m a consumer, I’ve had this disease, and I didn’t understand that until time went on. But the fact that we were so often in the room together talking about this made me really see the other side so much more about where there are difficulties with doing randomized for example. And I just think those conversations have to keep going on because, and you still might hold the position you originally came from, for me that randomized trials are good, but also we’re taking into account much more in the work we do what everybody, the whole community, the different communities feelings are and we can’t do that without being exposed to it constantly, and without constant challenges and debates to our knowledge and beliefs and it just can’t stop, there’s no point where you know enough. And so we have to put ourselves in one another’s presence all the time.

DR. PEDUSKA: Jean Peduska and I’m going to be very public healthy and I’m going to be very anecdotal, and I’m a trialist, so I guess that says it all. And I work with Shep in Baltimore, I’m the deputy director of the center, and I want to bring this up because I think it’s the issue of how do people also get training and you mentioned the thing about funding. Because it’s also how do we get very interdisciplinary and how do people have opportunities who may want to do these large projects, a lot of the trials we’ve been talking about get very large, and so how do junior people have opportunities, especially if they’re from different disciplines and we’re joining education and we’re joining anthropology and we’re joining public health, and we’re all wanting to do it but to get a $6 million dollar grant I’m going to work with Shep, or I’m going to work with John Reed, or I’m going to work with Jack Fletcher, I’m going to work with someone who’s name is out there but while we’re also trying to stop being kind of covert agents and move ourselves overtly forward. So I guess I want to also leave with a charge to the committee if I may be so bold is I know you’re thinking next about training issues and some of these issues and I think it’s not just how do people get the training but as you were saying how does there become an awareness that if we’re going to move these kinds of things forward, well, that it’s not just about funding, it’s not just about training, it’s about all of that together so that the comment that was made earlier this afternoon about making sure that the qualitative people in education don’t see themselves as second class or don’t get subsumed or don’t have a place, but what is that place as we keep moving forward in our research and really thinking about that as we go forward because I chose a kind of non-traditional path and I never looked back and what I’m finding, I was at Penn State last week and what I’m finding out is there are a lot more people who are considering how to work on larger things, they want to move forward in the world rather then have maybe kind of a more traditional academic career, which I did not choose intentionally, and how does that get supported, as they’re moving forward as the next generation.

DR. KING: Karen King from NSF. I hesitated to add anything because I kind of want to leave, too, but I think one of the things I hear as a tension about this community issue is also reflective of the fact that, going back to something that Rich Shavelson said this morning, is that people have questions that they want to answer and that’s where you start is with your questions, and so it’s not that I see randomized field trials as good or bad or indifferent but they aren’t something that answers the questions that I care about. And I think that two issues that didn’t seem to come up very much is what are the questions that people really care about and are they the most amenable to being addressed by randomized field trials. We’ve been so stuck in the conversation about the method and didn’t hear people saying well these are the kinds of questions that people are trying to answer, that haven’t been getting answered, that randomized field trials are helpful for answering as much as I heard we’re not doing enough of them and maybe the real issue is we’re not answering the right questions or the questions that people really care about.

And the other thing is that I worked on the Rand Math Study panel and in that document we had what we call a cycle of innovation, and one of the things I feel personally doesn’t happen is the movement of ideas around that cycle so that different questions get asked about a similar thing and so often I feel when I read something that doesn’t resonate particularly with me most of the time it’s because I felt like they were asking the wrong question. And so when I was listening to Bob talk about this curriculum thing that’s one of my sort of oh, I hate that question because I think it’s so unimportant, because I think it dehumanizes teachers in a way that Barry was talking about a little while ago in the sense that teachers then become just the transmitters of curriculum as opposed to decision makers in their classrooms and so what does it mean to even have a protocol of implementing a curriculum when I as teacher as a decision maker make decisions about what it is I’m going to do in my classroom --

-- [End of tape.] --

-- in a completely wrong question. But I think sometimes that’s informed by the fact, or not informed by the fact that they aren’t looking at other people’s work and so while I do primarily qualitative research, in my graduate training I took four quantitative research methods courses, and I think one of the issues that I would urge you to think about in the next discussions about preparation of future researchers is while there’s a tendency towards narrowness and getting people sort of into research quickly and then doing whatever it is they do and being very narrow, there’s a need for a certain kind of breadth so that you can look at other things that are going on in the multiple methods that people are using that might help inform your research and help you inform other people’s research.