DR. FLODEN: So we have a time for Q & A again, I’ll start with the panel members. Yes?

PARTICIPANT: Let me just clarify something as kind of a defensive remark about medical trials, and that is that they’re not all drug trials. We have a lot of behavioral studies looking at whether counseling people to stop smoking works, surgical trials, stuff like that, so I think they have some of the human component which is what makes trials so difficult, and there’s a lot to learn from them. Then the second point is it’s so interesting what you’re saying about would qualitative research become a sort of second class citizen if randomized field trials rise higher in the scheme of things, and interestingly enough in the health field all the discussion now at meetings is bringing in qualitative researchers and let’s all start working with anthropologists, ethnographers, do qualitative research to inform our randomized trials, so we’re sort of moving the other way and there’s a lot of fear about that.

PARTICIPANT: We’re rushing to meet you.

PARTICIPANT: Again, this is for Mr. Kelly. I have two points I want to make. The first one is if you know the study is superior you questioned whether in that case the results are meaningful, it seems to me there’s no reason once you know, you’re worried about somebody being, is it fair to have a control group is the point you make, but if you know something is already superior before making the study then I don’t see the point of the study at all. That’s my first point.

The second one is you question whether you can make random assignments under the influence of politics and otherwise, it seems to me mathematically you can make random assignments no matter what without any politics or anything just using mathematical models.

DR. KELLY: I completely agree with you on the first point, if you know something is superior there’s no sense in having to run the trial to determine in. In fact that’s why I think that if you look at these studies in practice that’s why they look to me more like engineering studies. It’s not clear in the outset what’s going on and to the extent that one could run an honest to God randomized clinical trial to judge as the effectiveness one should do that. Part of the problem is is that I don’t know if we have very good theory in education, I mean if you look at the Scientific Research in Education publication, and I wish Rich was still here, they talk about what to do when theory is weak. And when theory is weak then you do methods other then randomized clinical trials, so I completely take your point and actually have no contention with it.

The point about in spite of political considerations couldn’t you randomly assign, absolutely. There is an interesting assumption though about the randomization that it necessarily guarantees that the groups begin in equivalent fashion. I can’t imagine playing cards with anybody and they get a royal flush just by chance alone and saying I’ve got to turn this back in, I didn’t expect that to happen by chance alone. So you’ve got, for example, in Cooke’s study here, it is not worthy that the program in control schools started at about the same position for the non-achievement outcomes suggesting that the results may well be causal. Okay? Next sentence. However, the program schools started out behind the controls in academic achievement. So you’ve got the possibility of doing mathematical manipulation and then for a variety of reasons you may or may not have the statistical equivalence that you would expect upon which the rest of the argument flows in spite of all the ongoing problems with threats to internal and external validity, I mean if I were thinking about my kids who are in this area being involved in this study, the existence of the internet has so changed the way they experience their peers, homework is now a shared event, I can’t imagine, not even a shared event, unfortunately more and more it’s becoming a plagiarized shared event. As the good Lee Quandock(?) said generalization is decay and there are interactions by decade into any findings that you get, probably no interactions by year. I mean there’s a huge amount that’s going on and I’m actually very, very happy to live in a world in which randomized clinical trials don’t work. It allows for human creativity and response and the wonderfully interesting ecosystem.

DR. FLODEN: Anyone in the audience like to make a comment, pose a question? If you do please go to the microphone.

DR. KELLY: Please ask Bob some questions here, too.

DR. SLOAN: Bob, I pose this question to you, Barry Sloan, National Science Foundation. In the NRC report one of the conclusions of the report was that method should follow question and in my experience training becomes before either method or question, and I think both of you highlighted the notion of the human capacity issues associated with trials and associated with good research in education. What can schools of education do to ensure that students, that their graduate students are trained will in the first place?

DR. BORUCH: My inclination is to give a simple response that runs something like I don’t know, but because I’m being paid so well I feel compelled to actually provide a response. It has been a disappointment that if people in graduate, well, let me confine my attention to graduate schools of education because that’s all I know. It has been a disappointment to learn, find out, discover that most students rolling through graduate schools of education for masters and Ph.D’s will not have taken a single course on research design writ large much less a course in which a feature of the course has to do with the idea of fair comparisons, benchmarks, unbiased estimates, co-effect. That is true of most of the students who roll through the graduate school of education actually. So we’re dealing with a kind of a, part of it is a matter of taste preference, people just don’t want to take those course, they’re terrified of them, they don’t realize their value, there’s no press to do so. For example teachers, school principals and districts are not sued because they are using methods of teaching that are behind the times, that are not up to date or that are defective. Hospitals, physicians, are sued. Maybe we need a new, I don’t want to say this, this is being recorded and so on.

As to other ways of improving the capacity include bypassing schools of education entirely and trying to assure that teachers and principals who are of like mind are identified as they have been in the studies you heard about earlier, they’re engaged in the process, they themselves learn something about research, and it becomes a post baccalaureate, post masters experience. You could rig up internships that fix this, so it’s to place able people in various interesting research projects, the projects that are promising. So independent of the graduate schools you grow a cadre of people who are more sophisticated, more thoughtful, more able in this arena.

In effect, NIDA has I think done something like this because of its, Shep knows the acronym for this, it’s the nodal system in which one entity serves as the node for actually designing or identifying interesting interventions that might then be tested, it is associated with six or eight or ten satellites, hospitals, drug treatment centers and the like, which consists of professional staffers who may not know much about research but who in the long run become active productive eager contributors to the research enterprise. Now that’s an administrative organizational system that is reasonably well funded, five year stretches I think for each of these things, five year contracts or five year grants, we don’t have anything like that in education, maybe we should.

The idea that Judy Gueron gets repeat customers, that is to say a series of trials she ran five years ago, six years ago in welfare, a new set of trial opportunity come along, she wants to do employment, goes back to the same communities, they don’t chase her off. That is a very good sign, that is also kind of elevating the level of intellectual and social and professional capital that we have in this country to actually generate interesting research.

I might say in this business about alternative ways of looking at the world, their rate of qualitative, whatever, none of the big trials in which, in the crime and justice sector, or in the housing sector, or in the most recent welfare sectors that I’m aware of and in which I’ve been involved have not had people who know something about process, organizations, people, how to stalk, how to shadow a kid to understand who are parent, who are welfare recipient, or a drug addict to understand what’s up on the street and the best of the trials routinely include this stuff.

Assuring that you find a good ethnographer, anthropologist qualitative researcher, bring them along, train them up, get them to learn the vernacular just like you have to learn theirs because it’s strange at times is hard but it’s certainly worth doing because you pick up a lot of interesting information at the design stage, how to avoid problems in getting off the ground and into the street at the process implementation stage, where the trouble spots are, at the analysis stage because all kids are supposed to be signing their attendance cards and it turns out their friends are signing their attendance cards and you’re getting bad attendance rate, the qualitative, as a statistician or as an economist I trust the data --

PARTICIPANT: If I might just briefly respond. One thing I think we have to keep our mind on here is what the goal of education is generally, because we’re in a political climate in which driving up test scores is important. I don’t think you have to worry about schools of education being bypassed, I think they’re going to be bypassed by companies that are able to drive up test scores, and we’ve got to make a decision as a civilization how we want to view education and as generally as we want to view it. I don’t know in particular about whether or not you want to do all the schools of education, just leave mine intact if you wouldn’t mind, and it’s interesting to hear a policy recommendation from you about a randomized clinical trial as to its value, so I’d be open to participating.

DR. FLODEN: I’d just note that the committee is also taking up the idea of how you strengthen the community to prepare people to do educational research in graduate schools, wherever those graduate schools might be.

PARTICIPANT: Eamonn, you made comments about two issues involving reading studies and I just wanted to ask you to clarify or expand a little bit, one was that, was it the National Reading Panel could have or should have learned some lessons from qualitative research in reading. The other would be to say a little bit more about how the current political climate is constraining outcome measures in randomized clinical trial.

DR. KELLY: The two documents there that are relevant, if you go to the web and look at the National Reading Panel, and look at the methodological approach that they took I think there were 34 qualitative research studies that they didn’t look at, they had a set of standards, they were looking at the quantitative studies, and as a matter of fact, Camilli and others, I think it’s an EPAA publications 2003, went back and looked at the studies that were removed and have suggested that the picture of reading instruction might have been enriched by leaving them in. So the decision to allow certain studies in and other studies to be taken out, it’s not clear that it was done for example on the basis of peer review as you would like to see it in healthy science, it seemed to have motivations other then that. And this is just something that we need to bear in mind in terms of how resources are allocated in the future.

Your other questions about the freedom to use many measures, take for example Paul Cogg’s(?) recent work in statistics education, he was at a meeting I was running just last week in San Diego and he said he searched through literature, he could find four or five studies total around the world on how kids learn statistics in high school. So he has to go about thinking about those main constructs, how teachers understand them, how he understands them, how students understand them, and that’s something that is probably not going to show up on many standards of learning for various states, so he has to find schools that are willing to be involved in that sort of work because at the moment there’s no payoff, it’s not going to drive scores off.

If you look Jere(?) Contra’s(?) experience in Austin, Texas, I mean she was asked, and her group were asked to leave as I understand it from her presentation in AERA(?) 2000, they were asked to leave because the school felt that her pull-out program, while it might have been doing good things about children’s understanding in mathematics, it wasn’t driving up test scores in mathematics. In the state of Virginia they are now soon to implement graduate requirements, this is very, very high stakes tests, so the ability, if you look at it from the diffusion of innovations point of view the “marketplace” is changing radically to which our researchers can go and do their work , and if they’re not in line with what is seen as the high stakes or contributing to that they just have a tougher job then they used to have and I believe if you look at Vanderbilt’s experience in Nashville that will be borne out.

If you look on the walls back out here they’ve got all these species that are endangered, and there’s a comment right up beside them that says that if you change the habitat you can kill the species, well the habitat at the moment is changing quite radically in education and what it’s going to do to the various species, including species of education researchers, is unclear.

DR. KELLAM: Shep Kellam. Well, I’m a big risk taker, that’s supposed to be a predictor actually of drug abuse when I’m a teenager. It seems like what we’re doing is several things simultaneously and the problem is how do we accomplish this multi dimensional program when the outcomes are divergent and maybe even in conflict. I mean while we’re arguing whether the RFT should be a centrally important paradigm in research strategy we’re also faced with a whole set of parameters of how do you build capacity. And it’s not unlike the problem in psychiatry, in medicine, with drug trials in the late ‘50’s and early ‘60’s, where clinicians were fighting with, essentially qualitative researchers and clinicians were fighting with the randomized field trial, the controlled trial, and the issues of quantitative methods and now this is of covariance, which is a big time breakthrough and so forth. And the argument was finally settled by Congress and I think that it’s extremely important to recognize that science is ultimately driven by the social culture we sit in, both in terms of priorities, definitions, sometimes those priorities become advanced in relation to where the scientists are because scientists often fall into two groups, hedgehogs and foxes, there’s the big vision integrators who want to make things broader and integrate, and the others are very concerned about losing the sanctity purity of the perspective they take. The problem is how to resolve all of this simultaneously while moving ahead and I would only point out that Congress had made a very strong recommendation to us and at the local level when the community says join in our vision we think that’s a basis for going forward, that’s how you get to do RFT’s. Well, we have a national vision, the national vision is that we want to move achievement first forward. False starts all over the place, achievement tests poorly measured, statewide this and that. But the fact of the matter is we’re beginning to get a really national mandate that’s in my view, hopefully maybe, unlikely to go away. So the real question is how do we increase capacity, formulate the parameters along with we have to make policy about training, policy about criteria, how do we organize knowledge, those parameters. I think it’s quite clear that if we take the premise that we have a mandate which we either have to respond to or tell them why we’re not, we need to move forward. I think these parameters are going to require us to make partnerships in public health and public education together, I think that many areas that share problems of growing capacity, we need address. But I think that as I understand the purpose of the committee it’s a very seminal incredibly powerfully important point in time where there’s a very loud mandate to deal with the problem of increasing rigor, and so the question is what are the parameters for doing that while at the same time taking Dr. Kelly’s caveat’s in mind. I don’t know whether this makes any difference but it seems, well, anyway.

DR. FLODEN: I’d just like to add that the broader purpose is this promoting scientific research in education and strengthening the scientific basis for things in education. Part of that is figuring out which methods are most appropriate to which sorts of questions and building the capacity for people to both make those selections about which questions are addressable in the context by which sorts of methods. Bob?

PARTICIPANT: I mention one kind of take on this, which actually occurred at the World Bank discussions on trials that I mentioned earlier. Someone in the audience stood up and said well we have some choices here that are roughly speaking scientific, I think that you economists should behave more like astronomers, who are scientists too, and improve your ability to forecast a whole lot rather then do these randomized control trials. Well, that’s a scientific choice, we could dump a lot of money into measurement, at which dumping more money into measurement would be good in any case but just kind of do that. Now one of the kind of interesting wrinkles to this is that it is, a kind of a choice is tied to the stage of development of theory, to make the point let me refer back to our experience in testing bullet proof vests, body armor. When, I’ve forgotten what the stuff is called, but DuPont manufactures, Kevlar, in the I think ‘70’s. Typical case, police department would, in this case Oklahoma, Oklahoma City, take this cloth which was like a heavy drapery at the time, drape it over a pig in the back 40 behind the police station, take a large caliber weapon out, and fire that weapon at the pig. It’s a test. The intervention worked in the sense that there was no bloodshed. How many naked defenseless control pigs do you think you need to make a persuasive case that there’s a causal connection here, the intervention prevented the bullet from penetrating? Incidentally the pig invariably died of internal hemorrhaging but there was no bloodshed. No blood. So the point here is that we were able to make that inference because there’s been 400 years, well, the work on ballistic equations, the ability to forecast where a projectile will go is at least 400 years old, we have that, we have that equation, we have theory, we don’t have anything like, and we assume, we’re willing to assume that the weapon functioned properly and the bullet functioned properly. We don’t have anything like that yet in the social sector, the education sectors, and that’s one of the reasons for arguing for these trials.

PARTICIPANT: One thing to consider in terms of the point you just made about this mandate for learning is that there are, just like in randomized clinical trials there’s the intended and then there’s the achieved or the enacted. A lot of people are interpreting this move towards higher achievement and who can argue with that, as that which is reflected upon standardized tests. And in order to drive up standardized tests you have to have teachers follow scripts it turns out, that works. I’m in a program in which we have a technology and teachers track, we’ve got a technology and business track, some of the best students that are going into technology and business are by all reports the best teachers from the schools because they refuse to take the lowest salaries to follow a script to drive up test scores they don’t believe in. So you’ve got a larger construct validity question here, the intent of Congress notwithstanding, and I think those larger issues have to be seen through multiple methods, we’ve got to look at this larger picture because otherwise we could end up giving the impression that we’re making a difference rather then actually making a difference in children in some deep sense.

DR. FLODEN: One more question, Sue?

DR. KLEIN: Thank you, I’m Sue Klein, I currently work in the Office of Innovation and Improvement in the U.S. Department of Education, however on October 7^th I’ll become the education equity director of the Feminist Majority Foundation, and my question to you all today is I’ve heard a lot of discussion of the sort of the final end of the design and evaluation of education innovations that might have a positive impact on education but for example the Office of Innovation and Improvement is a new office, is dealing with a lot of national discretionary programs and some of us see it as sort of feeder to the Institute of Education Sciences, which is looking at more of the causal relationships. So one of the things that some of us have been discussing is to what extent does it make sense for the federal government or others to really fund more rigorous kinds of systematic evaluations that would probably just result in associational claims or evidence to support associational claims, not necessarily causal evidence and then going to the next step after that and saying well, if you have some good associational claims with your particular interventions then it might be the time when you would invest more heavily in the randomized experimental designs and looking at alternative comparative treatments that have shown a pattern of positive relationship from the associational claims. And I was wondering what you all thought about what should funders that are trying to encourage good solutions be doing before we jump to the randomized experimental design or do you think that everything from the little bitty prototype right at the beginning should be an experimental randomized design?

DR. FLODEN: You two will probably agree on that, right?

DR. KELLY: I was going to say that what you’ve got here is a clash of cultures in some sense between the way that a funding agency like the National Science Foundation thinks about its mission, right, and how the field responds. If you look at the mandate of the National Science Foundation they’re talking about mathematics, science, technology, and engineering, K through probably 16, and you’ve got a field initiated studies model, a field initiated studies model means the field comes up with the good ideas. That means that the chance that you’re going to get the progress that you’re looking for, you’re going to be lucky if you can find it. Also you’ve got a culture in the academe where a person is rewarded for being John Wayne, not being for one of the team, and they’re trying to get their own particular research focus. So I think the progression that you’re suggesting, which by the way is reflected in a lot of these articles in the Education Researcher, of looking almost as a product development or looking at a program of research or a portfolio research, that is something that is really probably a consideration for the funding agency, but for Congress in terms of having that vision, it’s a vision I support, but I think that if you leave it up to a field initiated studies model what you’re going to get is a dissipation of your research dollars across a very, very superficial pool and you may end up with very, very little. Considering again the amount of new money coming into the research program a the National Science Foundation years of it would be gobbled up with a really good randomized clinical trial to answer one question. Not that there’s anything wrong with that, it’s just you need more money.

DR. FLODEN: The last word Bob?

DR. BORUCH: I agree with Eamonn.

DR. FLODEN: On that note, please join me in thanking the two of them.