DR. LEVINE: Thank you both. I am going to also invite our colleagues in Room 110 who wish to raise questions to join us in this room, and again ask that those of you who have questions come to the mike, and start by identifying yourself, and we want to open it up.

DR. GIVEN: I am Barbara Given from George Mason University, and this is a little bit off the specific topic, but while the multiple methods are really I think the only way to go my concern is are you going to have forums like this for school administrators and human subject review boards because you have to get into the schools and I would just invite you to do that.

DR. CROWE: I am Ed Crowe and I am with the Carnegie Corporation and I want to, I guess, direct my comment and question at Dick but really at all the panelists. I drew another lesson from your discussion of cystic fibrosis, which is that there was an outcome that the Centers were working toward and the outcome was defined and people agreed on it. That is so different from the world we are working in where many practitioners, well, where the concept of an outcome is relatively new to education and many practitioners deny that there are outcomes or refuse to acknowledge them and where the programs that prepare teachers and administrators also do not acknowledge that we have outcomes and I think that socialization of education researchers probably plays into that as well. So, I guess in a field where people don't even agree that there is an outcome and if there is what is it and how can it be measured, how do we make progress in this area and again for anybody.

DR.MURNANE: I would like to say one word. I mean I would have thought, Ed that whether educators like it or not No Child Left Behind goes an awful long way to specify what the outcomes are.

DR. LEVINE: Also, I am not sure that is any more of a challenge in looking at educational outcomes than any other sort of behavioral outcomes. I mean there is always that question of how narrowly or broadly to define the outcomes in ways that both are measurable and doable in the context of interventions but it seems that as someone who has worked in areas of crime and other areas of health and well-being an equal sort of debate as to how to clarify that dependent variable. As many of you know I think some of these exceptionalist arguments about educational processes and outcomes are overstated. These are challenges we need to engage in but not idiosyncratic to our field.

Other questions?

I have one then to ask Tom, who I thought did not yet have an identification in the department of economics but was making this sort of economic argument about how we assess and choose both in terms of level of investment and intensity of work and there was something in this model of how you were presenting this that we needed to better engage with determining how we wanted to look at implementation; you sort of raised this as a challenge for the research community involved in implementation. I guess I have two questions. One, do we know or what forms of research could guide you in making those kinds of assessments about implementation methodologies? It seemed like you were giving us also a research challenge about the study of implementation. Second, how do you balance this going broad versus intensive in experimental designs?

I almost saw a mental model of a stratified experimental design that if you could determine from the qualitative work what kinds of things matter then you could stratify that experimental model to control for additional conditions.

DR. COOK: You understood it very well. I do think there is a need for a basic research agenda on the measurement of the comparative study of methods for the assessment of implementation in school. That was one of the major items from my talk. I think the other one gets to your second question, and maybe I didn't elaborate this well enough, which is the sense that if you go to the What Works Clearinghouse-the idea of which I am a 200 percent fan-it is a very sad experience. It is a sad experience even for someone who loves the idea of it in large part because of paucity of studies on any single one topic.

If you look at the recently released one on math curricula at the middle school level there were five studies that met the standards that related I think to three kinds of types of intervention, and it was made very clear that there were 50 or more kinds of interventions out there that schools use for which there are no studies. The maximum number of studies in any one of them was three that met standards. The standards are experimental and quasi-experimental ones-but good quasi-experiments not junk quasi-experiments. I wish that term had never been invented. So, we live in a world where there are very few experiments out there to know what works to help schools modify their practice, whether it is at the classroom level that Steve talked about or at the government's level or whatever level. Now, the implication of that is as a nation we need to be doing more experiments very, very quickly, but if we are doing experiments with 40 and 50 schools and spending 300 and 600 thousand dollars a year in the measurement of qualitative implementation processes then the requirements based on the number of schools for power and the marginally extra funds for very high-quality assessment of implementation of multiple measures is money that is not being spent on doing more experiments.

My point was the need is to do more high quality studies so that the What Works Clearinghouse does not have one study in it for each category and that is about trade-offs. I am not an economist. I hang around with economists more than most people and I actually like a few of them, but I think that is a common sense notion of trade-offs. If you are in a public policy school you are taught trade-offs from day one.

DR. LYNCH: My name is Sharon Lynch. I am from George Washington University and I think my question has to do with some of the conversation that just came from the last response and that has to do with at a lot of universities. We occasionally get ourselves involved in what might be called interdisciplinary research but that still seems to be a far cry from the kind of multiple methods or mixed methodologies that we have been discussing today and certainly the panelists are located in interdisciplinary kinds of positions and have years of experience that allow these kinds of explorations to occur, but I guess I wondered if there had been any thought as to how to get more universities to engage more intelligently in mixed methods or multiple methods design because it is one thing to want to do it but it is another to do it. It seems to me that there are resources and people who actually know how to do this.

DR. MURNANE: I think the only comment I would make is that I think some of the best research firms are quite good at this and in part because they have a very different set of incentives for their researchers than do universities. I mean universities, to succeed at a major university you have to be an entrepreneur. You have got to carve out a niche. You have got to make progress on that niche. The notion of working together with colleagues towards a common end is pretty rare. In contrast the best research firms take on a contract to do a particular job and their incentives are really to work together to produce a good product. So, oftentimes I think there is much better collaboration in the best of those research firms across disciplines and across research methods than there is in many universities.

DR. COOK: I agree with that basic statement but it shouldn't preclude the fact that lots and lots of universities now are trying their best to stimulate multi-disciplinary research. It is at place after place. You have provosts and deans out there beating the bushes trying to do it but in the flat organizational structure of the university they don't have a lot of power to make it happen and so they look for incentives.

The best things emerge out of people who have already got tenure, right, a requirement, and who just like each other and hang out.

Now, that is not the best model obviously. The incentivized one of the contract research firms is that you don't get paid unless you collaborate is another one but it is not portable to universities until we get rid of tenure and I am totally in favor of getting rid of tenure.

DR. TRACTENBERG: Rochelle Tractenberg from Georgetown. I am just wondering, both of you have had very descriptive talks about the complexities in research and I know that a lot of people would like a rubric for selecting which is essentially decision tree. Lots of introductory statistics tests have such things. You know if you have two samples if your outcome is continuous then do a paired comparison or whatever.

I am wondering if you can discuss the complexities that would be inherent in designing such a rubric and how it might best be sort of contextualized into the sort of difficult questions that you both raised.

DR. COOK: Dick and I may be engaged in the same tasks in exactly the same issue as competitors as a matter of fact in that we are both writing books on experimentation and maybe quasi-experimentation that attempts to among other things talk about what are better quality studies than others, and it is particularly problematic in the quasi-experimental field but I feel a great need to bite bullets.

Now that is just in the area of what works. It is not for all the surrounding issues about why work on that topic, why choose this class of intervention, why this instance of that class of intervention, etc.

DR. MURNAME: Let me just say briefly I am struggling with this. I think a lot depends on which of these two models that I described you embrace. If you embrace the first one it is quite straightforward and there are lots of resources including Tom's terrific book, Cook, Shadish, and Campbell that gives you insights on how to do that.

But however to the extent you embrace the second model it is just much tougher. That doesn't lend, there is much less of a tradition on how to think about that work. I think it is really very important but it is newer ground.

DR. LEVINE: We will make this the last question so that we move into the breakout sessions.

DR. DE HAAN: I am honored. I am Bob DeHaan from Emory University. I would like to go back to Dick's point about the sort of structure and culture of departments or colleges of education where it seems to be so difficult and I guess I shouldn't single out just education, in academe in general where it seems to be so difficult to get faculty members who are in the same department or in related departments to work together toward a common goal in terms of focusing the research goals and I would like to refer to two possibilities. One, and I hope the NIH and NSF representatives are listening, during the days that I was doing research in cell biology there were things called program projects that were specifically designed to do exactly what we are talking about, that is to bring groups of six or eight or ten faculty members together to think of common goals in their research, in their disparate research, but also I am concerned about what is going to be coming up a little bit later in the day; namely, the fact that we train graduate students to do research in education in various aspects, and so far as I am aware not very many colleges of education or departments of education are devoting much time and effort to this idea of multi-methods of interdisciplinary sharing of experimental approaches with colleagues and so on, and I wonder if the panel would comment on that?

DR. MURNAME: My experience is if you as a senior faculty member are trying to give honest advice to junior faculty members who clearly have a single objective, which is to get tenure, I have not seen a situation where I could honestly tell a person who is a junior faculty member that it makes sense to devote a lot of time to a project that cuts across several fields because it just is typically not what gets rewarded. I would love to hear what Greg Duncan has to say since he has been involved in several major fascinating projects that do involve a variety of research methods and whether his advice to junior colleagues can be different from mine.

DR. DUNCAN: Probably not.

DR. DE HAAN: But if the only way is through multiple methods, if the culture of the department is what Dick describes, namely that an individual is not encouraged to work with colleagues who have other methods it seems to me there is a disconnect there unless the culture changes.

DR. COOK: I wouldn't tar schools of education, in fact. I would say they are much more supportive of this than are disciplinary departments.

DR. PETERSON: I was going to respond. I am the dean of a school of education and social policy at Northwestern University, Penelope Peterson. I actually would tell our junior faculty to use multiple methods and I mean we have a pretty unique maybe completely unique school of education and social policy in that we have faculty from many different disciplines. Greg is an economist. We have psychologists. We have sociologists. Tom Cook has a courtesy appointment at our school. We have people in computer science. We have people in math education. We have people in literacy, language and literacy and what I tell them, the junior faculty is because they are publishing to publish in the best journals in their fields and to identify the audiences they want to address and publish in those journals. They basically then have to explain to the senior faculty. So, somebody who is working in computer science and artificial intelligence and modeling then is going to need to justify his work to Greg Duncan who might not be familiar with the best journals in that field but they need to be able to develop the rationale and publish in these journals and I think so far it has actually worked.

I mean it is an educational experience for a lot of faculty in our schools who don't know those journals but I think that is what happens when you are working in a very multi-disciplinary context on common problems of human development and learning and how to improve them.

DR. COOK: I think, Penelope and Greg that only works for junior faculty if what they are working on is a very loose umbrella notion within which they can carve out a line of research clearly their own. It might involve multiple methods. It might involve some collaboration but it is clearly identified as their own with their own name on it. So, it is partially collaborative facilitated by the umbrella but it is not deeply intermeshed because that would be a disaster for junior faculty, and the second point I want to speak to is that Greg talks about the importance of the argument that multiple methods are indispensable.

Now, that argument is a politically charged one. If you went to my colleagues in sociology and told them that they would say, "Do you mean to tell me that the epoch-making work by Howard Becker on the socialization of medical students is not worth much or is not a wonderful piece of work in its own right because it was mono-method? Are you trying to tell me that Irving Gossman's work on stigma and the reactions of stigmatized people who don't have a nose to how other people react to them is not worth anything? Are you trying to tell me that the research of Kathy Eden on why is it that welfare mothers can they spend twice as much as they report taking in as income; this was clearly by itself a mono-method study. We can do the same with quantitative studies. The original Rosenthal study was a quantitative study. There is a lot of great science being produced by mono-method research and the argument that multiple methods research is dispensable strikes me as being A, incorrect and B, alienating.

Now, the issue that it can contribute at the margin makes good sense but depending on what kind of multi-method research under what conditions and that has got to be the carefully done not very sexy rationale for multiple method research.

DR. LEVINE: That is a wonderful segue to the afternoon examples. I think it is another divide that perhaps produces false dichotomies between mono and hetero as it were, but education is a ripe area for an examination from a range of theoretical and empirical perspectives and I do see that as an arena that is more susceptible to interdisciplinary studies.

In the afternoon we are going to be discussing training and education. There are some models of training that better prepare junior colleagues like the Ebert model at the National Science Foundation which prepares integrative graduate education and research training around a particular arena and I think in our field we have opportunities for that synergism without having one strategy trump another but balancing when an integration with multiple strategies makes sense.

Tina wants to have the last word.

MS. WINTERS: It is just a logistical announcement actually. I forgot to mention before that lunch is being delivered to the rooms in which the concurrent sessions are being held to allow an opportunity for people to continue the discussions over lunch.

DR. LEVINE: That is right. So, we will reconvene back here about ten after one but your breakouts really can go as long as works for the group assembled.

MS. WINTERS: Okay, if you have a blue dot you want to go to Room 203 to listen about the evaluation of the Second Step program. If you have a red dot you are staying here in this room for a discussion of evaluating traditional and reform math curricula and if you have a green dot you are going to Room 109 to listen to a discussion of the evaluation of the New Hope program.

DR. LEVINE: Thank you all and thanks, Tom and Dick.

(Applause.)