DR. SUBOTNIK: I am going to ask for people in the other room if we have people coming in now.

Okay, so, people, I am repeating what Tina Winters said, people who are in 110 who would like to come over now for interaction with our panelists and with each other please come in.

I am also going to make an executive decision since we started late to add 5 minutes to our discussion. So that gives us 25 minutes for discussion. We are going to ask you once again to step up to the microphone because it is being taped and you can address your question or your comment to our panelists or you can address if you are not the first person, you can address it to one of the other questioners.

So, it is a discussion that the entire audience can be involved with, but you do need to step up to the microphone. Please, also, identify yourself before you ask the question.

DR. LESGOLD: I am Alan Lesgold from the University of Pittsburgh. I think we can learn a lot from looking at how the same kind of problem is addressed in other disciplines. Let me give you one example. I have a bunch of colleagues who are looking at the issue of exercise and weight maintenance. As soon as they [get] beyond 150 pounds then they get to be full professors, but if you look at what they do, there is one group of people who are studying exercise physiology. They are trying to figure out what happens in the body when a different exercise is engaged in. That creates a core theory that can drive the rest of the process.

There is another group of people that are saying, "Oh, yes, we can tell folks things, but just telling them that doesn't necessarily have any effect," and so they spend a lot of time trying to figure out which prescription, which activity, which instructional activity will produce the desired outcome.

We have another group of people who essentially do long-term qualitative studies of what actually happens when you do some particular kind of instruction? What changes in people's lives? And they also do studies of what are people's lives like; what could we change and what couldn't we change? When we look at instruction in schools we often miss some of those things. I think Susan came close to some of those points but we need to understand something about what actually will happen not only in the classroom but at home, what will happen 6 months down the road. Unless we want really tiny outcomes we need a combination of I think essentially epidemiological kinds of studies along with randomized clinical trials.

The randomized clinical trials probably have to be grounded in some amount of causal understanding of basic mechanism. The cognitive mechanisms are often a good source of that. That can be a first justification for doing some randomized clinical trials.

One thing we know for sure from the medical world is simply demonstrating that one thing works slightly better than another thing on average doesn't get us very far at all. It is a small step in the process and I think that notion of can we come to understand with enough detail not only what works but sort of how it plays out over time and how people become able to do it as teachers, how people became able to do it as students and what really changes in their lives, I think that has got to be the overall model. Randomized clinical trials play an important role in that process but by themselves they really don't tell us very much.

Jim Poppins said a number of years ago that if you analyzed all of educational research the effect is .1. We could demonstrate that with certainty but it still wouldn't tell us what to do.

DR. RAUDENBUSH: I would just like to say that if you look at the history of recent educational research as Tom Cook did you won't find very many randomized clinical trials until the last couple of years. So, it is not a question of, you know, I mean I think we have been way out of balance.

DR. TRACTENBERG: My name is Rochelle Tractenberg. I am from Georgetown University, School of Medicine. Actually I teach statistics and I am developing a statistics curriculum that I hope to implement across the scientific disciplines in the school of medicine as well as in undergraduate programs.

One of the questions that I sort of have for this forum is graduate student exposure to experimental design and methodology is completely removed from the K-12 emphasis on curriculum and integration of ideas. In the graduate school curriculum, you have a project. You need to achieve it. You learn the methods that you need to get through that. You take one course and then you focus on whatever your instructional training is going to be. That is the general mode, and so the challenge is to make people flexible with respect to considering different statistical methodologies and multiple methodologies.My question is how to integrate the desires of K-12 educational reform, which is the most well structured of all the reforms that are going on in education now, and integrate that into the graduate experience and as well into the undergraduate experience if people are, as you said, studying math and science and then going to be instructors in that domain having the effort be focused on people who are finished with their training who are professionals is probably a little late. It could be implemented much earlier with a focused emphasis on research in the training of instructors and educators.

DR. BODILLY: I will just make a comment on that. I would be interested in seeing how this panel or discussion goes this afternoon where a policy maker is sitting there, and I don't know who it is but say it was a district level person saying, "Here is my problem," and then talking with different researchers about how to go about solving that, and that may actually be a model that helps you think this through.

We obviously want to train our graduate students so that they are expert in particular methodologies, and oftentimes that means specialization, so they don't go broadly across methodologies. Within a seminar at the graduate level you could bring in the policy makers, in other words serve the policy issue and talk with them about it or look at some of the studies that you will be seeing later today; or you could look in many different journals to find those studies that use mixed methods to answer a very relevant policy question and push the graduate students towards how one particular narrow study doesn't address all the issues needed to be addressed by the policy maker to make decisions. You could imagine a forum or a seminar series that looks at different policy research design and sees where all the different contributions came from.

I am saying don't try to train people across all these multiple methods. Try to get them to appreciate their colleagues and the contributions they are making to the policy research.

DR. SUBOTNIK: When Marty said that the Organizing Committee was having a great time with that session, it is because we were thinking of it as kind of a quiz show where the policy maker would say, "I need this," and the researchers would have buzzers and then the policy maker could reject or accept the proposals, but we were having fun.

DR. MAXWELL: Joe Maxwell, George Mason University. I want to speak in defense of the qualitative/quantitative distinction. I think it is an important distinction. It is certainly a distinction that many researchers themselves feel is important they identify themselves one way or the other. I think it is important because these really are two very different ways of looking at the world and thinking about research. What often happens, and I have argued it is what happened with the National Research Council Report on Scientific Research in Education, is people say, "Well, this isn't really a very meaningful distinction. I think we should just get rid of it and really talk about what we are going to do," and then go on to impose a quantitative world view on the subsequent discussion.

Let me just take one example of that from Stephen's paper. He uses the distinction between kinds of studies of descriptive, correlational, experimental. Now that is a very useful distinction in thinking about quantitative research. It is not a distinction that makes much sense in thinking about qualitative research. Qualitative studies don't fall into that typology(?) neatly.

I would argue that one of the things that gets left out when you start thinking of qualitative research in variance terms, thinking in terms of measuring variables, which is not how qualitative researchers typically think, is that the main strengths of qualitative research get ignored which are the understanding of processes and mechanisms-how things actually operate, the understanding of meaning and how people conceptualize and perceive what is going on and the effect that that has, and the importance of context, not context simply seen as a set of extraneous variables that need to be controlled but as a key part of the process by which an outcome is achieved.

DR. RAUDENBUSH: I hope I didn't divide the world up into just those descriptive, correlational and experimental. I agree with what you are saying completely that there is a wide variety of methods and there are special insights you can get from observation research, detailed open-ended interviewing studies. I used some examples actually of those in the paper, but I do think that the quantitative/qualitative distinction has been perhaps emphasized to the extent that other crucial distinctions are not emphasized. For example, one of the things I argue in the paper is that survey research, quantitative survey research with close-ended interview or questionnaires are similar to open-ended interviewing studies in that the aim is to reflect what is going on, to describe. Essentially it is a passive sort of description. Your goal is in fact you are training people not to have an effect on, not to be an intervention into the world, whereas experimental research, and maybe some kinds of participant observation research similar in this regard, are attempts to learn by actively engaging and trying to change things, pry things out and try to change the world and then you see how the world responds. It is a different way of learning, but I think that sometimes in emphasizing the quantitative/qualitative distinction, which I think is at least partly because of our history in educational research and because of ideology and politics, we overlook other crucial distinctions.

The other one that I mentioned earlier is the distinction about scale. Sometimes quantitative and qualitative distinction really corresponds to people looking at things with a microscope or a telescope but what scale do you want to look at different things. I think that part of the multiple methodology sort of argument is that you need to understand how things work at different scales in order to understand, for example, what might be a good treatment and how it could be implemented well, and then thinking about how to test the causal effects of that first in a very idealized circumstance perhaps with a lot of resources and later on maybe on a larger scale maybe with fewer resources.

So, I think that I am afraid that I agree, Joseph with what you are saying. I just feel that if that is what we mean by mixed methods we might be impoverishing the discussion.

DR. DANISH: My name is Steve Danish from Virginia Commonwealth University. I can remember hearing back when you first start to do research, when you design research you shouldn't start with a message. You need to really get the method and methodology that you are going to use for the intervention together and do those at the same time, and I want to follow up a little bit on what Susan said with that because it seems to me that we are talking about the methods to use to evaluate an intervention and we are not really talking as much as we might want to about the intervention itself. What are the components of a good intervention? Why do we intervene? When do we intervene? How do we intervene? How do we make sure that the intervention is effective and what about the fidelity of the intervention in determining whether it was done correctly and that all leads to the dissemination part and until we really look more closely at the intervention the message we use to evaluate whether the intervention works seems to be almost secondary. I hate to say it in that way but it would seem to me that a lot more emphasis needs to be placed in understanding and designing that intervention.

DR. RAUDENBUSH: I just have to say something about that. I agree with that in general. We can agree with everything, but we have spent hundreds of millions and probably billions of dollars in the last 10 years trying to invent new instructional innovations, new technologies, new curricula and what do we really know about what can we tell people who are now highly motivated and have the resources about what they should actually do to teach math? The NRC just came out with a book on evaluating interventions on what we know about how to do mathematics instruction in middle schools and it said that the evidence base is incredibly weak.

So, I guess I have to respectfully disagree with the last comment. I think we have spent enormous amounts of money on that and what we haven't spent money on is figuring out, is actually learning what works and what we can reliably recommend for people who are searching to improve schools.

DR. BODILLY: Just a comment, an example of the way to get at one of the issues here. Probably everyone in the room knows about the evaluation of the 21st century community learning centers. It is federal funds being given to schools across the country to create after-school programs at the elementary level. Mathematica is conducting a random experimental design and they are finding not very strong impacts and I sort of look at that and some people are shocked by that but when I sat and thought about it I thought that is not a very shocking outcome for that study, the reason being that the only thing that is the common treatment across all of the different sites is federal funds being given to schools to start after-school programs and so in the end we will have spent a huge amount of money on a random experiential design that will tell us that if you throw money at schools you might not have very strong effects.

We would have been better off if we had been more thoughtful and I think this is what Steve is arguing, be more thoughtful, be more planful about what exactly the intervention is.

In the end we may conclude from that evaluation that after-school programs do not work. That would be unfortunate because in fact all we have tested with that program is whether giving funds to schools works.

So, if we had done a better job of developing alternative types of interventions to do randomized field trials at different sites we might have come up with something that would have added to our knowledge of what works in terms of after-school programs to raise student achievement and not come up with the sort of empty finding. I am sort of worried that in the future people will take the wrong finding out of that study.

I don't mean to pick on that particular study but it is the one, it is a recent one where it is coming up with these results and people are talking about what does this mean, and I hope we can get the right conclusion out of what it actually means.

DR. ROLLIN: Good morning. My name is Steve Rollin. I am from Florida State University and this is a question that I am sure has been asked many times before but it seems to me that there is a significant disconnect between the requirements of the gold standard and the nature of the way schools are organized.

When I go to schools and talk about doing research they ask the question, "Where do we fit it in? How are we going to in fact meet your needs of randomizing our classes? What is in it for me?" We are faced with having to meet the requirements of FCAT [Florida Comprehensive Assessment Test] of the daily stresses associated with teaching. So, the question to me is really access to payoff. I mean how do I increase my ability to have access to be able to do the gold standard quality, good quality randomized controls or any kind of mixed methods and provide payoffs that are effective for the schools that would allow them to disrupt their day in a way that would allow meaningful research? It is just kind of a real world question that researchers are faced with every day. So, any thoughts you have about that would be useful to me. Thank you.

DR. RAUDENBUSH: I think it has been said that you can't do or it is very difficult to do these randomized experiments in education. I think it is really just turning out that we haven't really tried very hard. I think they can be done, and they are being done, and we can probably do them a lot better; and part of it is developing a culture as was a very difficult uphill battle by the way in medicine because in the early days, and I am really only talking 50 years ago, it was very hard to convince heart surgeons that they should be randomly assigned, that their patients should be randomly assigned to have or not to have surgery. They thought everybody should have surgery. They were surgeons. So, Howard Hyatt and Fred Mosteller have written about this. They were involved in developing the agenda to have randomized trials in medicine, and it was very much of an uphill battle there, but in the NRC book that I mentioned about mathematics interventions there were 95 causal comparative studies and one randomized experiment. There are now a number of randomized experiments that are in the field that have adequate numbers of schools and have people enthusiastically participating.

I think it does take a great deal of skill, an enormous skill in knowing how to work with people and trying to create conditions where it really is in people's interest and they feel rewarded by participating in the study.

DR. ROLLIN: The payoff issue is a critical one, isn't it?

DR. RAUDENBUSH: Absolutely, and there is a win-win situation for people by participating in these. Generally it is going to be schools and classrooms, most often schools that are randomized not kids.

So, we are beginning to develop the science of how to do this and I think we really have to and we have to develop the culture that makes it possible to do that.

Thank you.

DR. SUBOTNIK: We are running short on time. So, I am going to ask the line to stop at where it is and if you would be brief in your comments.

DR. KOHLMOOS: My name is Jim Kohlmoos. I am from the National Education Knowledge Industry Association and I was really struck by Susan's comments about knowledge utilization because I think this whole seminar presupposes that knowledge ultimately will, new knowledge generated through research will ultimately find its way into the classroom and improve practice. It is probably not such a simple task and I was just wondering if through this multiple methods research approach that the utilization factor might be enhanced. Have you thought about how through this approach utilization by practitioners and policy makers alike might be enhanced?

DR. BODILLY: Off the top of my head there might be two parts to that. Certainly if we understand better from say a teacher's perspective the difficulties in implementation or what supports we need to put out there to enable them to use these new treatments that might be helpful.

So, mixed methods might help you collect and assemble the information so that users actually have what they need to undertake the treatment if it has been proven to be effective, but in addition to that you might want to think about a research agenda that understands what it is that teachers or principals or school level folks really need to change their mind about practice. You could run an experiment on this. Is it enough to read a sheet of paper that is the description? The alternative is to have someone, a salesperson come to your school and give you the sales job or alternatively you go to the school that actually has implemented this successfully and you see it in person.

You could understand through a series of different methods which has the most impact on convincing teachers to try something new. So, even this is susceptible to research. We really haven't done very much in terms of trying to understand what the best mechanisms are to convince people to try things that have been proven to be successful.

DR. SUBOTNIK: That is the analogy to public health that Stephen Raudenbush addresses in his paper.

DR.COOPER: Harris Cooper from Duke University. I want to say three things very quickly. First, with regard to the medical model that has come up a couple of times in relation to the use of experiments. The medical model can actually be used as well to support the underpinnings of today's conference, in that in medicine and public health there are epidemiological studies that both come before and during the course of experimental studies and there are case studies, which go on after experiments are over.

You have two very dramatic examples of that that have played out on the front page of the New York Times in the last 2 months, one dealing with heart attack and arthritis drugs and the other dealing with antidepressants for adolescents and in both of those instances they reversed the decision based upon the experiments or case studies where the experience of clinicians who were applying those things and discovering outcomes as Susan suggested which might have been more long term and different from the immediate ones would suggest that these drugs may in fact be counterproductive.

So, if we are going to adopt the medical model let us adopt the whole medical model and not just the part that deals with experimentation though experimentation clearly is central and critical.

The other question, I want to question a little bit the notion that people in education are averse and I don't want to steal or maybe I do want to steal Tom Cook's comments from this morning about being averse to random assignment because in fact in education random assignment is used all the time in regard to scarce resources. If you speak with superintendents or principals or school board members around the country you will discover that they just started an all-day kindergarten. They had 100 slots. There were 300 parents who wanted their kids in those slots and they used a lottery to decide which kids would get it. There are probably thousands of examples of data sets of that nature sitting around the country waiting for researchers to partner with educators and discover what the causal impacts of those things might be. Finally I want to reinforce Dr. Maxwell's point about not losing the distinction between qualitative and quantitative. I think it is important but I think Steve's notion that it is used too broadly is also true.

I think we confound lots of dimensions when we talk about what is the difference between qualitative and quantitative that go well beyond numbers and narrative, and we really do need to tease those things apart. I hope we talk about rubrics, what the true independent dimensions that distinguish research designs and how they couple properly with research questions will come out, but the qualitative/quantitative one I think really is one that needs to be discussed.

DR.KING: My name is Karen King. I am from the National Science Foundation and Michigan State University, and I wanted to come back to this issue of interventions and implementation of the intervention because I think it is the key and critical part that I was surprised not to see in your paper, but the issue of, for example, you keep referring to the NRC report about curriculum, but the curriculum is not the intervention. The intervention is using the curriculum in this setting with this group of teachers under this scheduling system. There is a whole lot of things and often you end up with multiple interventions. So, you have scheduling with this curriculum, with this professional development regime, with all these other things, and I think without a better sense of what is the intervention in this and then looking at how you can design a randomized controlled trial when you have all of these multiple interventions interacting. We need to talk about what it is to define an intervention well enough to be able to do a randomized clinical trial when it is very hard just to be able to define teaching.

So, when you can't even come up with a good definition of what it is when you see it and can't get five people to agree to it, then it is very hard to imagine how we are going to move forward from there and so I really want to stress that that is a major issue and the issue of the overlapping intervention also needs to be addressed in any kind of RCP.

DR. BIRMAN: Bea Birman from the American Institutes for Research. It seems to me that one of the issues that people have with randomized studies has to do with the outcomes, and I noticed in your paper that one of the points that you make is the better measures of outcomes and you call it the Achilles' heels of education research and I have been thinking about that throughout this discussion and thinking about the need in experimentation to have very precisely defined outcomes and yet that gets into trouble because you either get the lowest common denominator, the standardized achievement tests which people have difficulty with because it doesn't measure things broadly enough or sometimes in experiments that I have seen you have very precisely defined other outcomes that are sort of on the margin or something. So I have a question that sort of has two parts to it. One is sort of about if research is going to be cumulative and also there are scarce resources then do we have to really decide as a research community about what are the most important outcomes and figure out how to develop better measures for them. If that is the case, do we butt up against a whole set of values in society about defining what are the important outcomes and I know there is limited time now but I was wondering if there are some thoughts about that either now or later.

DR. SUBOTNIK: I think we will leave it to later because that is such a great question and we want to be courteous to the next panelists. So, it is ten after ten and if you wouldn't mind just taking a very quick stretch of 5 minutes and then we will get back on schedule.

Thank you so much.