DR. SUBOTNIK: It is with great regret that Herbert Walberg was not able to be with us today and so I am going to step in and provide a what we call compressed version and if any of you have seen reduced Shakespeare you know that the premise of reduced Shakespeare is to do the entire works of Shakespeare in an hour. I am going to have the audacity of summarizing a wonderful paper by Stephen Raudenbush in 10 minutes.

The format of the session is I am going to start off by introducing our panelists. Then I will do my compressed summary, and the purpose of that is to refresh the memories of those of you who have read the paper and also get those of you who did not-shame on you-up to speed. Then we will have a response by Professor Raudenbush to my summary and then we will have an opportunity for Susan Bodilly to discuss the paper. I am just going to summarize and she is going to discuss at which point Professor Raudenbush will respond to the discussion and then we are going to open it up and we should have time for a lively discussion. So, the format is designed in order to capitalize on and maximum time for discussion.

Let me introduce our panelists. Stephen Raudenbush is professor of education, professor of statistics, professor of sociology and senior research scientist for the Survey Research Center at the University of Michigan.

He is the Scientific Director of the Project on Human Development in Chicago Neighborhoods, an ambitious study of how family, neighborhood, and school settings shape the academic learning, social development, mental health, and exposure to violence of children growing up in Chicago.

Dr. Raudenbush's research continues to involve the development, testing, refinement and application of statistical methods for individual change and the effects of social settings such as schools, and neighborhoods on change.

More recently he has become interested in the assessment of social settings in which persons develop, such as neighborhoods, schools, and classrooms. Evaluation of the reliability and validity of those assessments of these social settings follows from and extends tools from psychometrics as explained in recent articles in Science, Sociological Methodology and the American Journal of Sociology.

He calls this line of work ecometrics or the assessment of ecological settings as distinct from psychometrics, the assessment of psychological traits and states.

Dr.Raudenbush is a fellow of the American Academy of Arts and Sciences, a member of the National Academy of Education and a member of the U.S. Title I Independent Review Panel.

He received his BA at Harvard. his EDM at Harvard and EDD from Harvard.

Our second panelist is Susan Bodilly. She is currently Associate Director for RAND, Education, responsible for domestic research and has worked at RAND for more than 24 years. Her primary research interest and expertise lie in K-12 school reform, resource allocation and its impact on reforms, forward evaluation, and implementation analysis.

She has evaluated an array of K-12 improvement initiatives such as the General Electric College-Bound Program, attempts by high schools to integrate academic and vocational education, attempts by the federal government to return Section 6 schools on military bases to local control and attempts by schools to implement Perkins legislation as evaluated under the national assessment of vocational education. She played a leading role in the RAND evaluation of the New American Schools Initiative. She recently led projects to manage review panels for the National Board for Professional Teaching Standards and to provide analytic support to the Walton family foundation on its choice program. She recently completed co-editing a book on scale-up issues in education and co-writing an extensive literature review on the use of out-of-school time.

She is currently leading an effort to assess the Ford Foundation's collaborating or education reform initiative and she has been the group manager for health, education and welfare, responsible for recruiting, hiring and professional development.

She received her PhD in public policy from George Mason University.

Okay, now, I am going to do the summary. Dr. Raudenbush begins his paper by saying that there are two approaches to multiple methods research, and it is interesting because he uses the terms "multiple method" and "mixed method" throughout the paper. So, I hope that maybe he will say something about that as well, how he might or might not differentiate that and the goal of mixed method research is that research will be more credible, useful or comprehensive than if only one methodology were employed. There are two approaches. One is the ex ante, whereby you would deliberately design a data collection strategy to capitalize on the strengths of each approach and that is probably what we are thinking about in designing from scratch a multiple methods project and he does a wonderful job of describing some examples. One of them is by Laub and Sampson and the beauty of the paper is it has so many wonderful examples.

I will draw your attention to checking on those. Another approach is what he calls a post hoc approach where you look at research and seek explanations or fill gaps in research that has already been conducted with complementary methodological approaches and the wonderful example he provides in the paper is for a series of studies on teacher expectations that began with the Pygmalion effect study of Rosenthal and Jacobson and then was followed by some work using qualitative studies by Rist and then followed by quantitative studies conducted by Brophy and Good.

Why is this conversation so salient now? Marty gave you some really good lead in to this that the questions that are driving methodological choices and randomized experiments provide the clearest answers to causal questions of importance. That is the central notion, and there are limited dollars to spend on educational research and philosophical opponents of quantitative research feel isolated by the current turn of events.

Multiple methods provide a more pluralistic approach-a third way, so to say, that capitalizes on the importance of providing vigorous assessments of interventions.

Dr. Raudenbush identifies some issue priorities for us as education researchers. One is gaps in literacy and the second is low performance on measures of mathematical and scientific knowledge.

Some current policy initiatives that are being conducted to address those problems include providing increased resources, demanding accountability, and providing school choice.

Each strategy assumes that implementation is uniform, that is when we give money to a school district or to teachers we are assuming that the money is given in some kind of uniform way and without that we don't really know what is the effect, exactly, of increased resources.

We have insufficient research to fill knowledge and implementation gaps and each strategy assumes that practitioners and parents will know how to implement the strategy in a way that improves instruction.

These are all assumptions that are held up by the fact that we have insufficient research to fill knowledge and implementation gaps and according to Dr. Raudenbush these concerns should be the central task of applied research in education.

Systematic experimentation should be central to the research agenda and experiments should be used to validate causal effects rather than to identify what is potentially effective.

Other research approaches are needed to define the aims, target populations and promising practices and threats of effective implementation. We have all heard the term "the gold standard." RCTs have been common in school-based public health studies but they continue to be rare in education studies focused on teaching and learning; given the fact that they have been common in public health obviously it is possible to pull them up.

In order to accommodate the difficulties of RCTs some evaluators have become very sophisticated at controlling for confounding variables and yet the challenge is substantial. There are always confounding variables.

Dr. Raudenbush also argues that school leaders and teachers will participate in randomized controlled studies if their needs and concerns are addressed in the design phase and they are convinced it will lead to improved learning.

Randomized experiments are the gold standard. They are necessary but not sufficient and Professor Raudenbush argues that, before we invest in rigorous large-scale RCTs, we need precision in defining the outcomes we want to pursue, which includes which students to target and which settings to target. Why? Because students with particular needs are often of greatest importance to us in for example, closing gaps.

The randomized experiment becomes a powerful tool for warranting causal effects after (and the bold letter and the italics are mine, but the language here Professor Raudenbush) a rather protracted process has identified the most promising interventions to change the most important outcomes for target kids in settings of interest.

Before conducting an RCT, we need to apply quantitative and qualitative methods to define the goals of the intervention, develop assessments to measure outcomes and validate these assessments.

Otherwise we may be measuring the wrong outcomes with high reliability or the right outcomes with low reliability or worst of all not know what we are really measuring.

RCTs are expensive to conduct and funding as we know is limited. Expertise is needed to judge not only how but when in the research process to conduct RCT. Research from a variety of methods conducted at different stages ought to be a prerequisite for the construction of a large-scale randomized field trial.

These are some examples of targeted populations of interest [shown on slide]. These and other groups may thrive or flounder in instructional environments that are effective or ineffective for others. Therefore a variety of research strategies are needed to identify the most promising interventions for these and other target groups of students.

Putting the pieces together, how the research is planned including the application of an RCT will affect the quality of the evidence. Concurrently research using other methods is essential to define the outcomes and validate assessments of those outcomes, test the feasibility of implementing the intervention in ordinary school settings and clarify the subsets of students with most need.

Implications for support of multiple method research: To maximize effectiveness, we need to form a relatively cohesive scholarly community; and the purpose of that cohesiveness would be to provide each other with creative ideas and responses as well as constructive criticism and that should be across disciplines and specialization.

Another implication is appropriate research training which we will be addressing later today, to maximize effectiveness we need adequate funding and Professor Raudenbush argues that if we strengthen the quality of applied research on interventions that address the kinds of educational problems that policy makers and the public are concerned about we may stimulate more research funding support.