DR. COOK: Hi, it is a pleasure to be here. We have only got about 10 or 15 minutes each. I am going to get right into it. I am going to talk today about using multiple methods within experiments that take place in schools.

I take it as obvious that in planned and sporadic programs of research that we would and should involve multiple methods. We ask about such things as the nature of the educational problem that needs ameliorating, what kinds of populations are afflicted, about the classes of intervention that might be relevant, about how well a particular intervention is implemented, about the subgroups that might be affected or not, about when it is that an intervention does and does not seem to work, about how the intervention is modified when it is disseminated across schools, and there are many, many other things that predate and condition a basic cause-and-effect relationship even when you do an experiment.

Since these are quite different kinds of questions and since in science we choose methods to answer questions it is inevitable that even surrounding just experiments will be in a multi-method context.

Now, today I want to talk about the rationale for adding comprehensive qualitative measures to experiments and what is the rationale for it, and I need first though to talk a little bit about language.

In traditional analysis there are many, many different kinds of procedures that are called qualitative: ethnography, open-ended interviews, structured and unstructured observation, documentary analysis, theoretical analysis of latent forces. These are all different qualitative methods that have quite different characteristics and skills that you need to exercise and they are quite heterogeneous. The same is true with quantitative procedures: the one-shot survey, the multi-phase survey that results in longitudinal data analysis including time series, the quasi experiments dear to my heart, and also the randomized experiments. This is a very heterogeneous group of methods all loosely lumped under quantitative.

Now, this leads to some confusions of language because you can have multiple method research within what is traditionally thought of as a qualitative paradigm for example, there are lots and lots of ethnographies that include documentary analysis of documents to complement them and multi-methods remain a qualitative tradition and the same is true with quantitative ones and so these categories are very, very fuzzy. For example, observation can be very loosely structured or it can be tightly structured that leads to the observation that is being coded and then subjected to quantitative analysis.

Moreover this use of loose language like multiple method research supposes if I were to combine any one of these quantitative methods with any one qualitative method then the phenomena involved, the processes involved, the trade-offs involved would be the same as if I substituted another quantitative measure or another qualitative measure.

So, it is probably better to speak about specific quantitative procedures and specific qualitative procedures rather than talk too loosely and I am going to talk about the experiment and the use of a number of different qualitative measures within experiments.

Now, we know that multiple method research is most needed when you want to ask different questions within a single project that require different methods. Now, in an experiment what are some of the questions over and above the basic one, which is, “Did the intervention cause a particular effect or series of effects?” Well, we also want to know about the description of the quality of program implementation. How faithful to theory was the intervention? We also want to know about how the intervention is seen at the ground level up. What is the phenomenological experience of it individually by teachers and jointly by the collectivity of teachers about this intervention? We also want to know reasons about why particular kinds of implementation levels are achieved and not achieved. So, there is an analysis of why the implementation levels differ as well as how they differ.

We, also, want to have an analysis of the theoretical processes after implementation and prior to the main outcome of interest. In other words, is the substantive theory behind this program well achieved in practice? Is the process it postulated the one that leads to effects and we also want to know a lot about subgroups, which subgroups are affected positively or not and which ones should we somehow get to look at in our analyses of the many that are available.

Now, these questions here are not at all experimental questions. These are additions to experiments, and, as such, we would expect them to require different methods. Are they important to answer questions about implementation, reasons for variation, levels of rotation, whether the theoretical process is implicated or not in a subgroup? Absolutely these are very important questions. Is there a lot of uncertainty about them that needs to be reduced and, therefore, we need to invest heavily in these questions? That is a key issue. My presumption is, it varies a lot from one area of education to another, but my guesstimate is that across areas there is a lot of fairly weak substantive theory. There is a lot of program design, which is haphazard and not very well thought through, and there hasn't been a lot of experience over 30 years with looking at the quality of implementation of interventions.

So, I presume that these are important issues there is a lot of uncertainty about them which requires therefore deep analysis of these issues.

So, off the top of the head, the rationale for multi-method research is that the methods for answering these questions will add to the yield from experiments, at least in the four areas I must mentioned, but how valid is this rationale? Well, there are many different kinds of so-called "qualitative measures." We have Cadillac’s and we have Yugos. For implementation for example we can put ethnographers into schools day after day who talk to teachers, observe things, talk to administrators, observe things, talk to children, observe things, talk to parents, observe how parent groups meet. They can go to staff meetings. They can sit in classes.

There are many, many kinds of ethnographies but they are all very intensive. You can of course restrict yourself to open-ended interviews with staff or some students. You will get a lot of implementation of theoretical process or you can do classroom observation whether you do it structured or not. You can do classroom observation to see what teachers do and you can analyze school records. In other words you have a multitude of options to choose from as to how you get at implementation process.

Some of them are better than others. Some of them are more obtrusive than others. Some of them are more expensive than others.

Now, of course, the randomized experiment is predicated on doing the very best thing to answer the basic causal questions of X causes Y and if you want to use the best methods for answering questions about the implementation and the process then you might be inclined towards the Cadillac methods which turn out to be more expensive.

It is also the case, however, that you can answer these questions of implementation and process quantitatively. Qualitative methods are not a requirement. They can be answered also quantitatively. You can use close-ended interviews with teachers and other staff members and you can code them. You can do close-ended classroom observation and code it. So, you have quantitative measures and qualitative measures both that can get at these important questions that the experiment itself in its basic framework doesn't speak to. The issue always is how does one select among the various qualitative methods and how does one select the qualitative versus the quantitative methods.

Now, in my reading of this literature I think the qualitative methods are much better for some things in general. They are better for giving an in-depth view of what goes on. That favors qualitative but much better for elucidating the subjective meanings of an intervention and outcome in a process than are the quantitative methods. They are much better in their ability to give you clues about unintended side effects, unexpected events than are the quantitative. Whether they are better for identifying how well individual schools implement something or which classes implement better than others I am not sure of. Whether they are better in terms of being less obtrusive I suspect usually not. They are usually more obtrusive. Whether they are more expensive, that depends. Usually they are considerably more expensive.

So, the rationale for multi-method research within experiments is that the increment in uncertainty reduction about important research functions that the experiment does not speak to is worth any losses that are incurred because of the extra obtrusiveness or the extra cost that is associated with adding qualitative assessment to a quantitative framework like experiment.

Now, to evaluate this more modulated rationale you have to consider some trade-offs and there are many and I am going to consider only two.

In my own work on trying to evaluate Jim Comer’s school development program in three sites-and here I am going to talk about two of them in Prince Georges County to get at these processes that are important over and above the experimental framework-we did annual interviews with the program facilitator in school, with the principal in each school, and with teachers in each school.

We also did a close-ended questionnaire just with the teachers. So, we had both qualitative and quantitative measures of getting at the quality of program implementation. Using these fairly Yugo-like qualitative methods as opposed to Cadillac was probably 5 percent of the budget or less spent on them.

In Chicago on the other hand we sent ethnographers into all of the schools 4 days a week for 2 years and 2 days a week for 2 years and they did the things that ethnographers usually do. We also had close-ended questionnaire measures from all the teachers. Here in the Chicago study, 40 percent of our total budget was spent on qualitative assessment and as to the assessment of implementation it was 90 percent or more of the dollars spent to assess implementation. So, what did we find out? We found out that when it comes to looking at which schools implemented better the qualitative and quantitative methods came to exactly the same conclusion. Charles Paine who did the qualitative part and I sat down every year, and we rank ordered the schools in terms of qualitative implementation and every year we had a 90 percent hit rate or more.

So, for this one question, and I am not saying that is the only question, Charles did much better than we did at being able to explain why things happened as they did in schools. He helped us guide which teachers to look for who could have implemented better. So, some things were better but other things weren't. It is a difficult trade-off but the big question is: Should you prefer the more intensive methods if for some major purposes the less expensive qualitative ones or maybe the less expensive quantitative ones do as well?

Now, let us consider trade-offs at the policy level. The Institute for Education Sciences has launched into many school level studies. These are studies which for power analysis reasons require 40 to 50 schools in each of them, a lot of money, 40 to 50 schools and many studies followed multiple years and it is a logistical headache to recruit that many schools of course.

Now, you add this intensive classroom observation, often many times a year all to be coded and analyzed. You add to this qualitative measures, interviews with teachers, administrators, analysis of records and what you find is you get triangulation on many things. That is very, very good. Some of these methods add information that the other can't get at but the costs for the qualitative component are pretty high. I can't estimate them exactly. I tried some back-of-the-envelope stuff. It depends on many other factors but these are quite a lot of money going into the analysis of implementation and theoretical fidelity.

Now, part of what is happening is that we do not know which qualitative methods are better than others or satisfactory enough for getting at implementation process nor do we know how various quantitative ways of doing it relate to various qualitative ways of doing it. We are not in a good position as a nation to know which ways of assessing these things are better than others and so there is tendency to throw in the kitchen sink at expense and of course we also as a nation need to know how it can be that we can set up studies so we can get the same power with 25 schools instead of 40 or 50 schools.

Now, the conclusion of all this is that I am not a Neanderthal. I want some measurement of implementation of process in almost every experiment at this time in the evolution of educational research. The problem is I don't know which ways of doing this are better than others in general. I don't know which ways are better than others for a fixed budget. I don't know of any research being done on this issue but I know that advocates of multi-method research within experiments cannot make the case for their predilection without explicit discussion of these trade offs that I raised.

That is it. Thank you.

(Applause.)