DR. ROWAN: Let me just start by thanking Pat for putting himself out on the line here and being willing to be the problem today. Then let me just tell you a little bit about how I tried to approach this problem. We had a phone conversation and we set up some guidelines. They went like this. We would first discuss some of the challenges and opportunities presented by the presenting problem which is the Austin blueprint and then we might try to sketch out one or more research activities, maybe even come up with like a research program and one presumes since this was at a forum about multiple methods that we would invoke multiple methods in that.

So, you know I want to tell you a little bit about how I thought about this particular problem because it is different I think than much of what we have been talking about today.

First, in my talk I am going to try to be particularly responsible to Pat's questions which he had up there listed. Now, Pat's questions are about the Austin independent schools. They are not about America in general or anything else and the questions are particularly about his context. They aren't about the larger question of how do we solve the problem of low-performing schools broadly across all kinds of school districts but we can think about the relationship between the kind of research I am going to talk about in that problem and maybe at the end I will raise that.

So, with that as background I will just jump right into this. Now, the first thing that struck me about what Pat was talking about, maybe it didn't come through so clearly in his presentation as it does in the materials is that the nature of the school sample that he is working with. One thing I think I have learned over the years dropping into school district offices and trying to get them engaged in research is that they almost always have some practical and urgent need and the practical and urgent need leads them to just find the schools that have that need and start working with them, and it is always an idiosyncratic group. This is for whatever purposes political or whatever you know there is some group out there that we are working with and so in Pat's case these are the most disadvantaged schools in the system who did not meet AYP. That is at least how I understood it.

So, Austin designs an intervention for this group. Now, here is the rub you know. The Austin sample probably isn't what you would pick if you went in there to do research. You know, researchers don't want practice-driven samples. They want design-driven samples and it might involve random sampling of some sort or random assignment of some sort or short of that some kind of a controlled or disciplined assignment of cases to conditions is something that we generally are used to working with.

So, the first thing then is we have got to work with let us just take what he gave he us and try to work with that. What can we learn from what we have in front of us? These schools are at all one end of the spectrum,high poverty, not making AYP and if you start looking around his district for comparisons you can't find any because they are all at the lowest end possible and I couldn't find any. You know you are thinking what is the natural comparison here? So, there is no comparison here. So, from that perspective I think one way to try to address some of his initial questions which are what is going on here, what happened in these schools, what things have unfolded differently had we not done this and the only comparison group we have are these schools to themselves in the past and so we could sit around and we recommend you do the same thing, ask yourself did the scores go up in these schools or not right after the intervention and were they going up beforehand?

Now, I confess to you because I am just an inherent data grubber I spent the better part of an hour looking at scores and trying to figure that one out. I came to the conclusion that the scores were going up prior to this but something else happened, too, just for what it is worth, but once you have got that is there a phenomenon here worth looking at, and work hard and think the best you can but this is not rocket science or statistics. This is looking at the data and convincing yourself. Then you might want to look within each school and try to find out what happened here and so you know I guess this is trying to find through multiple ways of looking at multiple forms of data a story which is internally consistent and to your best satisfaction explains what went on over there. That is my definition of qualitative research.

So, you know if the early implementers in a kind of real world setting are often just some idiosyncratic small sample of schools and we are treating them really as early implementers then perhaps the lesson to be learned is that qualitative research is going to be the mechanism by which you are going to try to understand what happened in that idiosyncratic sample because I couldn't find any control group and no method of comparison other than these people to themselves over time.

However, and this I suppose is where more conventional kind of research discussions we are all familiar with come in, let us say you want to move forward. Let us say you convince yourself by this inspection that something and I think you have convinced yourself and you probably have done quite a bit of this with your staff. You have convinced yourself that something was going on there. Let us go forward. Then the question is how do you go forward and take the researchers along with you and so I would urge that the board, that you, the executive leadership in the Austin schools and the board consider a more systematic or I guess what I would call a more researcher friendly way of proceeding in the future thinking about implementation.

So, here is one possibility. One thing struck me when I happened to be thinking about doing experiments in a bunch of large districts. It turns out most of them have so many elementary schools that in fact it is feasible to have sufficient power to do randomized experiments where schools, you know, clustered randomized experiments. Most of these places have 60, 70 schools. This is probably through the top 150 school systems in America.

So, hanging around the current research environment I started thinking randomized experiment, you know, but the question is could you really do that and so what are the constraints here? One constraint of course is resources. Do you have the resources really to pull this off? I don't think you have the resources to pull it off in its full-blown version which costs a lot of money and involves reconstituting schools and so on and so forth but you could pull off some version of this in a fairly large number of schools.

I think you need to do a power analysis to figure out how many schools you need. My minimum bet is 15 in each group would probably be sufficient if you were looking for a big enough effect, but today if you want to do this in 15 schools now the usual question that confronts people in practice when you say, "Let us do this, let us get 15 schools," after they say, "Yes, we have got the money to do it," is that "I want to do it in all the schools. I want to do it now. I am under political pressure to do this now. I can't wait around."

So, here are some political excuses which happen to correspond to researcher needs. One is to say, "Look, we only have the resources to do this in 15 places, and we know there is more need but we can only take 15 and you know we will give the rest of you guys treatment later after we figure it out."

Now, my method for the other way to do this, well, I have another way. Let me just work through this one. So, if I were going to try to do this I would take, you had you told us about 20 more schools that were down at the low end on AYP. I am going to take like the 30 lowest schools, match them and randomly assign them.

That would be one design and now another political question and the first question is do you have the money to do that and let us say you do, and the second question is do you have the political will to withhold. The third question though is really do you think you have an intervention that is so promising that withholding it would actually be a breach of sort of good practice? Do you really think that is the case? Let us suppose that you don't know. Let us suppose that the evidence isn't strong enough. Then I think you might be able to go ahead with this kind of experiment but let us suppose that it isn't the case. Let us suppose you are so convinced that this thing has to be done now everywhere that you don't want to withhold from the other 15 and the bottom 30. Then there is a lot of probability. More statisticians, and I am not a statistician. I am going to suggest a design which I have been looking for an example to apply. I have a solution in search of a problem I think. A while back I wrote about regression discontinuity designs and I find this fascinating especially for cases just like this.

So, you take some cutoff point in some sample. In this case you might use some measure of school socioeconomic composition, find a cutoff, give everybody below that cutoff a service. Nobody above it gets the service and you have got to have in this design, in a regression discontinuity design the guts to make that cutoff be inviolable and then what a regression discontinuity design does is it relies on the fact that there is some association between the cutoff variable and the outcome.

So, I don't know in Austin what the relationship is for example between SES at the school and mean achievement but in many places it is reasonably strong and so the hypothesis in a regression discontinuity design would simply be you know that relationship where the intercept in that relationship would just vary above and below the point of discontinuity. Of course the slope could vary, too, but anyway that is another possible design that is worth thinking about. So, we have gotten to the point where we have got a couple of designs to move forward in a bigger ways and then the next question I guess I had was you know I read that it is pretty hard to know what the intervention actually is. This was a conversation we had throughout. So, some clarification of what the intervention was moving forward would be helpful.

There are two ways to move forward with complex interventions. One is to just treat them as a complex intervention, put them together as a package and ship them out. The other way would be actually the set of experiments about the components, you know somebody gets different mixes or I am presuming you don't have the time for the latter. You only have time to get the big picture out there. So, I will work with that.

Now, there are lots of questions about the Austin blueprint that I found particularly interesting and they have come up over and over again. Pat and his colleagues want, and by the logic I think we would use in the research community to test the theory of their reform, that is in one sense testing the theory implies that you are going to observe greater effects on outcomes where the intervention gets implemented more faithfully or better and Pat adds a little twist to this one which I think is particularly interesting. He also says, "You know, I want to tweak this intervention. For example, I might want to save money by trimming the number of elements that are involved and so I got into thinking about trimming elements and of course you start trimming the elements and you minimize the burden of those who are implementing and anybody who has ever been in a school knows everybody is implementing too much anyway. So, why burden them with even more, and you might even clear some space to get some other stuff in.

So, the question is how would we stay in the dynamics of implementation; this is the question that Tom Cook was talking about. What is the cheap way of doing this because cost is always an issue and just do an ALAR(?) study, you know. You do the experiment. You have got all the data anyway. They did the stuff. You have your outcomes data. You might just find the three or four places that really seem to be doing really, really well with this and three or four places that seem to be doing really badly. You might want to match them up in some way and then go on in and redo all your qualitative work and try to figure out what happened in these places and generalize from them.

That would be one possibility. A second possibility would be to just go ahead and do a lot of kind of survey work inside a hall of the schools for the treatment and the controls. Depending on how you do the survey work that can be either done cheaply or done well and expensively, but I want to say a little bit more about surveys because in fact I think as a field we have learned quite a bit about how to do surveys of implementation dynamics in schools over the past 10 years or so.

I think we have really made a lot of progress. It is not hard now to build fine instruments that have pretty good psychometric properties and have been shown in one or another study to make a real difference and across a range of variable, school climate variables, school organization including the way that structuring gets organized, teaching practice, teacher knowledge.

So, with those kind of data now you can start using the full complement of quantitative methods that are available to study how implementation variation affects outcomes.

The trick here is to know what you are implementing and to find measures that align to that and now you are back trying to figure out whether variation in implementation is affecting variation in outcomes.

Worth noting of course is there is a big problem of causal inference here, right, because variation in implementation is not randomly assigned unlike the treatment. So, you have some problems you have to fool around with but there turns out to be a burgeoning literature about these problems that you could probably find some people to help you with.

So, mindful of that I might recommend that although you know depending on how much money and capacity is around we could figure out how to do that.

Now, then we get to the do you want to supplement that with something else, and so the something else being these kind of qualitative measures, and I have two kind of ideas about that, no, three I guess. One is you go ahead and do the one I said on the cheap only this time combine it, that is go look for outliers, right, go study outlier dynamics using qualitative methods.

The second would be to take and this is generally what people do to save money, take a subsample of treatment and control groups and do qualitative inside of that.

The third is what I might call responsive qualitative design. As you analyze the quantitative data you are going to have some things you think you don't know or need to know or want to know and you might want to use qualitative research to do that. So, qualitative research I think we already know why it is useful and by qualitative research I mean interviews and observations and documentary analysis and observational analysis and so on. You know in my own work I find it particularly good for capturing serendipitous kinds of stuff. You know, I mean you have a good feeling of how the world works but the world doesn't work like that ever and so there is always serendipitous stuff that you get from the interviews. You will never get that from your surveys unless you are willing to have people fill out 10-hour surveys.

So, the questionnaires are pretty good for assessing whether or not your theory works and a main effect but it probably may not help you explain a whole lot and so that is why I would supplement but again you know there are these questions of cost and feasibility and so on.

So, let me close then with a set of questions about this, about the feasibility of this kind of research in your particular setting but in fact in any large school system. To conduct the kind of research that I have been talking about in my view raises a bunch of questions and here they are. I just raise them. Who will pay for this? Will it be the Austin independent school district? Would the Austin independent school district align with somebody to get some grants and how would they pay for this? How would this work?

The second question is who has the capacity to do this kind of work. I bet although maybe depending on your budget circumstances this is a wrong bet but you have an evaluation unit. Does that evaluation unit have the capacity to do this or are they so overburdened with all the other kinds of evaluations that they have to do that that they don't have any excess capacity?

Then who would you find to help you do the research? What kinds of collaborations are really needed? The third and this is really a tough one and I would be curious to hear your answer to this is can the Austin independent school district wait around for the results of systematic research? I mean do you really have any time for research or are you going to move ahead anyway? Is the need to press on just too great for research? I mean you don't mind being in research but it is not going to drive any decisions because you have got to move on and the researchers are always slow.

Third is the research agenda that I just talked about is about Austin. So, how would we generalize the results to the larger population of school systems and in fact how would we spread the results? What is the knowledge utilization translation experience there and the final question I have is this strikes me as being reasonably feasible in America's largest school districts but it sure doesn't work in little tiny ones, you know; places that don't have 50 elementary schools can engage in these kind of large-scale randomized experiments. So, what is our method for learning directly about practices in those districts? You know, how do we spread knowledge? What do we do? Is this the place for us to do sort of you get coalitions of schools systems working together; how does that work?

So, thank you. Those are my comments.