DR. ORLAND: Thank you, Brian. I am going to open this up to our consultants and the audience in one second, but first, Pat can you give some initial reaction to what you heard and A, what might be appealing to you? I am not giving you a buzzer to give an up and down for any of these recommendations, but I would like your initial sense and also any questions you might have either Brian or Kathy before I open it up.

DR. FORGIONE: You have made me feel guilty. I mean you know it is hard enough doing this work when you think you are doing it for kids but when you realize the lost opportunities of what you could do but I have learned. I don't collect data if I am not ready for it. Don't be foolish in my business. You had better be ready for it and you had better be ready to do what is right with it and that is why this is the second generation. This is a terrific opportunity because I can go a lot of different ways. This is a mean accountability system, 29 trip wires and some of them you don't even get out a jail free card by participation at least if you are making improvement. So, one really has to think about it.

Brian, you made me think about it. Recently I had X number of units of reading intervention and I have 17 middle schools. I could take eight of them off the table because they are high performers. So, I have got nine. Do I put one in every school to help some kids to create a culture or do I look at where the test data are and where those schools may fall below the trip wire. You have got to do the latter. If you are not smart you are wasting your investment. You are being bureaucratic but my world is about acting equitably and equity is not equality. Equity is you put the dollars where the needs are but it is Sophie's choice. I need three times as many of those interventions because I am on a fixed income. I give away 158 million being Robin Hood. One out of four dollars the taxpayers give me I give to the rest of Texas and I want all the boys and girls to have equity but you kind of pointed out to me some of the difficulties of these resource allocation issues but this is an opportunity, Kathy and Brian that I am realizing I can do more with this moment. I am going to make my treatment effects tighter. I am going to use this as an example with middle and high school that you don't get the resource if I don't tightly control the implementation because 180 can be wasted, too, if you don't train it and get it in place and monitor it and so I am learning. We are a learning community. We meet and really think things through. There is always a scarcity of resources in our business but can you have the vision of teaching and learning?

So, I would really like to hear what the audience has to say, but I would like to get their notes because

I do have the winter break to kind of look at this, and I visit all six of these communities and I have got to tell them they have either made it and I have decided you are going to stage 2; you are still a blueprint, you haven't gotten the recognition but I want to honor progress, but I want to keep those teachers there. I don't want to have a diaspora where they all leave you know because of capacity and so it is really a challenge, but I am going to go out there and I don't send lieutenants I go out and do this. So, when I closed those six schools I had four meetings with 300 people, and they hated me, but I went out and did it right in front of them and showed them the test data, but the literature says America loves their own schools. It is someone else that is bad, and I showed them the data, and they still didn't care. They didn't want change. Who am I, this SOB that is going to force goodness on them and you know when you start to see progress and you build that community it does have a vision but it is very complex because change is very difficult and I have got a couple of people tracking me. They are going to outlast me and they will take me down and I know it and that is okay, but I am not going to give in to them but that is the whole nature, Brian, I think that you captured and Kathy. This is a very interactive thing. You have got to think about your power steps and where do you use some of your chits; where do you leverage it? My teachers still believe in the reform. I will not lose my teachers because you can't do the reform without teachers, but is hard when you are moving 207 of them, but again as long as you get them jobs and they fit in and then you develop them perhaps it can work. So, I think this has been timely and I thank you, Marty but obviously I don't have any capacity. I do have capacity but they are doing all these federal reports and if I don't do the federal reports I can't get the grants. So, I can't say, "Don't do that migrant report. I would like you to," you know because I have a scarcity of it. The accountability system is in big states, Florida and Texas have wiped out R&D. You remember Frieda Holly in Austin? You know, Bill Webster in Dallas? We had marvelous R&D shops in Texas. It got wiped out by all the heavy state accountability because you had to put your resources into testing instead of evaluation but thank you I do appreciate it and obviously I am a little overwhelmed.

DR. ORLAND: Let us overwhelm him some more. Yes, come up to the microphone, please and as in the past identify yourself and then make your comment or question.

DR.SUTER: I am Larry Suter at the National Science Foundation and before coming to the foundation I was a consultant in industry and there are just a couple of things. I think the point that Brian raised with respect to who owns the question is critical here because researchers own their own questions. It is really hard to work as a consultant in industry because other people own the questions and it probably reflects on where I am in my life at this point in time. When you raised the issue of the 17 middle schools I think the notion of those that are highly under performing and those that seemed to be doing okay, it might map an almost one-to-one correspondence with the regression discontinuity issue that Brian raised. You might be able to use your values where you want to use them without giving up the opportunity to do the research.

DR.FORGIONE: The problem there is even within that distribution I don't have enough dollars. I mean the need is so great between the gap between the performance standards and where the kids are, but you are right at least focusing on that would be a good technique.

DR. SUTER: Brian, also, raised an issue of looking at outliers and I would suggest that you don't. I would suggest that you come in out of the tails. When you are looking for exemplary contrasts that you come in a bit from the tails so that you add to the possibility of robustness in the qualitative data. One of the findings from the older school effectiveness literature is that when you did contrast in the tails and go back to the schools later on the ones that were exemplars didn't have the same degree of performance further out. So, I would come in out of the tails a little if I was going to do that cheaper contrasting.

The third component is any kind of correlational work that you do where you can I would split the data. One of the dilemmas with building correlational models is that you are using your data and testing the data with the same data. So, the degree to which you can have holdout samples to both build data and then test the built model after the fact will help make the results that you need to make decisions on more robust.

DR. ROWAN: What Larry is suggesting in the last thing is instead of looking at the data on the entire sample is put the sample in some way and test your hypotheses correlationally in the first sample and confirm them in the second and that is a good idea although with small samples you could get tricked both ways on this thing and so the issue here is how to do that given small samples but it is generally a good idea.

DR. STRAF: I am Myron Straf with the Academy. Pat, I share Brian's concern that you really don't have the time for research, that the classic research design won't tell you what works in time for the students you have at hand much less for your own appointment. They will contribute to the knowledge base but that I feel may not help you. I would suggest instead and this is a personal way some kind of hybrid between what engineers call evolutionary operation and what statisticians in sequential analysis call play the winner rule where the goal is not to find out what works best but to maximize the amount of time the experimental subject, the student spends on the better methods, if not the best method and that requires you first to constantly change things and I think you do know what works. You have a feeling for what works. So, go in there and start implementing and disturb the system that way. It requires constantly monitoring the outcomes and if you can't monitor the outcomes you have to monitor the outputs that you feel at least are related to those outcomes. It may be numbers of parents that sign pacts or whatever and finally to always move in the direction that makes things better. Eventually you will hone in on something unless it is a wildly chaotic system and unfortunately there is a chance of that. That will cheat the amount of time that we spend on this and it is an experiment in a way but not in the classic sense.

DR. ORLAND: Pat, what kind of monitoring system do you have currently in place in terms of what is actually going on in the blueprint schools and how much do you know and how much of an increment in resources and data collection, etc., would be needed to begin to approximate some of the collections that were talked about?

DR. FORGIONE: I didn't require any additional data collection because I just felt the systems were so fragile but district wide we have been improving our data collection to look at periodically parent involvement, you know, teacher involvement, the development of standards; are we penetrating the classrooms? I have been just trying to, look, that is why if I had data it would be on a district-wide basis and so that may help and that is why I guess I feel a little guilty that I didn't do a little more work just to -- but I was torn between I would spend it all on fidelity of implementation because I knew if they didn't do it right what is the use of even looking at the data. Yet I did want to get some information on did it get better. So, I have been dealing with the outcome measures kind of primarily but I enjoyed that comment about, that is exactly, keep vectoring and my board, I mean I have got my board, 60 percent of our board meeting is on teaching and learning. Now, that is something you have really got to measure. If you can keep your board dealing with teaching and learning on results that is what it is all about and it gives you a chance to kind of keep refining and building ownership because you have got to build that ownership. I mean it can just slip away from you but I appreciate the comment about maximizing time because I did away with block scheduling. I wasn't against block scheduling. I couldn't afford an eight-period day. You can block within a seven but they were into the mind set if you couldn't have four blocks, you know two times four, well, you can block one and two every day you know and three and four.

So, it shows you how the mind set is difficult to deal with. but time is the enemy of the good because it is the most expensive commodity and yet it is the most powerful.

DR. TUOMI: I am Jan Tuomi from MCREL and in the spirit of free consultation one element that I suggest you think about doing that I didn't hear about but maybe you already did was take this conversation outside the superintendent's office and you have some committed teachers and principals who have 3 years under their belts of being your partners in this and ask them what they think worked and why and you will have to filter that through whatever you perceive to be their lack of understanding about what you are doing or what is important and ask them what questions would you like to have answered that would really help you? What do you perceive would help you to move to your next best level of achievement with the students that you are working with? I think if you don't do that you are not taking advantage of a lot of professional wisdom and experience.

DR. FORGIONE: I have enjoyed your standards documents.

DR. TUOMI: Thank you.

DR. FORGIONE: We are doing that. In January we are visiting every school and part of it is for them to critique the 10 components and to give us feedback on what seems to work best in each of those schools. So, we are trying to gather that at this moment to find out what should stay and what should go because I have a lot of degrees of freedom in defining the second stage as long as they don't go low performing in the next testing.

You see that is the other problem. I am going to do all this work and then in April they take a test and in May I get the data and what if they go low performance? I can't run them out of jail -- you know what I mean? Yet you don't get the data in the right cycle to help the policy analyst because my teachers transfer and you have all these moving parts, master schedules of kids, teacher transfers. It all has to happen February and March. Yet you don't get the accountability data and you can look really embarrassed if those data come out wrong and so that is where it is a challenge, but I thank you for the comment.

DR. MAXWELL: Joe Maxwell, George Mason University. I hope I am not throwing just another monkey wrench in here. There has been a lot of talk about fidelity of implementation, how do you ensure that the intervention is correctly implemented? I want to raise the inverse of that and that is how do you successfully adapt the intervention to the context? There is a lot of research in the innovation literature that does that. In order for an innovation to be successfully adopted it has to be successfully adapted. It is not a one-size-fits-all operation here and that is an area where qualitative research can be particularly useful but that is not why I am raising it. I am raising it because I think that is what is going on and we need to find a way to address that issue as well as the issue of implementation fidelity.

DR. BORMAN: I would endorse what Joe is suggesting. For example, we know that Latino children tend to learn better working in cooperative small groups and there may be those kinds of differences that cut across instructions and would help to modify the intervention as Joe is suggesting.

DR. ORLAND: Any further questions?

Yes?

PARTICIPANT: What do you do with regard to your summer program for the, how does that fit into your whole blueprint and how does that factor into your results?

DR. FORGIONE: The summary program is a district-wide program and that is where we serve all children with needs. So, we don't give up on any kid and that is why my board wouldn't tolerate this controlled experiment where you deprive someone of this. That is where we contaminate it but the summer school has been powerfully now aligned for the first time with our student needs working with the children. We also used that as staff development time to develop units with teachers, let them try the units out during the summer school because I think that is a very important opportunity. So, we are trying to link that piece in because we recognize that summer effect can be very powerful towards that.

Marty, could I mention you know you all probably just had an opportunity to participate in all the IES competition. I am in three of them and I must say my first 5 years no one contacted me, and I am pretty well known, and I had my head in the ditch and couldn't quite give people time but two of those proposals people really did come and then no one in Texas. I mean there are national groups that wanted to deal with performance, wanted to know would a fourth and fifth grade reading intervention, you know we get the kid and 97 percent of my kids met the state third grade reading standard, 97 percent. Man, I can't believe it, equal to the state average but my fourth and fifth grade results comprehension gets harder and I couldn't put all the extra money in but I was very pleased that IES allowed that. Now, I will die if I get all three of these because I am sensing you can't win three grants like that but I guess that would be it, but I must admit that I think IES did create a culture that people were asking and those were conversations. I had to decide am I willing to put extra resources in fourth and fifth grade and train mentors or I couldn't have bid in that one proposal; the other one, would I be wiling to give teachers major performance bonuses and create a control group and another group that doesn't know who is in it, to really see does money matter and so I really think the federal effort is working to get attention on issues and I know it is hard because there is not enough money.

The third one was kind of a proposal where we are allowed to participate and learn with them which I appreciated as a policy center. So, I was feeling good about that because obviously they needed sites and at the same moment you know you don't want to get into something that you can't fulfill the obligation but I was very taken back that this competition that just happened did call some of the best people in America to be looking for sites and you have got to negotiate it up front obviously but I think there are districts willing to be your partner if they are in the front end negotiations. The purposefulness of knowing that that fourth and fifth grade reading has got me scared to death and I am willing to spend money to learn what works and it will be really a real tight design which I could never do but I can get away with it now because the board has said, "Yes." Bless them and so I was able to buy my board on the front end because it sounds good but if we get it is going to be hard picking the sample. Who doesn't get it? But I think I can pull that off. So, I think a lot of how you organize your world and communicate and negotiate with us will be valuable. So, I do hope we will learn from this next generation, and I know it is a decade long before we will get any real insights but I do want to say, Marty that something different is happening and it feels better and I know today is a methodological thing, worries a lot. I am more worried about getting some good evidence as Brian said but I am not sure it will be there on time, and I will make the decision without it. There is no doubt. I am going, because you have got to make the decision.

DR. ORLAND: Well, with that I think it is an appropriate closing for this session. Let me thank Pat whom we have all gotten to know a little bit better yet at this session as well as Brian and Kathy and you all.