DR. LEWIS: I too would like to join the other presenters and give my thanks for being invited to participate in this forum. We were asked to respond to four questions, so my presentation will be my response to those four questions, and being a former high school math person I added another question, so I’ll respond to five questions. And throughout my brief presentation I’ll also make comments about some of the things that we’ve heard so far today.

The first question was how do urban districts generally perceive the push for scientifically based research. When Lisa, when I saw this question from Lisa I smiled because this was a topic of discussion that we had this summer. One of my responsibilities is to meet annually with the research directors of our counsel member districts and so we had someone from Westat lead a discussion about what scientifically based research meant and how it would impact the school districts. And interesting enough it’s not very high on their radar screen right now. I was telling Lisa I had to go back and use some skills I hadn’t used in a while to get them to really engage in a meaningful conversation about this. It’s probably not very high on their radar screen because of NCLB, but it really isn’t and at this point perhaps because a lot of grants have not been let at this point they’ve not been approached by a lot of different people. And the other thing I think many of them feel that they know what scientifically based research means because the publishers unfortunately have been telling them that their programs are scientifically based all along.

But we know that most programs and strategies and textbooks fall short, that the research just isn’t there, there’s a lot of research on beginning reading and we’ve been hearing about that, but when we talk about English language learners and we talk about older readers and that, one of the things I’m going to say to you in some of this is that much of the research that’s been done recently, and in the past, they don’t meet our needs, there is no good research on how to educate what they’re calling the struggling reader, the reader at the ninth, the person at the ninth grade level who reads at the second or third grade level, so there’s no research on these things and so that’s some of the conversation that we had.

But I did get them to elicit some items that they feel should be included in the components of randomized field trials, and so I’ll quickly go over them. They said if they as a result of some conversation they felt that it was very important that when people come to them and talk about the fact that they had a scientifically based program that they would look at the treatment size and they said many time the treatment size is just too small, it’s just too small. The demographics don’t look like them, remember I’m talking about large urban school districts, and so one of the things that we’ve learned from some of the studies that have come out very recently is that large urban school districts can’t jump on the bandwagon and do what others do, we are different, we have to behave differently. And so when we see research now and you don’t include from a large urban school districts we’re not going to buy those products because we know that what happens and it’s effective in one district, especially those small districts like Hantrannick(?), Michigan and they have 2,000 kids, it’s very different when you talk about Broward County, Florida that has hundreds of thousands of kids. They’re going to look for control group, they’re going to look for independent assessment.

One of the conversations, one of the arguments that we continue to have with researchers when they come into school districts, particularly if they’re program researchers or they want to judge the efficacy of a program, they always want to use their own assessment, they always want to use their own test. That won’t work anymore, we’re not buying the own test, you must do an independent assessment of it. And unfortunately as Wesley said and others said the name of the game now is the state assessment.

Other favorable components that they wanted to make sure that they saw was, and remember I’m talking about large urban school districts, so they would want to see a significant rate of progress, the old research that we used to say one year, growth for every, one month of growth for every month that you’re in school, that no longer works, it has to be some type of accelerated program. They’re going to look for a cohort analysis, the sustain effects, we’re working with one of the companies at this point trying to conduct a study in one of our districts, in several districts, this particular company came to us and said that they wanted to be able to show that this particular program worked in many large urban school districts. And one of the things that we had to convince him of was that we needed to have some evidence of sustain effects, we know that a program may work for the first year or the second year while you’re in the schools but what happens to the child after the treatment, so those are some of the other components.

We’ve talked a lot at this meeting about the quantitative and qualitative data, a good description of what makes the study work is extremely important, particularly when we heard today about the Haan Study, and the Haan Study talked about using the best teachers. Well, if I were in a school district and I saw something that said they used the best teachers, I’m not sure that that study, that project would be appealing to me because I typically don’t have the best teachers, I have teachers in my school districts, I don’t have the best teachers, so I would be really concerned about that particular program when you talk about replication of the program. And of course conducted by a third independent party and you know enough about the disaggregated data.

You’ve also talked about the ethical and legal standards around randomized experimental design and so are you going to look at the differences in preschool programs or are you going to look at preschool and non-preschool programs? Those are two different kinds of studies, we feel that preschool programs work and so it talks about denying something that works to all students, you’ve had that conversation today, so I won’t go into it anymore but many of the districts would not do, they might look at the differences in preschool programs but they wouldn’t look at the second one.

I was also asked to respond to what are the benefits of the trend towards more RFT’s. Of course more effective programs and of course improved student performance, that’s the bottom line.

But then the drawbacks. The drawbacks is going to be time taken away from instruction, and particularly in the question of NCLB, you’re going to be going to the same schools or the same districts, again, I work with the largest urban school districts in the country and you’re going to be going to those same school districts. And many of the researchers are not skilled in working with people in urban centers. We are difficult to work with, yes we are, but many of us have been burned, so we have reason to be difficult to work with. The focus again is not on issues that are important to us, they’re on issues that are important to you but not necessarily to us. And staff time, when I talked to people and when I was in a school district myself, we had staff who spent an enormous amount of time working with the researchers in order to make sure the project worked and to get the necessary data.

What kinds of capacity are required from urban districts? I think you’re going to need staff with some research knowledge to review and interpret the preliminary findings, to determine if the topic is a good fit, to determine if the research can be replicated because I don’t think we’re going to buy into research that can’t be replicated, that’s going to be one of the critical pieces. And to identify program challenges, I rarely, I don’t think I’ve ever read a program that didn’t sound good on paper, they all sound wonderful on paper, but the devil’s in the detail and in implementation. There’s one particular program that I can think of that many of our districts are beginning to buy into, it reads extremely well on paper but for the content area experts that we’ve worked very closely with they say that in order to implement that program it’s very heavy with professional development and that you have to have teachers with a certain capacity or knowledge or skill level before you can even introduce the program, so those are critical pieces that I wouldn’t want a project that I couldn’t replicate and I can’t replicate it if I don’t really know what the program is, I don’t know about replications unless I have good understanding of what the program challenges and designs are.

I want to go back to the piece about good fit and take another view at a comment that someone made earlier about we wouldn’t be interested in, I think the comment was about teaching statistics or something like that because it might not be a part of the state standards or etc. I would probably look at that slightly different because the advice that we give our districts is that if you strive for higher standards, for instance Charlotte Mechanberg(?), which is a very successful school district, if you look at their math scores or their particular individual math goals, you will know that their goal was to get more kids in AP courses, their goals was not anything around their state standards, that was one of them, too, but they’re major goal was AP courses. So what did they do? They revised their entire K-12 curriculum, so they increased the rigor of everything in order to meet that goal and in increasing the rigor of their program and having prerequisite skills to algebra at a much lower level then they did indeed do better on the North Carolina end of course exams, so you can look at that another way, too.

The question that I included that wasn’t a part of this was what kind of capacity are required from the researchers. Talked about the ability to work in urban centers, to work on mutually agreed upon topics, I think these teams got to look a little different then what they are, you can’t have all white teams coming in there looking at all black kids, that makes us suspect, might not need to be, may not be a reason to be, and you can’t say, it’s no longer good enough to say you can’t find the people, you can go to AERA, the Black SIBE(?), the Special Interest Groups on Black Education, I can’t remember the proper name of that but anyway it’s full of people of color who would be interested in working with you so that’s not going to be an excuse. But I for one would not let you in my district if you didn’t have a racial diverse group. And competent, too, it has to go hand in hand, I don’t want just people of color but they’re not competent but you can get both of them if you take the time to look.

We talked about incentives, incentives, you already heard about financial and paying teachers and etc., co-authorship, I can give you Broward County, Houston, Atlanta, Charlotte Mechanberg, all of those places have outstanding research units and they would love to work very closely with you to do real research around issues that were important to both of them.

And finally I’ll end with my concerns, I’m concerned that people will be looking for a silver bullet, that even though you may have research that says these programs work we all know when you get them in the individual schools and you try to go to scale it may work in one school but not in another. I’m concerned about faithful replication of the model, particularly when you don’t have enough information about the contextual variables that make those programs work. The council, my organization in particular we’re more concerned about what happens at the district level when you talk about going to scale. And finally there’s a lot of good work going on in the districts and I’m afraid that much of this will be lost when it comes to additional funding because they won’t have the so-called research that says that it works.