DR. MYERS: Thank you for having us today. I’m going to describe a study that we call P4K, it really stands for the Power For Kids Initiative. I think this presentation is going to be a little bit different then usual, typically you hear the evaluator talk a lot about the challenges they face. Today Donna is going to talk about because she’s a partner in this, both on the intervention side and the evaluation side, from their perspective a lot, and I’m trying to save as much time as we can to hear from the school side as a participant in this rather then hearing the war stories that most of us evaluators have, we can all talk about being accused of being part of the Klan or doing something like Tuskugagee(?) experiments, we’ve all been through that. But I think Donna is going to provide a unique perspective here.

First let me describe the actors in this evaluation, it’s a little bit different then most studies. There really are five key actors here, the first is the Haan Foundation, which is a small foundation out in San Francisco that has an interest, a strong interest, in developing an evidence base for reading research. And they’ve organized this evaluation, we’ve obtained funding from the Department of Education and the Smith Richardson Foundation at this point. To give you a sense of scale, the evaluation is in one community, it’s about $5 million dollars for the evaluation component, what about $3 or $4 million for the intervention component? I’ll tell you more about each of these as it’s set up, it’s running over four years.

Besides the Haan Foundation Florida State University, specifically Joe Torgason(?) is instrumental in the evaluation and the intervention, particularly in designing the interventions. Mathematica is leading the impact study and the data collection. American Institutes for Research is leading the implementation study where they’re looking at fidelity. We’re going to talk really today just about the impact study and the interventions. Allegheny Intermediate Unit, which is clearly a partner in this, this is where the evaluation is taking place.

I should also note, and this is a little bit of a context for this evaluation, there’s another set of key actors here. It’s a supplemental study, we call it the FMRI study, Functional Magnetic Residence Imaging, what it means, brain scans. A small subsample of the students in the study are receiving brain scans, both before and after the intervention. I think they’re going to be three brain scans, one follow-up a year later. And that creates some issues when you’re out there trying to recruit sites. The idea, Judy is looking at me like what the heck is going on here, that’s how I looked at it the first time, too. You ought to try selling that to school superintendents, how would you like to have your kids scanned? I’m not going to go into the FMRI study but it is brain scanning, they’re looking at different parts of the brain to see how it’s extenuated during and after the intervention.

The thing I want to talk about a little bit today are the context for the study, I want to give you an example of what the study design really looks like. It’s a bit unique, and this is ongoing. A lot of people are using past tense when they’re talking today, we’re not, we just randomly assigned the schools two weeks ago for this so we’re in the field and I think maybe the presentation will reflect a little bit about that. It’s not a well crafted presentation where we’ve had weeks to think about this, it’s happening as we speak. Then we’re going to talk about the challenges, both from the evaluators perspective and the participants perspective.

Context. About 40 percent of the nation’s fourth graders haven’t reached their appropriate reading level. You think of Reading First as a program, we still suspect that even after Reading First, if it is effective, that there will be a large number of children that still need some kind of remedial reading intervention when they hit grades three, four, and five. Now the point is is that most public school interventions that we think of at this point, their goal is to improve kids reading, even from a remedial standpoint. Joe Torgason has come along and said but it’s not enough. There’s typically a standard deviation difference between these kids, those that truly need more reading intervention when they hit third and fifth grade and those that are normal readers. His goal was to show that certain kinds of interventions could have very large impacts and maybe bring these children up close to grade level within one year.

As a fall out from that we’re looking only for very large effects here, Judy talked about five percentage points, ten percentage points, I know in education a lot of times we talk about a tenth of a standard deviation as being meaningful. Here we’re looking at one level where a half a standard deviation impact, which effects the number of schools, the number of students that you bring into the study.

The questions that we’re going to be addressing, somebody, I think Rich Shavelson said earlier you got to get the questions right, the design falls out once you get the questions. Really there are four basic questions, can remedial reading interventions make a substantial impact, and here we’re talking at least a half standard deviation, on the reading achievement of children? Can the interventions effect all critical reading areas? Now our interventions focus more on the phonics side of this but you don’t want to lose sight of the other elements of reading. I’m not going to go into reading too much because Jack’s going to correct me here pretty quick. Do some students benefit more then others? Classic subgroup analysis, here we’re looking to see if certain students, those particularly at the bottom of the reading distribution, and these are struggling readers we’re working with but those that are even close to the bottom, if they improve as much as others. And what intervention approaches make the most difference and for whom?

Alright, let me tell you a little bit about the interventions. We have four interventions that are being looked at in the evaluation, there are two classes of interventions and two within each class. The interventions came about through an RFA process. Last year the Haan Foundation held a meeting out in San Francisco, I think 15, 20 publishers were invited to show their wares, show us your evidence. You’d be surprised what kind of evidence you got from the publishers. Some of them had done small randomized trials, it’s interesting, there’s a lot of pre/post design work in there where they show it. The one that always struck me the most, I probably pick on him too much, what impressed them the most was the spelling test from one child, that was an example of how effective their program was. Anyway, we went through an RFA process, show us your evidence, and we ended up with four different interventions.

It’s a pull-out program, all four interventions tried to fit into a basic structure, had to be a pull-out program with three children, are taught by one teacher, one hour a day, five days a week, the goal is to get each child about 100 hours of instruction, so it’s about a six month program. Tried to factor in snow days, all of that into this. The children selected for this are basically in the bottom 20 percent of the reading distribution, so these kids are really struggling as they hit grades three and five.

Part of the reason for looking at grades three and five, you might say well why not three and four or something like that, I think in the reading area a lot of people think that if you hit grade five and you have these kinds of problems it’s really tough to make a difference for them. Grade three there’s still a chance, grade five it’s really getting difficult and so the team wanted to see if we had the same effect for both groups.

The reading instruction emphasis of the various interventions, one is a phonemic word level interventions where you’re trying to establish phonemic awareness, decoding skills and reading fluency. The other is the word level plus comprehension approach, we also have two interventions there.

I know I’m going fast but Jack swore he’d put me down if we took too long.

Let me tell you a little bit about the design, the Power For Kids Initiative. I’ve already talked, well, let me tell you about the design and then I’ll talk to you about the recruitment. Our power analysis, which we used to tell us how many schools and how many kids we need in the evaluation, suggested that we wanted to randomly assign about 40 schools to the interventions, so we have four interventions so we’re assigning ten schools at random to each of the interventions. We’ll be looking at two classes of intervention so each class will have 20 schools to it.

Now, you might think that’s simple to randomly assign 40 schools, it’s not. What we ended up, and you’ll see the word here units, school units were assigned. When we went in to this we were thinking of course the stylized design where every school is teaching K through five, K through six, for example. I don’t know how many schools in Allegheny are able to do this but there are quite a few schools that teach K through three. We’re trying to have the intervention for kids in three and five, there are other schools that are four through five, some schools have too few students in grade five but they’ve got a bundle of them in grade three. You’d be surprised how the enrollments go up and down year to year, it was shocking. So what we did was we paired up schools that were close to each other. Why did they have to be close to each other? Because the same teacher who was providing the instruction in one school would have to drive to another school to provide it, so one teacher would provide instruction to four groups within each school unit a day. So we randomly assigned these 40 units to the interventions, a lot of work to pull this together and Donna is going to talk about this. One of the critical ingredients of, and I’ll mention this, of the design, is we have a local site coordinator in the Allegheny Intermediate Unit and this is about 40, over 40 school districts, who has contact and has a history of contact with the school. She’s been instrumental in helping us gain access to the schools and with the study. I’d hate to think trying to replicate this study in other places because it’s so conditioned on having somebody like this to help you.

Within each of the schools that’s been randomly assigned to a different intervention we then asked the schools to identify their struggling readers, about 20 per school. We test them, we identify an eligible group, those below the 20^th percentile. Among those students we randomly assign some to the intervention group, enough to construct the instructional groups, and some to the control group. So it’s two levels of random assignment here, you’ve randomly assigned schools to one of the four interventions and then within schools you’ve randomly assigned students to either the intervention or the control group. This way if you look across all the control students it gives you a control group.

Every school gets to participate in an intervention this way, you’ve never denied a school the opportunity to participate and to receive an intervention. But some of the students within the school go about their regular school experience, counter factual. It also allows us with this design to compare intervention classes, we can compare class A against class B at the same time.

Donna, do you want to say a little bit about the, Donna is going to describe a little bit about the Allegheny Intermediate Unit so you have a context there and then we’ll start talking about challenges.

DR. DURNO: The word educational service agency is a generic word but there are 600 of them in the United States, as you know probably that there are about 15, between 15,000, 16,000 school districts in the United States. They go by different names, IU, intermediate unit, and that’s what we’re called in Pennsylvania. Stands for we’re intermediate between the state and the school district, we’re constitutionally requested to provide certain services. BOCES(?), primarily in New York, Board of Cooperative Educational Services, Education Service District, Education Service Center, County School Offices, they actually serve, 81 percent of the students in the country are served by one or another educational service agency and that includes private school students as well as public school students. And 79 percent of all the staff in the nation are covered by an ESA. And you might say well why is that important? What we have learned is that it’s important because of the nature of us as being service agencies we have created a relationship with the schools that is credible, we are there to serve them, and so when you’re looking for someone to participate it works out very well if we do the recruiting. And we kind of play good cop/bad cop, we tell them all the things that are nice and that they want to hear, and then when it’s something that they don’t want to hear we blame the researchers. And it works out well.

While he’s getting started on that he mentioned the onsite coordinator who we have learned now is a very, very important role, and what I would say to you if you were selecting an onsite coordinator, she’s an excellent communicator, she’s very well organized, she had previously garnered the trust and respect of our schools, and I asked here the other day what do you think is the most important thing and she said I meet them on their terms. So all of those problems that grow in this process she solving but she’s solving it within her framework of knowing what that school is experiencing.

DR. MYERS: I’m going to quickly go through some of the challenges from an evaluators standpoint. One of the things about the Allegheny Intermediate Unit, this is the belt around Pittsburgh, there are over 40 school districts. Now I think when most of us think about these, doing these large scale evaluations in school settings we think about okay, we’re going to pick four or five school districts and we’re going to go and work with them. In this case there are over 40 school districts that have to be dealt with and each of them can have its own review process and its own IRB if it wants. So there are lots of actors involved and that’s why having the local site coordinator and Donna and her staff involved has just been really instrumental here. They have relationships with these people, they understand the boards, they’ve given us warnings ahead of time about the political issues that we’ll walk into. One of them was the Tuskugagee experiment, people started reminding us about that, especially when you start talking about brain scans. It’s just part of the challenge, these challenges can be overcome and I think sometimes we make too much of them. I think Judy made a really good point, the set up is critical, you pour your resources into that, if you don’t and screw up it’s over, and so you just hang in there, you try to figure out solutions, there are very simple principles to random assignment, you don’t need to do much, you just need the probability of assignment and you’re off and running. You’ve just go to maintain that.

Some of the issues that have come up besides getting approval from more then 40 school districts include, and there’s an interaction here that goes on between the intervention and the evaluation, a lot of the evaluations that we’ve done in the past are existing programs that are out there, I’ve done evaluations of the Upward Bound Program, for example, it’s been out there for 30 years, we didn’t have to interact much with the intervention itself. But here we’re trying to construct the intervention, it’s much more like a demonstration where we’re constructing the intervention and we have the evaluation so part of the issue here was is we had to identify fairly quickly what schools would be in the evaluation. I think what, 60, 70 schools showed interest initially in this, now we needed to come up with about 40, ended up with 50 because we had to use these pairs, but we had to tell them very quickly who was in the evaluation or not because we needed a teacher from each of those schools to provide the instruction. Why’d we have to do it quickly? We didn’t have enough planning time and we’re trying to get the intervention going by October 15^th, so we had to teach the teachers how to teach these programs quickly.

DR. DURNO: The original design included hiring teachers to do this, and one of the things that the Intermediate Unit insisted on and the group that was planning it was very responsive to is don’t go out and hire other teachers, the teachers in this study are getting the benefit of a great deal of staff development and learning, and I didn’t want new teachers come in, get that and then they go away, I said let’s have the study pay for the long term substitute and then the school district teacher will be the one that participates, he or she learns, and then is resident in the schools. So the knowledge and learning that is part of this stays resident in the school. And it also cost a lot less.

DR. MYERS: We talked about one last contingency besides this interaction between the intervention and the evaluation. In trying to plan these studies we need information, for example, on how many kids you would expect to be in a school who would fall in the lower 20^th percentile. There’s a lot of guessing that has to go on here. Even in enrollment, I have so much more respect after the last couple of studies we’ve done in terms of looking at what principals go through at the beginning of a school year, it’s a nightmare. And I can understand why they say I don’t want an evaluator there, it’s horrible, they don’t know how many teachers they’re going to necessarily have, they don’t know how many students they’re going to have, they don’t know if the teachers will show up or not. But here we come in, we say okay, how many kids do you think are going to be below the 20^th percentile because we need to have this structured intervention that’s going to operate in your program. In general they can give you rough numbers, what we’re finding right now as we’re going through and we’re building up the sample, that we may actually have to adjust the criterion of being in the lower 20^th percentile, there may not be enough students in the schools, maybe we have to go to the 25^th percentile as a limit. We’re investigating that now. Things happen when you’re out there and you have to be flexible about it.

Some of the other things that come up that we’ve been dealing with, and these are common things that come up in these kinds of evaluations for example, what are you going to do with a sibling? You have grades three and five, there’s a good chance you’re going to have two kids there, maybe they’re both struggling readers. You’re going to randomly assign both of them to the treatment or the control group as a family unit? Or are you going to keep them separate? What it comes down to is asking the intervention, those that offer the intervention what they would normally do. You want to reflect their practices and I think most of them would say in this case take the sibling along with them. But that actually creates some problems when you think about doing randomization, you don’t just to go SPSS and say randomize. It’s very tailored what you have to do here with these siblings.

There are other questions that come up, what do you do if you have seven kids in the school? Now remember the instructional groups were three treatment, and I didn’t tell you this, two control, within each grade level they have two instructional groups, three students each, and then four control students. You can imagine coming up with odd numbers of students where too few for the control groups, now you have to think about balance issues. And we have a lot of schools here to contend with and every one of them is a little bit different.

So Donna why don’t you talk from your perspective?

DR. DURNO: Some of the challenges that we experienced in working with the schools is the difference in the climate and the culture of each school. They have different ways to get things done, they reflect different parts of our community, for instance our poorest school raises $17,000 dollars in mail in tax, and our richest school district raises about $1.2 million dollars in mail in tax, so you can imagine the variation between the schools, which is nice in that it gives you a wide variation of participants but it also means a very big difference in their culture. How willing are they going to be to accept outsiders and what, this was one of the big ones, what pace do they set for accomplishing tasks? That was a tough one.

And then overcoming some of their philosophical differences, there was a lot of initial need for them to understand what research is and the implications of the research and the need for the research to be, to have great fidelity. And even the topic of reading was, there’s a politicization, it’s a hot topic, there were a lot of questions asked about how that was going to be done. And then there’s still that question about well, the federal government was putting a decent amount of money into this study and there are people who don’t like the federal government. So there were a lot of issues to contend with.

The accessing the decision makers within the district, as much as we know about them and we work with all the various groups in a school district, we found out in the informal network who’s really making the decisions. Is it the superintendent? Is it the curriculum coordinator? Is it the principal? And that was a struggle trying to get to the decision maker, particularly with the timeline we had. And the timeline was a problem because we decided as a group in about the beginning of May that even though it was going to be a rush we were going to not let another year go past and we were going to move forward with this study. Well by the time we got selection of teachers, we asked each school to provide three of their best teachers and then those teachers were given a brief test and the best one for the study according to the criteria was selected. And then you had to get the boards approval and so the timing, the first round of teacher training was being held the second week of August, so there was a lot to do in that interim. And who really makes the decision? In some school district the board will do whatever is recommended, in others you may have one board member that’s going to, as we did, pound the table. And the FMRI introduced a lot of fear at the beginning but that actually is working now very well. And then the timing of the board approval, there were boards that didn’t meet over the summer months so it was really interesting.

The other big issue were union issues, we were dealing with 42 different union contracts, and they were, some of the unions were quite upset that we would give the teachers assessments to select the best teacher in the schools so we had to work around that. Some of them felt that the teacher selected for the study should be the most senior teacher, and we had to work around that. And some of them wanted the positions to be posted, you couldn’t select your teacher until it had been posted, actually we worked through all those and we still got the best teachers in the school for the study.

Then the whole issue of the control group. I would say the FMRI and the ethical part of thinking about a control group were the biggest issues in talking to the parents and the school districts because they felt it was unethical for the students in the control group not to get a treatment and so it just took a lot of patient explaining and we finally worked through that as well.

DR. MYERS: One thing to note about the control group was that a lot of these kids already have IDT’s, for example, they’re already in pull-out programs and it was important that while this is also a pull-out program that the kids continue their normal school experience if they weren’t selected. They weren’t being denied any service other then something that we were hoping to show would have an effect on them, that these kids had a lot, and some of them stayed in the classroom all day, some were pulled out, but a lot of these are very motivated parents. You can imagine working with special ed kids and their parents here, they want the best for their kids and if they see this intensive intervention there’s a strong desire to get them into that.

DR. DURNO: And there are both Title I students and special ed students in the treatment group because it was the lowest 20^th percentile of readers, regardless of what else they were experiencing they were eligible for this.

So another challenge that we had had was the concept of the pull-out program. Scheduling students into a pull-out program. If we were working in some instances, the fifth graders were in a middle school arrangement so they had already started to be departmentalized, so what do you pull them out of, and so the scheduling was really difficult for that and then of course the concern is what instruction are they missing and that varies, some of them are missing reading and some of them are missing social studies, some of them are missing math. But the feeling is that if they can’t read they’re not going to be successful in those programs anyway.

And just the time commitment, the scheduling of the students, allowing any outsiders to have access to the students, the testers, now they were independent testers who were trained and paid to deliver the test, the four intervention groups have provided the training for the teachers and are now in there doing follow-up with those watching the teachers teach and they’re using groups of fourth grade students to practice their skills under the tutelage of the four training programs that they’re administering and it’s going very, very well. There’s a lot of excitement about how fast even these fourth graders who are sort of the practice students are growing and the teachers are already saying can we continue to work with them because it won’t effect one way or the other since they’re fourth graders, but there’s, the teachers are very excited and we’re really pleased.

And the last thing I want to say is what we’re also seeing is the teachers are really beginning to understand research and really becoming researchers in their mind, the 40 intervention teachers that are working from the 50 schools are thinking like researchers, it’s great.