DR. FASHOLA: Good morning, welcome to everybody, and I would like to thank Lisa and Jason very, very much for inviting me to speak among this wonderful group of individuals. The presentation that we’re doing is very, very timely as was said by Richard Shavelson earlier, and pardon me if I refer to him every once in a while as Dr. Shavelson, he was my stats professor in graduate school, so I see him at conferences, I see him all the time and I still call him Dr. Shavelson and he goes it’s Rich. I would also like to thank Ms. Loretta McClairn for coming and let her know that I really, really appreciate her taking the day off and being a collaborator in this process.

I wanted to make one correction, though, Ms. McClairn is the program coordinator for the Child First Authority, for Child First at Dr. Bernhard Harris, which is an after school program. The Success For All part came in and as we’ll talk about it, but she is not the coordinator for Success For All, Dr. Bernhard Harris is an open court school, Success For All was the intervention that was used on the one and one tutorials, so I just wanted to correct that.

On each of the slides, we’re both standing together because we’re both going to make comments about each of the presentations, sometimes I will lead off and sometimes she will lead off. The nature of the intervention was, we wanted to know at Johns Hopkins University and with the Success For All Foundation first, based upon some of the work that has been done in the area of after school we wanted to know what is the extent to which an after school tutorial program, that was a one on one tutorial program among first grade students in need of remediation would work in an after school setting in public school. That was the nature of the question and we’ll talk about the broader historical context later on but that was the question that we were asking. Based upon issue of access and based upon issues of relationships that I have with the Baltimore City Public Schools we added the third bullet, which was students enrolled in the Child First Authority. As has been said before it is very difficult for researchers to walk into an after school setting, or any type of a school setting, cold turkey and say we would like to engage in a randomized field trial study, so we added the Child First part second.

We’re going to go to the next slide and then I will introduce the first two bullets, the first three and Ms. McClairn will introduce one of them. The broader historical, political and policy context, IES, the Institute for Education Studies, was former OERI, the Office of Educational Research and Improvement. Out of OERI, OERI had the comprehensive school reform design, OB(?) Porter(?), which is now changing into or metamorphasizing into No Child Left Behind. There’s been a big focus on randomized field trials, on experimental field trials, the golden standard by which a lot of programs are being implemented and the effectiveness of programs is being judged.

Our research center, Cresbar(?) at Johns Hopkins, oh, I also want to acknowledge our Cresbar colleagues at Howard University, but Cresbar at Johns Hopkins has also advocated randomized field trials, we have advocated scientifically based and evidence of effectiveness. But anyway the political policy context was that the time was right. I’d also been involved as a member of the evaluation task force for the 21^st Century Community Learning Center programs and in that area I’d done a substantial amount of work in the area of after school programs. Additionally the Baltimore City Public Schools around this time also had after school programs as a part of its master plan. One of the ways that they were going to increase academic achievement during the regulator school day, as well as after school, was to implement academically based after school program. So with all these factors working together it made it easy for us to approach after school programs and say the time is right, this is what we would like to do with you.

And finally I had had a relationship with the Child First Authority, which is an after school program, as their evaluator for seven years, six or seven years, so they knew who I was. They were familiar with research, they were familiar with Johns Hopkins University, they were very familiar with you, so this is the broader historical context that framed the feasibility of our being able to go into the school. Now Ms. McClairn is going to explain what Child First is because I’ve been using the word a number of times.

MS. MCCLAIRN: Good morning. Child First is a comprehensive after school program located in eight schools in Baltimore City. Right now it’s at Dr. Bernard Harris Senior Elementary School and it’s completely supported at this time by community partners. It’s an after school program that operates four days a week, Monday through Thursday, from 2:30 until 5:00. Our segments include an academic segment, a culture enrichment segment, a homework segment, and we have a lot of parental involvement. The whole purpose of the program is to serve as an intervention, as a link between the regular school day and the home. Currently we are serving about 170 students per session, we have three sessions per year.

Child First is the governing body but we also get funding from the state and other agencies within the city. We’re hoping that we will get many new partners, currently we have 11 partners, they include Goucho(?) College, Clay Works, Port Discovery, and numerous other community agencies because at Dr. Bernard Harris Senior Elementary School we believe that we need to use all of our resources to give our children the best education possible.

DR. FASHOLA: See why I brought you? I couldn’t have done that. It is very important to understand that one of the things that made our intervention feasible and possible was that the one to one tutorial program was planted inside the Child First after school program, so many of the issues that effect establishing the after school program, the parental signatures in terms of signing the child out during the after school program, ensuring the safety of the child, they were already covered because we were a 30 minute intervention per child and when the child was not with us the child was taken by hand back to the after school program. That would have been difficult to do had we gone in and tried to set up an after school program cold turkey, so it’s important, and we will address this a little later on.

Moving to slide number three, we approached the schools and once again the schools knew who we were. When I say we, sometimes it’s I and sometimes it is myself and partners who were the Success For All Foundation. It is important to understand that I’m a research scientist with Johns Hopkins University, I am not a research scientist with the Success For All Foundation. We were partners but I am not Success For All, I am Johns Hopkins, but we collaborated on this effort, and they did a lot of groundwork.

But we approached the schools and we approached the principals and the program coordinators and once again this was easy because we already had a relationship with them. Prior to doing this, however, I had only existed as an external evaluator. I had not done any form of collaboration with the schools, so for the first time I went to them and said I am interested in adding this. It also happened that during this year the executive director of Child First said this is wonderful because we now have an academic mandate, we have a mandate to improve test scores, we have a mandate to improve performance, so this is a good thing. But we had to tell them why it was going to be a randomized field trial study and not an intervention for all students, and some of these have been addressed by our previous presenters.

The first thing there was not enough money for everybody, the funding for this project was coming out of my grant, out of my Cresbar grant and whatever was left over when I went into the red was provided by the Success For All Foundation, we did not have enough money to serve every eligible child, either in the first grade in these schools or even in Child First. We said to them, the results being reported today are the results of the second year, we conducted a smaller study and this is a larger study, we said to them if you let us come in this year we will do our best to come back next year, we let them know, we were honest with them and let them know we didn’t know how much money we would have, but if we had the money we would try hard to come back next year. And we did fulfill that promise, we said to them we would pay the teachers for training, when they went through the training process we would pay them. We would also pay the instructors for teaching, all this would come out of our pockets. We let them know that our goals were similar to their goals as in the master plan for Baltimore City Public Schools in terms of after school and improving academic performance it was in line. We also said to them that we would provide them with the materials, which was a lot of stuff, at no cost. And then I’m going to defer to Ms. McClairn, I’m going to skip one and then I’m going to defer to Ms. McClairn for the second to the last bullet.

This is going to be addressed I think in the second to the last or the last slide, we said to them that the control groups would be eligible for tutoring in the future, in other words the first grade students who did not receive tutoring would be eligible for tutoring, for free tutoring, if we had the money. We are trying to fulfill that promise, if we had the money. These were promises that went not just to the schools, not just to the program coordinators, these were also in the letters to the parents and this also went to the district. But I’m going to let Ms. McClairn address the issue of using the Child First teachers.

MS. MCCLAIRN: In the Child First program in grade one we have three teachers and in our regular day program we have four first grade teachers. And one of the things that we were specifically told was that they did not want the regular first grade teachers teaching in the after school program because of the materials that were being used. So it was a problem for us because in our after school program we use almost half of our staff because we try to keep our ratio very low, so we had to go out and find people who were interested in working with first graders who had the expertise and who wanted to go through the training. So all of the teachers in the program were teachers other then regular first grade teachers. And they had to have some real yearn for making a difference, and we always state that we’re there as partners, we’re doing teamwork, and that’s how we recruited the other teachers from other grades.

I thought that she had told you that the other teachers came on board as I said because they were team members and they didn’t feel that we were in competition, because we’re teammates they knew that it was going to benefit our children, and most of what we do is a collaborative effort rather then being competitive.

DR. FASHOLA: And this is just one of four schools, Ms. McClairn is representing one schools. It was important to let them know once again that we were paying money and we were not going to try to recruit their teachers to come into our program. The other thing was that we wanted to protect the fidelity of the program and so we did not want first grade teachers in the after school program that would then go and implement it during the regular school day in their classrooms.

Features arising from its nature as a randomized field trial study. First of all we went through the IRB issues with Johns Hopkins University and also with the Baltimore City Public Schools. Hopkins has been in and out of the news for violating randomized field trials so they were hard, and around the time we did the very first study people had pink eye and yellow eye and green eye and everything happened and we kept saying, you know we were fighting the clock, the seconds of the clock were ticking and the days were going by and the amount of time that we had to implement the program was narrowing and IRB said this is where we meet and this is when we meet, but we could not do a thing until IRB said yes. Once IRB said yes we went to Baltimore City Public Schools and we tried to get their permission and once again it was the same scenario, and until they said yes we could not provide permission forms to the teachers, but this is what we had to do and this is what we did. Once we had the okay from Hopkins and we had the okay from Baltimore City Public Schools, I have forgotten which came first, but once we had both then we said let’s go. We went to the schools and we sent the letters out and we said this is what we want to do. It was very important that we put in the letters that your child participating in the study does not mean that you’re going to receive tutoring, it means you will be a part of the study, and they mandated that we put that in the letter.

The other thing was that we had two groups, so we had a large group of students drawn from, okay we had a large pool from which we randomly assigned the students, they had to be enrolled in the Child First program and I talked about why that was important in terms of safety and all that. They had to be in the first grade, they had to be in need of remediation. We did not work with students who had special needs, there were some students who had special needs who were special education students, in some of the schools we had third grade students reading on a first grade letter. After we got the signature forms we had to take them out and say we need students who are enrolled in the first grade. Once that happened we did the pre-test, the students were randomly assigned, not by us, let me backtrack. We trained, we went out and recruited testers, we trained them on how to test, we paid the testers, then we sent, we went into the various schools and we practiced tested the students, I oversaw the practice testing of the students and made sure that it went the way it was supposed to go using the Woodcock, the Woodcock test we used, and then they went and they pre-tested the students, the Foundation used SPSS, the randomization thing to randomly assign the students to the treatment and the control group. The teachers who taught were not first grade teachers, the teachers were also randomly assigned to the children that they were going to teach. And then the treatment was given, at each school we gave three days per week per school, we gave one missing day in case you miss a day because Child First runs four days a week, so the program was for three days and if you miss a day you get an additional bonus day. And at the end of the study we post tested the students.

Students in the treatment group got three things, they got one on one, they had materials, and they had a curriculum by which the teachers had been trained. The teachers in the control group did not have one on one but they did have the academic component of the Child First program. They did not have a specialized curriculum like we did, like the students in the treatment group did, and as I said it was a group program.

Did you want to add anything?

MS. MCCLAIRN: There was one other thing, parents were not as hesitant to allow their children to participate because the children were already in Child First, there was an element of trust there, so because we partners already more of the people were readily available to sign the slip.

DR. FASHOLA: Thank you. Having said that I’m going to go to the next slide, slide number six please, relevant legal and ethical issues. The schools that the study was in, the areas, some of the areas, many of the areas, are extremely impoverished, and so it was hard for me to go into the schools and even though I’m a research scientist and an educator, but it had to be done as it was said, you cannot contaminate, but it was difficult to go into the schools and turn people away. And initially what happened was we went after schools, I mean students, and we continued to try to recruit and initially at the time when we had maybe two weeks to go we were hoping for a sample of 120 and we had seven signatures, we have seven. And we went out and recruited and recruited and we showed up at the schools and we did what we could, but we let the people in the school do the recruiting instead of us do the recruiting because of the element of trust. It was hard. After school is not mandatory, after school is voluntary. But even if after school were mandatory it was still difficult and so waiting for the IRB, waiting for the district, waiting for the principal, turning away students, for example, second grade students and third grade students who were in dire need of a program like this, but because they were not in the first grade it kind of tugged on my ethical conscious a little bit.

Secondly, we were only using students who were enrolled in Child First, so in some cases there were some parents who would sign slips saying I want my child to be able to have the one on one tutorial thing but for some reason the parents were not able, we could not catch them and say while you sign this also sign the forms saying your child is going to be in Child First, so there were some students who were turned away because they were not in Child First and once again they needed it.

The final thing, and we’re going to address this in the next slide, the next slide is going to be Ms. McClairn’s, but there was a number of things that took place that limited once again the amount of signatures that we had, and so one of the things that we did was we extended the deadline time, the time, we extended the sign-up time. Even after we had done that we still ended up with a much smaller, and the original end we ended up with was 70, and this was for the start. 10 attrition from the control group, 10 attrition from the treatment control, so our final end that we ended up with was 50.

After the deadline, I mean people only understand so much about randomization and so even after the deadline we had some people all of a sudden deciding to sign up, they went after children who really, really, really needed to be in the program but because the treatment had already begun and because the testing had already taken place and the clock was ticking we had to turn away students who were in need of the program and that was hard, and once again we also, because of the fidelity, because the registration deadline.

And finally students in the control group were just as much in need, and so when students would attrit from the treatment group the teachers would want to put students in the control group into the study and we had to say no, you cannot do it. And so rather then tutor more children we ended up with a smaller number of students that was tutored and a smaller number of students in the control group. And that was hard but we had to continue to say it is good for science, it is good for science.

The next slide is going to talk about some of the barriers and the challenges that we faced and I know Dr. Shavelson presented something, he presented some of the slide about the conversations, I think the one at the end said my battery has run down, but there was some additional challenges that took place at some of the sites and one of them once again was not enough signatures. In some cases the very students who really needed to be in the program were the very students whose parents did not sign the form, the very students who were at the bottom of the barrel who needed to be in the after school program were the students we could not get. But Ms. McClairn is going to talk about some of the barriers and challenges even though the program was taking place anyway, the program was in full motion, we had the cooperation.

MS. MCCLAIRN: The barriers basically included time, life situations, and the weather. To be more specific life situations, the one that really impacted us a lot was last year unfortunately we had four of our students to die in a fire, the Dawson family, and that was the week that we were doing the registration for this and consequently because of the devastation it caused within the school we had to close the program down for a week. So by the time it was our deadline we were not even open, and you can just imagine the impact that had. Another thing was the transition between the funding, our funding, part of our funding was contingent upon a state grant and because we were in the process of getting a new governor it made a big difference in the funding, and the funding was on hold for a period of time. So we had to just kind of stand still and do nothing while we were waiting to see if we would be funded. We were funded like on a month to month basis --

-- [End of tape.] --

-- and the terrible weather, because if you think about it and can remember we had one of the worst winters ever in Baltimore City, so these are the things that impacted our program.

DR. FASHOLA: Thank you. The day that, remember I said around the beginning we ended up with only seven signatures? I was leaving Johns Hopkins University to go Dr. Bernard Harris with a file saying I need the signatures with an empty folder and I showed up there with a plant and a card and I said you know what, just do what you have to do and I’m not even going to ask for signatures at all. These are all real things that take place, from this school we were supposed to initially have 60 students, we were supposed to have our biggest enrollment, we ended up with a lot less then 60 and at some point people were just not into it and these are issues that you have to deal with.

Pertaining to the funding, I wrote an article, I think it’s in the back of my book called Building Effective After School Programs, and one of the things that I say there is that it is hopeful that someday we will realize as educators and policymakers that the needs of certain individuals do not rise and fall with whoever comes into power and whoever leaves power, okay, after school is very necessary and the money had been promised and the money was there. The money had been allocated but we were switching governors and basically the previous person decided to tap dance his way out of the seat. Meanwhile, as she said, the complete after school program was shut down.

Now I talked about the importance of the partnership and I’m going to elaborate on that a little bit. At some point it could have happened that we could have said you know what, Child First is shut down, but we can still do the one to one tutorial thing because remember they are not Child First teachers, right, and we are paying them out of our own money, but the bottom line was that we couldn’t do that because if we taught them for 30 minutes where would they go, these were first grade students, what classes would they go to, would they walk home by themselves, remember we just talked about the neighborhood that the children come from, so the entire program was shut down.

In spite of all this the program took place anyway, I have three more slides and five minutes and I think we’ll be fine. Solutions employed, we had no solution for the first one, for the barrier, but we just had to wait until we received permission in terms of time. Secondly, solutions in terms of the number, we had to work with what we had and we had to understand that we had limited generalizability. And the third solution, we extended the sign-up time as I said earlier but we ended up with far fewer participants, but in the long run we were able to implement the study.

Costs associated with the study. Groan, groan, sigh, sigh, oh my goodness, we paid the testers, we paid the teachers for training, we paid the teachers for implementation, we paid the testers for post and we gave money for materials, books, manuals, and they kept them. When I got my master’s degree I had a badge that said I’m a grad, are you a job, and I wore that. I am wiped out, put all the money into the study, I have no funding, the control group, I am writing grants right now and I go to the schools and I tell them all the time it was expensive, in retrospect would I do that again? Absolutely. It is very important to put good money into good research and get programs and good interventions. So it was extremely expensive, at one point we had this illusion that we were going to write a letter saying that the Child First Authority was going to partner with us and they would pay for half of the first study and we would pay for 50 percent of it, that did not happen, remember they didn’t even get their money from the governor. Technically they still owe us money, do I think I’m going to get it? No. But this is real.

The other interesting thing though is that the question becomes, okay, oh, the study was significant, we administered the Woodcock test and the tests were letter identification, word identification and word attack. What the results showed was that the treatment group out scored the control group on all three tests, letter identification, word identification and word attack. But it was only significant on one measure and that was word attack. The school, during the regular school day, uses open court, the schools were using either open court or direct instruction, these are programs that are phonetically based and so the letter identification thing as it turned out once again the very students I believe that probably could have benefited absolutely the most were not enrolled in the program, so the main skills that the students needed to improve and the main skills that the program addressed in this case were the word attack skills and these were the skills that were significantly out scored. The grade equivalence moved up but there was only significance in one area. So the question became as I go to the schools, they say, you know they call me Dr. Toks, Dr. Toks, we did everything, it was significant, the students did better, we have all this wonderful information. Now are you going to give us money so we can continue to implement this program? And my answer is always I’m writing grants, I’m trying, I’m trying.

Did you have any final words Ms. McClairn because I think we’re almost out of time.

MS. MCCLAIRN: My final words would be that the study did a lot for our first graders, if you read the results you would know that our first graders did extremely well last year and I think we owe a debt a gratitude to the one to one study with Dr. Toks Fashola and Johns Hopkins. Thank you.

[Applause.]