DR. FLETCHER: -- and then we’ll ask for questions from the committee members and then from the audience in general, I’ll remind people to please use the mic and identify yourself when you ask a question.

One of the things that interest me, and it was something that occupied quite a bit of discussion of the original Science and Education Committee, are the issues of IRB’s and informed consent, and sort of how people in educational research deal with that. And I thought Dr. Fashola’s experience was really interesting, I’m going to be a little brutal here, but it looked to me like not only did the IRB hold up her research, they biased the research by requiring written informed consent for a minimally invasive intervention. And the reason I say they biased it is that it’s a reading study and guess who doesn’t return written consents when you said them home? I mean it’s people with reading problems, for example, so I thought it would be interesting to get our other two speakers to talk about the issues of IRB and consent and how that was dealt with, starting maybe with Dr. Myers.

DR. MYERS: In general I mean if you’re doing federally contracted work you’ll go through OMB clearance, and you don’t typically have to go through IRB clearance. What you often find though working with the school is that even if you go through OMB clearance each individual district will have its own clearance process that it wants you to go through. The Power For Kids Study, we’re actually using multiple forms of consent, to go through an IRB, the Florida State University’s IRB reviewed the study for us, that’s where Joe Torgason was, and the consent that we proposed and that they agreed to, there’s both a passive and an active consent side to this. The passive consent side operates when we’re screening the kids, we’re giving them these very short tests to see if they’re potentially eligible to participate in the interventions and the evaluations. Those that are eligible we then send an active consent form to the parents, we believe strongly in that that when you’re doing these kinds of experiments and people are being held out from an intervention, for example, that you do need a parent to be well informed to give that active consent, I think most IRB’s would go that way as well. So once we get that consent back we also have a form in the packet that asks if a parent would be interested in having their child participate in a Functional Magnetic Resonance Imaging Study, so participating in the evaluation is not contingent on saying I’m willing to have my kid’s brain scanned, so we actually have these multiple layers to it. In general we don’t have much problem, we’re right in the middle of, we just sent out the consent packets this week and so we don’t know how it’s going in this study but in general it’s not a big problem, usually it is an issue of information, sometimes parents will say no when they don’t understand it and if you can get to them and explain it to them well they’re usually pretty receptive.

DR. KELLAM: Well there are two aspects, I’m big on multi level thinking, it may be some kind of Rube(?) Goldberg genes in my head or something. But in fact the IRB at Hopkins, you all may not know that we have a rich history of IRB argument at Hopkins, the place has been closed down on occasion, so that’s a serious issue at Hopkins and the interesting thing is that when we were first going to get written consent back in the mid ‘80’s there was this roaring argument because the Board of School Commissioners in Baltimore took the view, as Linda was describing, that IRB is a Board of School Commissioner issue, it’s not a John Hopkins School of Public Health issue. I mean what do we have to give, suppose they gave us consent or permission to go out and do this, that still leaves the Board of School Commissioners out there. So when you’re thinking about the RFT problem I think you have to think, and you all were sort of implying in talking about the same thing, the multiple levels, the formal IRB is essentially that education stuff, that this Board of School Commissioners is by law supposed to do, Board of School Commissioners wants to have oversight, and that’s Linda most immediately and Jim Smith, the two area chief academic officers. On the other hand, at the parental level, you have to get permission, they’ll throw you out, I mean the local community level is vitally important so it’s all part of the whole multi level partnership issues that are the foundation for this kind of work. I think that whenever we go after more specifically measures of psychological well being, the kind of behaviors that the school district doesn’t ordinarily do, it’s required by law to go out and get written informed consent. And in fact we do that, but we don’t do it when it is in fact a measure that the school district thinks is their by law mission to carry out, so they don’t want to ask permission from parents whether they can do achievement testing, for example. On the other hand if we ask about depression and anxiety, which we do, then we need to inform everybody about what’s going on at these multiple levels. Ultimately the best thing is to think about all this in the context of Linda’s description about the multiple levels of support, buy-in and ownership because that’s where all of this has the germane impact.

DR. FLETCHER: Do you want to comment Dr. Fashola?

DR. FASHOLA: I was going to add a little bit, just a slight comment.

DR. FLETCHER: And then it will be Dr. Clark.

DR. FASHOLA: When we were initially going to conduct the study we wrote the IRB and asked them to be exempted and they said no, I mean the Hopkins, separate from the school district we asked for exemption because we said this is something that will be done anyway and they said no. And also it was small enough that we could send individual things out. This was Homewood(?), I don’t know if it differs at the School of public Health, and in fact this coming year I think that the first control group will have expired so even if we just want to give more treatment that we had promised them maybe two years later I believe we will have to fill out a whole new IRB, a whole new permission slip and all that from Johns Hopkins, but not with the parental thing from the teachers but through Hopkins we’ll have to.

DR. FLETCHER: If I can just comment briefly, there’s a new NRC report on human consent, and one of the points it makes that I think is very interesting is that many IRB’s are preoccupied with obtaining written consent and the task really is to make sure the people are informed about research, and there may be other ways beside written consent to ensure that people are adequately informed of some of these other methods, particularly for non-evasive research, may be more desirable in terms of adequately informing people. But let’s go to the committee now, Dr. Clark.

DR. CLARK: I want to continue this question about IRB’s, not so much on the procedure or who has jurisdiction over which part of the permission but what kind of return rates you’re getting and how that variable effects your ultimate analysis. I know that as IRB’s get more and more constraining and the letter you send out to parents has to be longer and longer and list all the horrible things that won’t happen to participants, participation rates drop, and as they drop then you have real problems with sampling and representatives of your sample. I wonder in all of your studies if you could just first of all just give us an estimate about what percentage of the people you wanted in the sample actually gave permission, and then whether you thought that had implications for the kind of conclusions you can draw.

DR. MYERS: I’ll jump in on the Power For Kids Study. At this point we’ve just sent out the consent packages there. In the past, I’d have to look it up a bit but I wouldn’t be surprised if maybe ten percent are refusing, I just don’t have the strong sense, and Donna and I were talking about this a little while ago, that there’s strong, a large percentage refusing to participate in these studies, at least so far. I have some concerns about, for example, the amount of work that IES is pumping out this year and the effect it’s going to have on the schools and the parent, you’re going to be in a lot of the same schools and they’re going to see a lot of consent packages potentially. There’s $30 million, $40 million dollars worth of work out there and you can imagine being in Chicago and places like that have a lot, they’re going to get hammered a bit and there could be problems.

DR. DURNO: On the fourth grade piece we had a passive consent and there was no problem at all. There’s already been, because we’re using teachers that are in the school district rather then substitute teachers there’s already a good relationship in many instances between the teacher and the parents and in between the school and the parents. So we don’t anticipate a problem because the communication and the knowledge is there.

MS. CHINNIA: I’m going to let Shep give you the percentage in terms of consent but I would say that I think the process that we have used has made it if not close to 98 percent at least that, and that’s the fact that we started at least six, seven, eight months before the actual trial in meeting with groups of parents at individual schools, having that community based person who then continues that conversation one on one with parents along with the administrator of the schools, and then spending the summer and the beginning of the year, and we’re currently doing that right now so the actual written consent is now going out but that may end up with a handful of students per school that will need additional contact with someone to discuss, but the rate has been extremely high but I think it has to do with the process of informing parents and really making that a systematic process.

DR. KELLAM: I think, well, the second generation was 98 percent across the board, nine elementary schools were participating. The first generation was pretty much, a little bit lower, in the 90’s. It’s a mistake to think of the IRB process and the consent and informed consent process separately from the partnership. The partnership you develop, which we were talking about from the very beginning this morning, and Judy raised it, I think is fundamentally important and it’s multiple levels and it’s the same year, year and a half that it takes to build a base to do a randomized field trial properly that it then requires to get consent, I mean because consent is very much part of the process of buy-in and ownership. If you identify mutual self interests at multiple levels everybody, I mean consent sounds like a silly word, everybody owns this joint.

DR. FASHOLA: I do not have exact numbers here but I know, for example, for the school, I was just asking Ms. McClairn, we believe she had 42 first graders and I believe we ended up with about 16 from her site. It definitely did influence the return rate, it was pretty low, it was under 50 percent. So it did play a big factor. When we extended the deadlines in some cases the teachers would get excited and they would fax me additional ones but it was the same parents who would sign the consent forms again and again and again, so our numbers would go up and then they’d go back down.

DR. FLETCHER: David, I have to say that I think you’re going to be surprised at your consent return rate if you expected to have them mostly by parent signing and returning. It’s not refusals, it’s parents who simply get material home and don’t do anything with it and that’s the big problem, that’s why you have to go out and contact them. And there are relationships between who bothers to return things that schools send home, I mean it’s people that read less well and people who have lower levels of education are less likely to return consent forms. So it does potentially bias samples and so on. I think Dr. Henley is next.

DR. HENLEY: I wanted, Ernest Henley, I wanted to ask if David Myers in particular, whether you follow-up on the study whether the intervention, the difference you find at the end of the intervention year persists to higher grades.

DR. MYERS: Well as I said in the presentation we’re not at that point yet, we just randomly assigned the schools to it and we will a two-year follow-up period after the intervention year so we’ll be able to talk about that.

DR. SCHNEIDER: I wanted to talk a little bit more about the front end of the interventions. When you were all describing informed consent or implicit consent, the first thing that you said is well it’s really important to build this partnership, but some of the kinds of things that we haven’t talked about at all that I was kind of hoping that maybe you might say something about pertained to design. So how many people do you over sample? What kinds of considerations do you think about with respect to over sampling? Are there some other kinds of procedures other then just partnerships that make this possible? If you had to give some advice to people about what they would need to do at the front end in designing these kinds of studies what kinds of things might you say?

DR. MYERS: Well, in the Power For Kids Study there was no over sampling at all. Ideally while you would have some over sampling the problem you run into when you set up a demonstration, Judy can speak to this, I’m sure Shep can, when you set up a demonstration there are huge dollar costs attached to over sampling. Imagine that we over sampled several schools for each intervention. Now our intervention costs may go up 30 percent so instead of $3 million we’re at $4 million because we have teachers that we have to pay, students that we would have to instruct because you can’t say once you’ve done the random assignment, and let’s say that everybody shows up, what a wonderful thing if that happened. Well, now you have more then enough sample then you needed and you’re not going to pull back instruction from kids, so you’re stuck with it. And that was one of the interesting things I found, typically working in school settings the number of students you’re given is fixed. To think that you’re going to adjust the number of kids in the classroom, it doesn’t hold the way we normally think of sampling, it’s a fixed number that you’re working with. You can sample kids from a school but if they’re going into instructional groups or you’re taking classrooms, you get what you get out of it. You can sample schools but if you’re setting up a demonstration and providing the intervention and it’s not already there you’ve got huge costs and so you don’t over sample so it really puts a strain on the staff, particularly I think in the AIU case, to help us keep those people there. It’s a very different dynamic that takes place in the demonstration versus doing big quasi experiments where you just, well, I’m going to take lots of comparisons and I’ll take what I need when I’m done and that kind of routine.

DR. FLETCHER: I’m just going to mention this, one of the obstacles that David didn’t mention, I’m on the Haan Advisory Board, was the discussion at some of the early meetings about two big issues, the first was whether there should be a control group, and there was great opposition to the idea of a control group, and then the idea that since there was going to be a control group that random assignment should be used. That’s really where that study started and I think the evolution from simply taking every single student, which is motivated by the idea that of course these programs work and everybody needs them, to a randomized trial is really quite significant because at the beginning it was nothing but an efficacy study. Dr. Tobin?

DR. TOBIN: I have a question that has to do with how specifically or narrowly we define the interventions. In Dr. Kellam’s presentation, at the end looking across your studies you suggested that one thing this demonstrates is the importance of teaching classroom management to first graders but yet classroom management could range from a sort of law and order kind of paradigm to a democratic community version. And to even complicate it more it could be argued that maybe it’s not just any one approach to classroom management but the fit between the approach to classroom management and the local community in which it’s being taught and fit with home values and school values. So it’s a question to you but also to everybody about how narrowly you define the intervention, some people say it has to be so narrow that you have to script it and then they have to follow the script or else you won’t really know whether the teachers may be introducing some other variation through the force of their personality or drawing on their prior knowledge.

DR. MYERS: I think, Shep can address this, too, obviously but I think one of the things that really is important in this situation is to ask yourself what is it you’re trying to assess the impact of. A lot of what we talked about today was assessing something that’s offered with high fidelity, and that’s a wonderful thing to do because you want to establish can I make a difference. But you also can ask the question if this intervention is available it’s going to be implemented with lots of variation and you’re going to have motivated and less motivated teachers and all kinds of things come into play. That’s much more of a policy question and you don’t want a script in that case, you want to see what happens when I’m out there. I think that’s one of the things that’s often lost in these studies, people think that you’re going to do the thing with high fidelity but that’s not always the question. If I’m a policymaker putting money out there I want to know did it work as implemented.

DR. FASHOLA: Thank you for asking that question because as we looked at the conclusions of our study one of the questions that I was asking myself was in the one sub test that did show significance was it significant because of the one on one nature versus the group nature? Was it significant because the teachers had received specific training on a specific curriculum that has been around for a while versus teachers that were using a generic training method? Was it significant because the instructors willingness and yearning, the selection of the people who taught the class? Or was it effective because of not the curriculum but the materials? I mean it came with books, I mean the kits were big and they were expensive. And the only thing that I can say is I do not know the answer, I have to be honest, but the fact is that we tried to control for a number of these things by saying that the students in the control group at least did have access to tutoring anyway, they did have access to academic intervention, specific academic intervention as Ms. McClairn did explain when she introduced Child First, prior to our being in the program they always had academic programs. So that is one of the questions, the other thing though is one of them is a student teacher one to one ratio versus the group nature. The other interesting thing though was the anecdotal information that came, the students were running to the tutoring program so the individualized attention above and beyond just I’m working with a teacher on a one on one basis, but the students liked it, they joined the interaction, every time we went to observe them the children were just on cloud nine as were the teachers, so these are some questions that we do not necessarily have answers. Is that answering your question? Thank you.

Can I say one thing? There was one study that studied something similar to this, this is in one of our publications called Show Me the Evidence, Elizabeth Fenma(?) CGI, Cognitively Guided Instruction, she did something very similar to ours but the control group had the materials, they just did not have the instruction, it was an incredible study and I said that should be the model. Interestingly randomized and everything, I believe it was randomized, the program is no longer being funded, the last time we checked them for an update it is no longer being funded.

DR. KELLAM: Well, it just goes to show how muddled the conversation can get, we can sort of, let me go back to Barbara Schneider’s question and try to pick of Joe Tobin’s and then try to figure out how to connect it up with what we’re talking about. In the first place what we’re doing is coming at this field of randomized field trials from different disciplines, I mean we’re coming at it from different concepts, different ways of perceiving the world. That’s what struck me by Judy’s talk, she was coming at it from an economics point of view about welfare and so on, it’s a different field. And I’m sitting here from a public health field, which happens to have a lot of overlap in risk and so on with public education, achievement has always been in our analytic models. But the problem is that we have, see, let’s go back to Barbara’s point.

Suppose you over sample, is that a way to solve the problem of not getting consent? We got to that from somehow difficulties getting consent. Now the argument that some of us are making is that in fact you can’t back off from getting consent, you have to get consent not because of the law but because that’s part of the partnership. If you have a 50 percent rate of return that means you haven’t gone out there and established mutual self interests, you haven’t understood the vision of the folks you’re working in the institutions of. So you can’t solve that problem by over sampling. Now what’s this got to do with public health is where I come from and psychiatry no less. Well, I don’t know what it means unepidemiologically, I mean if you in fact have a sample and you want to study the population first of all you define a population and all the people in that population are your subjects for study. They’re also your partners, they’re the ones that can kick you out if they don’t like you. Now every kid then is a real kid, it’s not just a representative of some nameless denominator that nobody ever heard of, that does a number of things.

First of all it makes us not want to go over sampling because in fact we want to know about this population, in this community population I want to know what the variation’s all about. And I’m not really interested in the mean for that population on anything, what I’m really interested in is the variation, I want to get the mean, I want to see if it’s an 05 level, but really what I want to know is what works for which kids under what circumstances. That’s an interaction of kids in their environment, classrooms, school buildings, families, community and so on. So I’ve got a stick there because of ecology where I’ve stuck my spade to do the work. So I don’t know how over sampling helps, I do think that we need to reinterpret the IRB problem so that it really is identical in part and parcel of the problem of ownership and partnership, those are the folks that let you do the randomized anyway and there are many other aspects as you’ve heard and that you know that have to do with the way that partnership has to operate.

PARTICIPANT: One of the themes earlier this morning was the importance not just of looking at the treatment fidelity but looking at the control groups and seeing to what extent people in the control group end up picking up some of the treatment things and it’s particularly problematic within school randomization which I think all three of your things have. So can you each speak a little bit about what you’re doing or what you plan to do to look at the control groups and if they are starting to pick up the treatment to discourage them from doing that if that’s the approach you would take?

DR. KELLAM: Let me start with that and maybe Jack, on the verge of talking too much, but this is really a critical issue because in fact what we really are talking about is the degree to which we measure implementation or fidelity or the intervention in the intervention group, but equally in the control group. And as we heard in more general terms this morning the control group is usually the standard program without the enhancement, but the standard program really done as well as it can be done within the limits of the standard program, that’s the proper yardstick, there’s no other point in having a no treatment control, we know that there’s some treatment going on, it’s not noise. When you’re measuring the variation in fidelity or the variation in implementation it becomes an extremely powerful tool for understanding a whole variety of things, A, how much is going on in the control group and if it’s a representative population you’re control group is indeed telling you what the population is doing without the intervention. And in fact when you think about it developmentally in growth modeling terms the baseline is not a point in time it’s the trajectory and the intersect, and that’s the developmental, that’s what the population is doing over time. And so you measure the intervention, we’ll use Barbara Foreman and Jack’s measures on the control group, just as with the same intensity we’d use in the intervention group because we know there’s going to be some teachers who do know how to manage the classroom, other teachers who don’t. In fact the rate that you saw up there, if you can remember that rate, was about half the teachers do fairly well managing the classroom, and the intervention tends to help the teachers who don’t, that’s 50 percent, not 100 percent, of the teachers. So we’re talking about some of the intervention going on and it bubbles, and some of the intervention in the intervention it varies.

The one thing I would say is that, I don’t know if the language of efficacy, effectiveness, sustaining, going to scale, those are terms that have traditions in public health and I don’t know. For me an effectiveness trial is the most rigorous trial, it’s the efficacy trial taken into the field with the exquisite research design and training you could do but you know there’s going to be variation in circumstance, in training, and in implementation, but that very variation is part of your analytic model, it in fact has very powerful functions that tell you how much of the intervention you need, how much variation is tolerable in terms of the impact, and for whom and under what circumstances, it sits side by side with the other moderating influences in your analytic model, and some of those, it’s nice to think of, you’re aiming at a causal factor but you could be aiming at a moderating factor and that’s part of the model.

MS. CHINNIA: Can I add one thing? And I’m not the researcher, but on the part of the educators and the teachers and the staff in schools I think a plus is in terms of what is happening is the inclusion of those persons that are working the school as a part of the planning and the true implementation. In the past we have been actually a part of the research in some cases where we’ve been asked to do things but don’t really see the big picture, and so I think some of what helps to in terms of, and again, I won’t use all the correct terms in terms of the efficacy of what Shep wants to get out of all this is that the people who are doing this, who are actually going to carry this through in the trials and the classrooms as well as those teachers that are working with the control group and knowing that we want them to do the standard programs the best they can understand what the big picture is, what the end result will do for them, for the children, and for the system, and I think that does --

-- [End of tape.] --

PARTICIPANT: -- that I had was that if they are asked, I mean this is a very serious, committed group of 40 teachers and if they were asked by their trainers to not do something they wouldn’t do something. They’re very much into their role of this year which is carrying this research out and learning and growing. So I think that it would be very unusual for any of them to do anything they were asked not to do.

DR. KELLAM: Well, let me go back to our design for a minute because one of the things that interested NIDA was the fact that we had a two pronged design, one was following kids over time in effectiveness of the program, the other was following teachers over time in terms of the quality of instructional practice, classroom management, parent/classroom partnering. So in fact that’s the first level mediator for that whole analytic model so in a sense what we’re saying is as part of the research directly in the center of the analysis is the impact of the program on the teachers quality of instruction, classroom behavior, and parent/classroom partnering. In other words the implementation measurement is about the teachers. Now what it doesn’t tap is whether the rates of teachers leaving the job, leaving the field, burning out are influenced by increased mastery of their job. We would certainly hypothesize and one of the things we talked about in the higher levels of the school district was whether this would in fact lead to greater teacher retention. We know from another AIR study actually that one of the major reasons for burnout of teachers is indeed failing to do classroom mastery or failing to do the task of teaching, so these, many of these trials that have as the mediator what the teachers do in the way of instruction, management, and the like are deliberately focused on measuring that very issue and I think that indeed the focus on implementation measurement is a paramount focus to keep in the RFT business.

DR. FLETCHER: David Colar has the next question and then we’re going to open it up to the floor and Barry Sloan is first.

DR. COLAR: This workshop is convened by a committee whose broader charge as I understand it is to look into the quality of science in educational research, catch up with the big boys so the research in education looks more rigorous, more theoretically motivated, more empirically based, all those sorts of things. I wonder in today’s presentations the extent to which you view these projects as demonstration projects or as projects that are going to bear on educational theory, on learning theory, on theory of teacher training, on any kind of, what the theoretical import is of these studies, whether you think that’s down the road and whether theory really shaped the design of the studies or whether it was a set of pretty good intuitions or how you can see it.

DR. MYERS: Speaking for the Power For Kids Study, I mean unlike a lot of policy evaluation, which aren’t, a lot of times policy studies are based on hunches, intuition, I think here at least in the reading area there is a body of evidence that shows that certain approaches to teaching reading are effective, at least in clinical settings, and Jack you can comment on this much better then I can in terms of how theoretical it is, but I think it is. We’re not at, if you think of sort of the FDA as different stages, we’re at a different stage of that research and we’re now saying if you put this on a much larger scale can it be effective for kids who are struggling readers and can these packages of interventions be effective. If you look at the National Reading Panel work, for example, much of it talks about specific approaches but not interventions, not packages where things have been put together. We now have gone to the stage of saying okay, let’s put together these critical elements, and it’s corrective reading, failure free reading, the Wilson model, spell read, PAT, these are packages that have been put together and they have certain things in common, and so we’re at that stage of seeing does the package work.

DR. DURNO: And the question also is does the intensive intervention work.

DR. MYERS: Yeah, make the difference.

DR. DURNO: And does it work to bring the student up to grade level, which other remediation programs have not, they’ve kept them from going further behind but they never catch them up so the goal is to see can it be caught up. My sense in communicating this to the school districts all along has been that it’s very theoretically based and it’s very specific and the outcomes should be worthy of replication or consideration, particularly in the scaling up of what has been proven on a smaller basis.

DR. KELLAM: Well there are two things. One is that at least in the prevention public health field if it ain’t theory driven it’s unlikely to be funded. Now what people mean by theory is another question, there are patterns of how people define theory. At one level the analytic model that I showed of the whole day program is a theoretical model. It says that if you change these three things through these paths this stuff on the other end will happen. And it’s a broader theory, it’s called Life Core Social Field Theory that first guided us from a Kurt Lewin kind of thing where mastery is a good thing, kids in classrooms which was a lot of mastery going on tend to master better and so on. And it’s heavily related to psychological well being. But there is now a lot of empirical evidence that these risk factors are associated in meaningful ways developmentally but achievement really does reduce depression, for example, depressive symptoms, and there are gender differences that may account for internalizing, externalizing, or at least may be revealed in those patterns. So it’s very much theory driven. You have to have a theory for why you’re aiming at the antecedent and you have to have a theory of the mediation and the moderation to explain both the path to the outcome and the variation in the path to the outcome. And those are all parts of the modeling. Without theory you ain’t got no excuse to aim at anything. So we suggest if you want the money and you’re going after the grants, at least at NIH these days you’ve got to have a theory that says why you’re going after that target.

PARTICIPANT: I’m going to add a little bit in terms of where we really, I mean I’m with him on that side but on the side of the instruction the actual intervention in terms of the reading instruction is based on the five critical elements that are coming out from Reading First, so we are also testing out the highest level of instruction for those areas within that particular intervention also.

DR. FASHOLA: My response is, I think it’s a little more of a question. Are we attempting to do this to advance the science or are we attempting to do this because we want to be like the big boys in the sciences as I was said and why are we trying to do this? So I think trying to advance the science and trying to make things scientifically sound, etc., but the question again as somebody who’s doing educational research, the question behind all that is that what are these answers to these research projects, as I embark upon mine, what is the answer to the answer project going to do to and for the field of education, theory, research and practice? Money that is coming out from our federal government today, yes while it is attempting to advance the science and advance the sciences and try to be like the big boys, the money that is coming out of there is being, I mean it’s coming from money that is allocated for schools, for teaching children on a daily basis, so as I embark upon my studies my goal is to advance the field of education and to be able to provide solutions to the field of education that are theory and research based that will show that if this is done with this solid design the eventual outcome will be that all students are learning “no child will be left behind” but that there are effective methods that can be used either through pre-packaged or non-pre-packaged but ultimately there are effective things that can be done for children that will advance their education and increase their academics. I mean it’s hard because there is some, as a psychologist, somebody who has a background in psychology, you know you do different types of studies, some are just exploratory studies and some are interesting studies, and then there are some studies that are done because you have this certain intervention that you’re trying to apply because there is a problem.

In the field of education there is a problem, students are not achieving, whether we’re like the hard sciences or the soft sciences or the middle sciences, students are not achieving and so as I looked at this study for example I look at a practical thing that happened. Around the time that this study was taking place the Mathematical Study came out, and I’m not going to take one side or the other but one of the things that the Mathematical Study showed was that after school intervention was not effective, but it was after school, and please correct me if you’re in the room if I’m wrong, but that basically after school was not effective academically. I would go to sessions after that and people would say after school programs that are academically based do not work.

Around the time that the study came out they were talking about cutting 21^st Century Community Learning Center programs because they do not improve academic achievement. One of the things that our study is trying to do is to say if and when you have an after school program that is academically based that is planned, that has a rigorous research design, yes, you can have effective academic programs. So in addition to being like the sciences but the other thing is once again to serve the students that are so much in need of the programs that they’re being studied for. Does that make sense?

DR. FLETCHER: Thank you. The next question is from Barry Sloan, who’s going to also tell us what federal agencies are going to do to get money to people in time so they can actually do studies in schools.

DR. SLOAN: I think the first we learned, you’ve got to take my comments with a grain of salt being Irish, is that we need to provide a lot more lead time to studies. One of my colleagues said that maybe the no cost extension should come in front of the study rather then after the study. That said, one thing that was common, I want to make a comment and ask a question. And the comment is each of the studies presented were central to reading and on one of the central features of the current reading program is common measurement, and measurement as a phenomenon has not really been spoken of. But it’s kind of hard to go to trial if you don’t have measures in place. So that’s just a comment. We don’t see people from math or science at the table, which is certainly troubling for us at the NSF.

DR. FLETCHER: You can talk to Lisa about that in the hall afterwards.

DR. SLOAN: I’m sure they’ll have a panel on measurement in math and science, don’t worry about it. I think there’s a sort of a, as a statistical analyst I find one troubling issue with the, less over Shep’s, but with the presentations, and it’s the notion that teacher effects can be treated as fixed effects rather then random effects, fixed with some variable effect. In the first study on the tutoring the student is clearly the unit of analysis, I’m not sure what the analytic cumulative analysis is in David’s study, one they haven’t done the analysis, but in terms of the power structure it would suggest that it’s much closer to the student then it is to the school irrespective of where the randomizing features went on. In Shep’s analysis in the growth curve analysis, while he would have some sense for what teachers do differently in that analysis unless the teachers are again nested in some other level then there was no capacity within the model to treat the teacher and put as possibly being random rather then a fixed effect, it might be fixed with some variation. In real terms if you’re thinking about bringing this stuff out into schools is a major, in my mind is a major drawback to the studies as presented. Clearly it has huge effects on sample sizes and costs so I can understand why people would be happier to reside in a space where the student is the unit of analysis, if you can get at students more readily if you get one teacher, you’re going to get close to 30 students depending on the design. But the idea that teachers bring a fixed effect to students, while it would be nice if it was true. Thank you.

DR. FLETCHER: Do you want to comment David?

DR. MYERS: Like you say we haven’t done the analysis but when we conceive of the design we actually look at is a multi level design, I don’t use the same terminology that Shep does but when you do our power calculations for example we do think of it, we do take into account that there’s between school variance for example. And so the actual power is somewhere between having the student and the school. In this case we had one teacher assigned to each school as offering the instruction, you could think of other ways of randomly assigning a pool of teachers. In this case it just logistically one had to do it this way but we do take into account that there is clustering going on within the school, so we do conceive of it as multi level like Shep. I hope that answered your question.

DR. SLOAN: If my memory holds me you had 41 districts and you had 40 schools to represent the 41 districts --

DR. MYERS: No, we had a pool of 40 schools, there’s no representation of 40 districts.

DR. SLOAN: Okay. But you say now you have 40 teachers, one per school.

DR. MYERS: Right.

DR. SLOAN: My question was are you treating the teacher intervention analytically as a fixed effect?

DR. MYERS: No --

DR. SLOAN: So your analysis is within and between teachers or within and between teachers between schools? At your multi level?

DR. DURNO: It’s between the four programs.

DR. MYERS: I think we’re using different terminologies and I need to think a little bit more about this but in the simplest case 40 schools, 40 school units have been randomly assigned let’s say to two approaches, one teacher is in each school. So in a sense teachers have been randomly assigned, you could think of it that way, to each of the interventions. Then within each school students are randomly selected for the treatment and control group.

DR. SLOAN: But you are making, analytically then you’re nesting students within teachers.

DR. MYERS: Yes.

DR. SLOAN: At the second level of that model the teacher effect will be treated as a fixed effect, it has to be.

DR. MYERS: Why is that?

DR. SLOAN: Because you’re going to have students nested within teachers and if you’re using the HLM(?) or multi level modeling tool that will be a fixed effect, it may be, you may allow the effect to vary but you won’t be able to estimate more then a fixed effect.

DR. MYERS: Right.

DR. FASHOLA: Ours is the same thing, I think the only, we do mention that, I think the only thing that the teachers have across the board is that we required all credentialled teachers. But the thing that one should note though is that once again this is a voluntary program, this is an after school program so it’s not even the regular school day program but that is the case, it is a voluntary program. And I’m seeing this as a limitation to the study, also that the teachers, not only were the students volunteered but the teachers also volunteered themselves into the program, that is very different from a program that would be offered during the regular school day. But I do agree, yeah, we’re treating it as a fixed effect and it is a limitation.

DR. FLETCHER: Do you have other questions? Gil?

DR. GARCIA: Gil Garcia, Institute of Education Sciences. I appreciated Dr. Kellam’s comment about the number of teachers in the control group who already were good classroom managers because I think it’s related to the point that we’re making in this latter part of the conversation. My question has to do with how you’ve gone about, clearly we’ve got three different approaches to reading across these three projects, I can’t imagine that you’ve got, whether you’ve got a package or not that the reading interventions are identical interventions, so you’ve got, you all are focusing on different aspects of reading in each of the, in each of the projects. So my question was more akin to what have you done to understand about reading in the control groups to ensure that your intervention is distinct from what would normally occur in the school? I mean how have you gone about trying to understand that given that at least in David’s case you did this random net search for these schools so you didn’t spend too much time investigating what’s present in these schools before your intervention is going to be in place. So in your case maybe you can’t answer that question but for the other two projects, how is it that you went about trying to untangle or trying to identify the distinctive features of what they already were doing because from my understand I think what you’re saying is you’re assuming that whatever the schools are doing is either, has got to be different from what you’re proposing to do or what they’re doing is not good enough because your intervention is supposed to turn them into readers.

DR. MYERS: In our case we did talk to the schools and I can remember a number of conversations where the school said we’re already doing this failure free reading or corrective reading or something and we walked away from those schools because they were already implementing it there. Now this was a very intensive intervention, one of the things that made it distinct was the 100 hours of instruction within roughly a five and a half month period which made it distinctive from what would already be going on in the classrooms for these kids. These kids were also kids identified as needing remediation, they were at the bottom of the heap, and typically the teacher wouldn’t be teaching just to them in the classroom. So yeah, a fair bit of work did go into it and I think as Judy pointed out you want to always measure what’s going on in that control groups very carefully, want to spend as much attention on them as you do in assessing the fidelity of the intervention, so we are doing that.

DR. FASHOLA: In our study, and I’m sorry I did not bring this up in one of my slides, but in our study prior to going into the schools we knew that they were not SFA schools, in fact that was one of the terms, that the Foundation had a list of schools that were SFA schools. In addition when we went into the schools we ensured that even though we believed they were not SFA schools, which is the curriculum that was used, that they were not indeed using the curriculum. So that was the first thing, and then secondly as was said before, the kit and the materials were only in possession of the teachers who were in the treatment program. An interesting thing though was that one of the schools was a former SFA school, long time ago, in fact I believe two of the schools and maybe more but I know for a fact that two of the schools had incorporated the curriculum a number of years ago but as is the nature of school wide reform every three years you go from this to that to that, like there were some teachers I believe who had been in previous schools who had used the model who were now in this school that had never used the model. But when the curriculum was initially, the teachers in the treatment group some of them when the curriculum was presented they remembered some of the things like the little books that we used that had not changed. But what we did know was that teachers during the regular school day, even if the school had been a previous SFA school or if the teachers had used the model in previous schools, they were not implementing it during the day or after school in their sessions. So we did know that. Does that answer your question?

MS. CHINNIA: I just wanted to say that I think one difference in our model is that we were using our regular reading instructional program, open court is the instructional program, is the resource that’s used in our reading instructional program, all of our schools were using it, that is the standard. The intent of what we were doing in terms of the enhancements in the trial schools is to do just that, to enhance the open court program, especially looking at the portion which we discovered through our own internal observations and analysis was weak and that was the workshop portion which was differentiated instruction. And so we were looking at specific strategies to make that portion of the instruction better.

DR. JONES: Vinetta Jones and I’d like to make a comment and ask a question. First I’d like to say that I’m not from NCADE(?), I’m from Howard University, and we did go through NCADE reaccredidation last year and I’d like to correct the comment that was made about no requirement for our classroom management. In fact it is much more rigorous then that. We for instance are an urban school of education focusing on issues of diversity, and we have a model for what our students have to know and be able to do that’s research and theory based, and there’s an outcomes assessment. You can’t just point even just to a course, you have to show, demonstrate what your students, your graduates can do and you have to show through their experiences, a variety of experiences with documentation of where they had opportunities to learn those skills and where they demonstrated it. So with it still so clearly in my mind because we did use that whole effort, we were three years in the making for that as a vehicle to look at our whole program and update it and so on like that, it was a lot of work but I think that our program and our students benefited from it.

My question has to do with David. I was wondering about the teachers that you were referring to that were selected as a result of assessment aimed at getting the best teachers, now was that for both the teachers who were doing the experiment and the control and, well my question really is how do you know the extent to which any effects that you will be seeing are not just on the basis of what a good teacher would do? Also you mentioned that they were the real first grade teachers and that there were long term subs placed in their classes for a year, and I was wondering if you’re getting reaction to that, if I put on my parent hat I think that if my first grader came to school and I found out that they were going to have a long term sub for first grade that I would take them to another school.

DR. MYERS: First, the teachers that were selected, these are teaching third and fifth grade, the teachers were nominated by the schools to participate, to provide the intervention and then we gave them a screening test. Donna, do you want to comment on this issue of reaction to having these full time subs in your sense of the difference between the full time sub and what might have been there?

DR. DURNO: It was the school’s choice for the substitute and we’re essentially paying them to pay the substitute, both the wages and the benefits, for the whole year. And the schools would choose, schools have a list of subs and they have their favorite subs, and so they would bring in, because schools are interested in children they would bring in their best substitute or their best long term substitute. So we haven’t had any issues at all with that, there’s been no complaint of any kind because I’m sure they brought in a qualified teacher. The three teachers were nominated by the school, the teacher was selected, and other then union reaction to the selection of the teacher there wasn’t any. And the assessment was geared to determine how well they could quickly adapt to this new intensive instructional model and so that we didn’t have the concern of having teachers not able to start the intensive instruction at the time it was due to start.

DR. KELLAM: Well my question is a question actually, I’m curious, I’m very interested to know, when you were talking about NCADE were you talking about NCADE requiring schools of education to have such programs? Because the last time we looked actually was part of the faculty at Morgan that did a search and must have been about three or four years now but they didn’t find any requirement.

DR. JONES: I was referring first of all to the NCADE 2000, which are the new standards and a whole new generation of it, and I was referring to the fact that things are standards based and what it is that you want teachers to be able to do, then you go back and look to see where in your program they have those experiences and course are just one of the kinds of experiences. Yes, we do and every place that I know of has within a course experiences and readings and so on, classroom management. My point was that it’s more then just a course, it’s not even anymore, you can’t pass standards or satisfy standards by just identifying a course.

DR. FLETCHER: Thank you.