THE NATIONAL RESEARCH COUNCIL

AND

THE NATIONAL ACADEMY OF EDUCATION

WORKSHOP ON

SCIENTIFIC INQUIRY IN EDUCATION:

IMPLICATIONS & NEXT STEPS

January 10, 2002

National Academy of Sciences Auditorium

2101 Constitution Avenue, NW

Washington, DC

Proceedings By:

CASET Associates, Ltd.

10201 Lee Highway, Suite 160

Fairfax, VA 22030

(703) 352-0091

TABLE OF CONTENTS

Page

Welcome & Overview:

Dr. Feuer, NRC 1

Dr. Noddings, NAE 2

NRC Committee on Scientific Principles in Education

Research, Findings and Conclusions of Scientific

Inquiry in Education:

Ms. Towne, NRC 16

Dr. DeHaan, Emory 22

Dr. Boruch, University of Pennsylvania 30

Dr. Fletcher, University of Texas, Houston 36

Discussants:

Dr. Apling, CRS 45

Dr. Graham, Harvard University 52

Dr. Grissmer, RAND 65

Q & A 78

Report Out From Focus Group Moderators:

Research - Dr. Noddings 124

Federal Policy - Dr. Hakuta 127

Associations - Dr. Berliner 131

Schools - Dr. Feuer 136

Invited Closing Remarks - Dr. Conaty 143

Q & A 155

P R O C E E D I N G S (8:25 am)

Agenda Item: Welcome & Overview - Michael Feuer, Director, Center for Education, National Research Council

DR. FEUER: Welcome to the National Academy of Sciences, to our cozy little Academy. I'm very grateful to all of you for joining us today. This is, as all of you know, a big week in education policy in America. The president, the day before yesterday, signed a landmark legislation called No Child Left Behind. And we are meeting already today, just 48 hours later, to discuss an issue which, as most of you probably know, is in some ways quite central to the current wave of thinking about education policy and reform in America.

I understand -- I have not read the entire law line by line yet, but I understand the phrase, "scientifically-based research" appears over 100 times. And so a report on scientific inquiry in education is indeed timely. And I'm very gratified that with the foresight with some of our friends in the federal government, and with the hard work of our volunteers and staff, we have been able to produce a report that I think is really in many ways, significant to the current debate.

This morning I just doing to say a few brief words, and then I mainly want to introduce Nel Noddings, the president of the National Academy of Education, who is here with me. And then I will say a few words about the NRC, and about what we are doing today.

But again, let me start by thanking in particular, Kenji Hakuta, who, in his capacity as chair of the policy board, had the foresight to encourage the NRC to take on this kind of study. As he knows, the time between the policy board approved the study, and when we released it, the policy board itself didn't meet. So we were very proud to be able to provide meeting-to-meeting service with our report. This one was about something like 14 months in the making. It took the work of a number of distinguished scholars and superb staff. I will mention their names in a moment.

So with that, let me subside, and I want to turn this over to our special guest, Nel Noddings.

Agenda Item: Welcome & Overview - Nel Noddings, President, National Academy of Education

DR. NODDINGS: Well, thank you, Michael. I am happy to add my welcome to his, and say a little bit about what the National Academy is, and what we are hoping to accomplish.

The National Academy is now almost 40 years old. It was established to promote inquiry. And in the original document it said, "inquiry into the means and ends of education in all its form in the US and abroad." And I think that holds true. We are interested in education in schools of course, but education takes place in places other than schools, and we are well aware of that.

Some of the things that the National Academy has been involved in are well, first of all, I guess the thing we are best known for is the post-doctoral fellowship program that we administer with the generous funding of the Spencer Foundation. Now this is an important way really of promoting educational research, because we identify and nurture -- the word that we prefer to use -- young scholars. People who are particularly talented in doing work associated with educational research. So we have been doing that post-doctoral program for about 16 years now.

We also run some projects. That is, we sponsor or co-sponsor some projects. And we have a major project going now in teacher education. And our question is, how can we better fulfill this original mission, which is to promote scholarly inquiry in education? At this stage we are very much looking forward to collaboration with other academies, with the National Research Council, and with practitioners and policymakers.

I want to point out a sort of systemic problem that we have in research. People sometimes look at educational research and they say, this is terrible stuff. Educational research is bad. Well, there is bad educational research. There is bad research I think in almost every line of scientific enterprise.

There is also some very good research. But, if we are going to make it better, it requires something other than just concentrating on educational research as a separate enterprise. That's one of the reasons that we are so excited about collaboration, because the connection among practitioners, policymakers, and researchers is extremely important. It influences the whole enterprise.

Now let me give you an example of the kind of thing that I'm talking about. It is sometimes said, and you hear this complaint a lot, that practitioners don't read educational research, they don't know educational research, they don't apply educational research. Well, that may be true. It's at least partly true.

But on the opposite side of the picture, there is an even more troubling problem. And that is that policymakers often seize upon research before that research is ready to be used. That too, is an extremely important problem. And we can see now how all of these things are connected, because if researchers are going to take their time and do a really thorough job before releasing results, it means a sort of different way of funding research. It's trying to ease up that pressure for immediate results and immediate application. Take the time to really look at things.

Let me finish up here by just using a medical analogy. When we do drug tests, it's wonderful to find out that a particular drug has an important effect on a particular condition. But see, that's not enough to know that. We then also have to know whether this drug has other kinds of effects on particular people -- on the very young, the very old, pregnant women, people with other ailments, people who are taking other medications. And sometimes we only find out about these things through tragedies.

Now in education we are not facing the same kind of physical tragedies, but we are facing exactly the same kind of problem. That is, hasty implement of results that are not as solid as we would like them to be, can be a very great mistake. So envisioning something for the long-term, what I would like to see is long-term funding for projects that would involve monitoring and evaluating and revising and changing and reporting. And meetings of this sort, where we have an opportunity to talk with one another, and correct one another's error.

So I very much appreciate your being here, and I hope we have a very productive day.

Thanks.

DR. FEUER: Thank you, Nel. It's exactly the kind of background that helps me launch into three minutes worth about this Academy, that is the National Academy of Sciences, where you are sitting this morning. The story of this institution is really quite germane to the purpose of today's meeting. So if you will indulge me in two minutes worth of this, I promise not to tell all of the same jokes that I tell about the history of the Academy, so that some of you will keep coming back.

This was an institution chartered in 1863, by an act of Congress, and articles of incorporation, the charter was signed quite enthusiastically by President Abraham Lincoln. This was an innovation in its time. It was a variation on a concept that already had existed in at least some other countries such as the Swedish Royal Academy of Sciences, which has been established in 1739, I believe.

Lincoln's innovation was in some ways, a reflection of the peculiar genius of American democracy. And that is that in granting a charter to the National Academy of Sciences, he attached a proviso that required public service. In essence, a requirement that on request, this Academy would advise the federal government on matters of science and technology and public policy. And furthermore, do it pro bono.

And since then, this has been a somewhat radical experiment, I think, in the application of scientific thinking, and in the scientific enterprise to the betterment of public policy that is indeed the mission.

Now another very interesting variation that was reflected in the charter of the NAS was that from the beginning, the behavioral and social sciences were included. And this was again, a departure from the European tradition. So anthropology and psychology were really part of this enterprise from the get-go in a way that in the European academies, the separation was much more stark between the natural sciences and the behavioral and social sciences.

There are some very interesting archives and histories of the work that the Academy did in the early days. Some of it was on aspects of human behavior -- training in the workplace, the psychology of sexual behavior. Of course in the beginning, a lot of the work was what we call sort of more kick the tires kind of technology and analysis.

And in fact, one of the first -- not the first -- very interesting problems that this Academy had to solve had to do with an issue in the Civil War, which had to do with the ironclad ship. Which, as you may recall, was an invention that ultimately enabled the North to actually win the war. The problem with the ironclad ship, however, was that they couldn't get the compass to work because of the magnetic field. And so this was one of the first projects turned over to this young Academy.

And as you can imagine, it was a quite example of the intersection of technology and public policy, because in a situation such as the Civil War, you do want to get north and south right. And getting the compass to work on the ironclad ship was no trivial matter of public policy. I am happy to report that the Academy's results on that study are now cleared through review, and we will be releasing them shortly.

Education became a focal point of the Academy, and more specifically of the National Research Council when it was established by executive order in 1918, or right in that period. This was during the days of the first world war. And President Wilson came to the realization that the Academy had provided some extraordinary support for the federal government, and encouraged an expansion of the franchise to include people from business, and experts who were not elected to the honorific society of the Academy.

Today, this enterprise, the National Research Council and the National Academies produces a report just about every working day. Last year we had 230 reports released from the National Academies. We work with a group of anywhere from 7,000-10,000 volunteers who serve on our boards and committees, many of you, many of these people, and a staff of about 1,000, who are altogether quite committed to the idea of science and the betterment of public policy.

It is in the last eight years, perhaps most powerfully manifest under the leadership of Bruce Alberts, the president of the National Academy of Sciences. We have released just in the last eight years, over 150 reports just having to do with education. And this reflects Dr. Albert's abiding passion and commitment for the application of science, and the humanitarian and honest and ethical and rational ethos of science to the improvement of education.

So this report on scientific inquiry in education is really the result of questions that were asked about the basis of scientific method as it applies to education research. And as I indicated, it was a request from the Office of Educational Research and Improvement Policy Board, the NERPPB, National Education Research Policy and Priorities Board, that we undertake a kind of inquiry into the scientific basis of education research, or more precisely, aspects of scientific education research. That's a distinction that I hope comes across in the report, and that I think my colleagues here today will have a chance to say more about.

So this is a great pleasure to able to go more public with this report, which was just released about a month ago, and to have a group of folks who are from the worlds of research and policy and practice to explore some of the issues in this report, and to think about next steps.

I want to get right into our program now, and first say a few words of thanks to some people who are not here. First of all, to Rich Shavelson, a great friend of the NRC, and a great scholar of education and public policy, who, in another weak moment, answered a call from me and agreed to chair yet another NRC panel. And this was a daunting challenge, because of how little time we had, and how complicated and tough the questions were.

Rich couldn't be here with us today, but as I told him on the phone yesterday, he is certainly with us here in spirit, and his footprint is quite clear in the report, and in a lot of what you will hear today.

I also want to thank in addition to Nel, who, working with the National Academy of Education, has agreed to try to collaborate with us on some of this. Kerith Gardner, who has been so ably helping us with staff work on this. And then people on our staff, Tina Winters, and Teresa Williams, and Linda DePugh, who were an extraordinary team putting together this event today. Just managing the logistics of this is no small thing. We are deeply indebted to them for their hard work.

Adam Burns, who helped us sort of in the final hours, set up this afternoon's program. You will see later on what an impressive bit of work that is. So we thank Adam.

And finally, let me just introduce the person who really was -- well, first let me introduce the panel for today, and then at the end of that I will introduce Lisa. This report that is the basis of today's conversations, like all NRC reports, was written by a committee of scientists and experts who worked as volunteers on this effort. And some of them are here with us to share the wit and wisdom of this report: Bob Boruch from the University of Pennsylvania; Robert DeHaan, who is actually now on the staff of the National Research Council, but was on this committee in most recent prior capacity as a professor at Emory University; and Jack Fletcher of the University of Texas are here on the panel, and we're very grateful to them.

Unfortunately, Ellen Lagemann could not be with us today. I'm sorry to tell you that Ellen lost her mother yesterday. We extend our condolences to Ellen, and wish her long life and good luck.

And finally, let me introduce Lisa Towne, who was the study director for this project. And if any of you are wondering how you get a committee of experts and scientists to agree both to the concept of writing a report like this by consensus, and then actually doing all of the work, it's because of the glue provided by the staff. In Lisa's case this was an extraordinary actually first effort here at the NRC for her. And so it's a great pleasure to introduce her. Lisa will tell you more about the logistics of today, and other things.

MS. TOWNE: Thank you. Hi, everybody. Welcome again. I'm going to just briefly walk through a couple of the key pieces of our agenda, just so we are square on the program before we get started. And then I'm going to start off the presentation that my esteemed subpanel of the committee that actually authored this report will then deliver.

So let me start just briefly with the agenda. You probably have looked through your packet of materials already, and just want to make a couple of key points that I wanted to make sure that you saw. First of all, we are recording this event. That's why we have got microphones. We will making a transcript of it, and posting it on the Web. That's just something that we like people to know in advance, so they temper their remarks -- no, you can say whatever you want, but it will be public.

And that also means that when we have question and answer sessions at various points in the agenda, if you could come up to one of the microphones that are in the audience and state your name, that would help the transcription process tremendously.

Just a few things on where places are in terms of various agenda items. You will see that there is a map in your packet for some of the room locations for later this afternoon. You should familiarize yourself with that. We'll also be trying to help direct you, and have arrows on walls and such to help you make your way through what is somewhat of a labyrinth back here.

[Administrative remarks.]

That is about it. What I wanted to spend a few minutes on in terms of logistics is the afternoon program that Michael mentioned. This is something a little different that we are trying out at the Academy. We are actually putting together some focus groups. There was a sheet in your packet that was called, "Focus Group Methods and Instructions." It has a little bit of information on this. I just want to go over this briefly, because it is important. You all individually play an important role in this.

We are setting up these very small groups to try and gather information about how you think the National Academy of Education and the National Research Council could collaborate and continue work on this important topic of discussion today, and have a specific idea that we actually want to run by you and get your input on.

So we have structured these focus groups to be small group discussions. We had to do some selection, some not so random sampling. It's actually very purposefully selected. You will notice that some of you have a colored dot on your name tag. If you have a colored dot on your name tag, that means that through selection criteria according to your background, and the size of the group that you are most like, we made decisions about putting you in these groups.

If you have a C also on the back on your agenda, there is a categorization by color of what group we have put you in if you do in fact have this dot. If have a dot, we have then put you in a focus group, and we are expecting you to attend. So if you actually were not planning on behind here in the afternoon, and you can't be in this focus group, if you could let someone at the registration desk know that, so we can make any last minute changes to get these groups to be a good size and representation. That would help us tremendously.

If you do not have a dot on your name tag, you are most likely someone from the Department of Education. We have a lot of you here today. We're very pleased you are here with us today, but we had more people here than we could accommodate in these small groups, so what we have done is in each one of these rooms is put together some chairs where people can sit around and listen in on the discussion. You are more than welcome to do, and to sit in on any group that you choose to.

If you do want to provide formal input to us, you are more than welcome to do that in writing at the end of the day, and leave that with us. So we thank you for your help in trying to pull these off, and to let us know if you can't be here, or if you think your categorization is off. Many of you could fall into a lot of different categories, and we do have some space constraints, but if you think you have been grossly miscategorized, you can also let someone at the registration desk know, and we'll try to do some midcourse shuffling.

I think that's it on the focus groups for now. With that, why don't we go ahead and get to the meat of all of this. With this first slide I just want to say that I am aware that it is 2002, but I didn't actually notice that until we just got up here this afternoon.

What I'm going to do is just start off this presentation, and then my colleagues will pick up were we left off. I'm going to be basically covering the material that Ellen would have covered, had she been here. So I will do my best to fill those very big shoes.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Lisa Towne, National Research Council, for Ellen Condliffe Lagemann, Spencer Foundation and New York University

MS. TOWNE: I'll just give you a brief bit of background that Michael has already given you. This is sponsored by the NERPPB, the National Educational Research Policy and Priority Board, and our work began in the fall of 2000. In fact, the committee first met in December 2000. So the report was produced in less than a year's time, which by NRC standards is lightning speed.

The other thing I will mention about this particular slide is this is sponsored not only by a bill that is pending in Congress to reauthorize OERI, that many of you are aware of. This bill in part defines what scientifically-based educational research is, as does the new No Child Left Behind legislation, which prompted a discussion that actually has been going on in the educational research community, and related communities for a very long time, about 100 years. So this is not new, it had just kind of a new policy spin to it when we took this up.

Dovetailing from that, there are a couple of goals that we hope this report will serve. One is in part, to inform the pending reauthorization of OERI. There is a chapter in the report that takes up specifically the implications of the committee's work for the federal research agency in education, and Jack is going to talk more about that later.

We also hope that it informs more generally, this ongoing trend, as Michael mentioned, for scientifically-based education research. This has been a priority of the Bush administration since they have come to town. And we know that it is something that they want to take very seriously, and we hope that this report is a timely contribution to helping make that a reality.

And finally, we hope this report starts a bit of self-reflection within the field of education research and related communities. It's certainly noteworthy that the Academy was even asked to write this report, and related that the legislative body tried to define what education research was in a political arena. So we hope that this report, while obviously not the last word, can help spark some debate and some self-reflection about what that means, and how to move forward from here.

This is probably not readable. I won't go through the committee membership in much detail. I just want to point out that most NRC committees, it's also in your report, so you can actually read it in 12.5 font if you would like to do that.

I would just point out the diverse perspectives that we had on this panel. We have people who are historians, anthropologists, psychologists, sociologists. We have people who are experts in statistics, economics, philosophy. Bob DeHaan actually is a cell biologist by training. We also had a research chemist on this panel. So as you imagine, putting 16 people of these different walks of life into a room and asked them to tackle these questions was quite an interesting and challenging thing we did.

I'll just give you a little bit more specifics about our charge. We were specifically asked to consider the nature of scientific education research. And the word "science" here is important. We weren't asked to talk about education research or scholarship writ large, which is obviously a larger body of work. We were asked to talk about science, and given the fact that this was a panel of the National Academy of Sciences, that made sense to us. And in particular, as I said, how a federal agency could take these general ideas and build them into their own work.

An important part about the way this committee went about their job is that we did not, in addressing these questions, evaluate existing literature in education research for its quality. We didn't evaluate existing research areas, I'm sure a lot of you in the room would be happy to know. And we didn't evaluate comprehensively, the existing chief federal education research agency, OERI.

We didn't think that actually we could pull all of that off in time that we had. We also didn't think it would be particularly constructive. And so what we have done instead, the committee decided to take a more forward looking approach, to take as a given that there is skepticism and doubt about the quality of education research, and try to set out some principles that fine that, as a way again, to spark some discussion and move forward.

And the same is true for the federal agency. We did do some background work on OERI and related social science agencies in the federal government to try and get a comparative sense of how these agencies do their job, but we did not comprehensively evaluate the work of OERI.

What we are going to do is walk through chronologically, the chapters of the report. I'm going to briefly talk about Chapters 1 and 2, which again, Ellen would have covered would she have been here today. I'll actually not talk too much about this. I'm going to just skip through a lot of this.

This is very important and interesting context, however, that I would encourage you to read, because it shows that in fact there has been a tremendous amount of progress in the last 100 years in thinking about the epistemology of education research, and the nature of scientific research in education. And then in fact, the epistemology and philosophical work on the nature of science in general goes back for centuries.

So these questions are not new. They have been talked about for a very long time, and this introductory chapter tries to summarize very briefly, some of that thinking, and some of the evolution in our understanding of the nature of science, and the nature of social science, and education in particular as well.

That historical and philosophical context led very directly to some assumptions that the committee made when it set out on its work. Those assumptions are described here. One is that there is no one definition of science, and we don't intend to have anyone thing that this report actually offers any definition of science per se, but it is our characterization of that.

We have made a number of other assumptions, one of which is important to point out, and that is that the quality of education research is one aspect of the overall value of education research. In other words, we are not making any kind of assumption that the quality of the science in the research is perfect. That for example, research would be used in an effective way in schools, for example. So that there are other related questions in this larger set of issues related to educational research that we did not take up.

And finally, the last point may seem odd, but it says, science can uniquely contribute to understanding. We wanted to be clear that the committee feels very strongly that science can be a powerful tool in the improvement of education and teaching and learning and schooling. But again, that it is one part of a larger body of knowledge and understanding and insight, that when combined with scientific understanding, can be particularly powerful in improving education for all learners.

So that's just sort of a brief overview of the context and the assumptions that the committee had when it started its work.

If I could have the next slide on Chapter 2. I'm not going to spend a lot of time on this, because the real beauty of this chapter, in my humble opinion, is in the reading. This chapter just goes over a bit -- the report itself starts with the general, and ends with the specific. So at the beginning of the report we are tackling the broader question about how science as an enterprise moves. How knowledge is generated, and accumulated, and understood, and integrated over time.

And what we do is, through a number of examples from a range of fields, education included among them, is show that the way that science progresses is actually very common across fields, despite the fact that the objects of inquiry, and the type of questions that are asked can often be quite different.

And we talk about some of the conditions that we saw that tended to enable progress in the examples that we gave, as well as some of the common characteristics of the way that scientific understanding has moved over time. And so I won't spend any more time on that, but just point out that they did show some examples in a range of fields that show this, and it is quite an interesting read.

And from that broad prospective of how science as a whole moves, we then moved into a discussion of the core principles of science. We call them the guiding principles in the report, that we think apply broadly again, to all sciences. And to talk about that in a little bit more detail, I will turn it over to my colleague and friend and new person whom I get to see on a regular in the hallway and at the coffee machine, I'm happy to say, Bob DeHaan.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Robert DeHaan, Emory University

DR. DE HAAN: Thanks, Lisa.

Let me say that this experience of being on this panel with the Rich Shavelson, Lisa, and the other members has been an enormously gratifying experience, because the diversity of the group, the range of subjects of expertise from philosophy and cultural anthropology and hard science and so on, and the kinds of discussions that we entered into, some very deeply and very passionately, has been an enormously valuable learning experience for me, and a great way to make friends.

The question that we were dealing with, as Lisa said, is there, among the various fields of intellectual endeavor, a body of knowledge, a body of principles that can identify what is scientific from what is not scientific? And the answer, as you can gather from the title of the document that we finally produced is yes, there is.

Now the question is, how does one find those? How does one get to them? And so the guiding principles -- that's what is on this slide -- tell us about the sorts of discussions that we started with. First of all, it became apparent from discussions that started with the philosophy of science and the nature of science, that science is based on the concept of warrant. The concept of evidentiary base. What kinds of evidence warrant a given truth claim?

And any endeavor that bases itself upon that set of assumptions, if there is a body of information out there that we can investigate, and that we can provide evidence to give answers to, has the beginnings at least of a scientific endeavor.

Secondly, however, we sadly decided that there was nothing algorithmic about the principles that we eventually came up with. In other words, you can't simply say, okay, here are six principles. Now we are going to plug those in, and anybody who follows those principles is doing things that are scientific. That turns out not to be the case, as you would imagine, and we'll talk a lot more about the obverse of that, that is, what it is that really does define science.

One of the things that we decided does define science, and we talked a great deal about this, is in addition to the cognitive aspects, is the ethical aspects. There is a body, a core of community norms within what we call the scientific community that is absolutely essentially to the scientific endeavor. And unfortunately, I can't spend a lot of time on that this morning, or else I'll spend the whole hour that we have.

And so let's quickly go into what were the scientific principles as we defined them. And again, partly because we started so late, and partly because we don't have much time anyway, I'll just mention these, and you'll have to read the document in order to get the full flavor of what it is we are talking about.

Any scientific investigate, irrespective of whether it is dealing with being cardiac cells, which happens to be my field, or the star evolution, or social groups in deepest Africa, has to start with an answerable question. That is, one of the things that characterizes good science in every field is the ability of the investigator to select a question which is both powerful, incisive, and also empirically answerable.

And that is one of the critical factors, the questions that are selected for scientific investigation in any field are empirically answerable, are highly significant to the field, and where there is a body of theory from which to draw on that question. And that in fact leads us to principle two.

Effectively all, with very few exceptions, effectively all research questions are, all investigations arise out of a body of previous theory, a theoretical framework. Which in fact, since we are talking about in science, trying to get a generalizable response, a generalizable picture of some aspect of nature, the theory that lies behind the specific investigation that an investigator engages in is of course crucial.

Now it's not only true that theory generates investigations, questions. It's also true of course that questions generate theory, and in fact, that's a completely circular, complete feedback loop kind of situation.

Science uses methods that are variable at the time. Those are the only kind of methods that can be used. Either that the investigator invents on the spot, or that are well documented, investigations that can directly ask that question. So that if an investigator asks a question which is powerful, comes out of a great theoretical framework, but in which there is no way of asking the question, it's not science. It's not even a good way of spending one's time. So we've have got to have legitimate methods for doing science at any particular time, in any particular area.

A very important aspect is evaluating whether a scientific investigation is indeed science or of setting up an investigation. Is the rigor of the logic of the chain of logic that the investigator and his or her colleagues used in order to get from point 1, asking the question, to point X, which is providing some kind of answer to that question.

And that coherent chain of rigorous reasoning, as we called it, has to be something which is public, has to be something which is sharable with other colleagues, and with readers of whatever is going to be published later. The assumptions have to be clearly stated, estimates of error usually in most cases can be provided, and so on.

Starting back with the very earliest history of science was the concept that any valid investigation has to be replicable. If I do it in my laboratory, or in my classroom or wherever, I have to be able to do it again and get something similar in the way of an answer, and I have to be able to have colleagues who are equally knowledgeable, able to do the same experiment and get the same kind of answer.

And to the extent that that is true, we have wonderful science going on, or we have successful science going on. To the extent that that is false doesn't necessarily mean that we don't have science. It just means that there are variables that we haven't controlled. And those are the kinds of considerations that have be thought through very carefully in evaluating the scientific value of any particular investigation.

As I mentioned before, there is a set of ethical norms, community norms that defines the scientific community within any given discipline. It is that open society of researchers that is key to the validity of scientific investigation usually, because whenever one investigator does a piece of research, not only does that research have to be replicable, it has to be in effect debatable.

It has to be open to, and accessible to other investigators who are going to say, well, you have done this or that, and you've gotten this result. And you are attributing that to a particular set of variables, but the fact is that there is this ultimate explanation that would be just as good. And I, as a scientist, have to be open to those alternative suggestions. I should have thought of them myself, but at least if they are suggested by my colleagues, that is a very important part of the whole issue of what is scientific investigation.

Having said all of that, the fact is that we understand that there is no single set of principles, these or any others, that can as I said, algorithmically distinguish real science from pseudo-science, or from bad science. This is a very, very complicated issue that we are dealing with.

And I think we spend a lot of time in the dark, not shilly-shallying, but trying to examine as carefully as we can, what are those complexities that really need to be discussed and talked about and though through very carefully in order to distinguish what is good science, and what is bad science from what is non-science.

Now one of the most important questions that obviously we dealt with, and we had to deal with, especially in education research is, is what I have been talking about, valid for the life sciences, the physical sciences, the earth sciences, but somehow different for the social sciences? And I can tell you that with this very wonderful group, we had some extremely interesting discussions, sometimes passionate discussions about this subject.

And the conclusion that we finally came to, which was embodied in this document, is that these principles, and probably others, apply absolutely across the board in physical science, in natural science, and in the social sciences, and since education is a social science, in education research as well.

What differs is the vast, vast diversity of application that can be made of these principles. As you may have noticed, they are very general as principles ought to be. That's what differs. Social science has different guiding modes, obviously different methodologies, different approaches, bracketing and a lot of other kind of things that one has to do. The investigator as intervener is a concept in social science which is really quite foreign to many of the physical and biological sciences.

But the fact is what the research is able to control in terms of variables differs enormously between what I can have in a test tube full of chemicals, versus what a teacher has in a classroom with 25 or 30 kids. But the fact is that the principles are the same, the applications are very different.

And with that, I will turn the discussion over to my colleague, Bob Boruch, who will talk about Chapter 4 and Chapter 5.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Robert Boruch, University of Pennsylvania

DR. BORUCH: Thank you.

Let me echo Bob's earlier remarks about the value of working with such committees. It also reminds me about Kurt Vonnegut's sort about advice that Kurt's father gave to him. His father said to Kurt, always hang around with people who are smarter than you are. And then he added with a note of regret in his voice: For you, that ought to be easy. It is great fun to find out how smart people are in their own areas, and to understand how to link what they know to what one knows, or thinks one knows.

I was asked to discuss Chapter 5 mainly, and I'll do so. One basic principle here is that the methods used to produce the principal science evidence in the social, educational, and physical sectors evolve over time. The idea of probability sample surveys, although we can trace it back to the fifteenth century, including the history of the probability of statistical ideas in ancient Arabic and Hebrew literature, the fact of the matter is that that technology and its development is less than 100 years old.

The same is true of controlled trials, and a lot of other methods. So that the one major premise here is that these methods are going to evolve over time. That we ought to exploit the ones that we find trustworthy. And their is use is encouraged in this document.

The second major premise is that studies are scientific when they meet those principles of science that Bob described. And understanding the methods of generating the evidence is only feature of that series of principles.

The third basic notion is that the specific questions that are addressed ought to drive the methods that are used to produce the answers. This may seem kind of obvious once said, but the fact of the matter is that there is a fair amount of confusion out there about when this ought to be used and the like. And part of that confusion hinges directly on the fact that the question itself is not specified well enough, and may not be even empirically answerable to yield the scientific evidence.

Rather than get into the endless discussions about the character of what's valuable about which methods, we decided to break up the world in terms of the questions that are addressed in scientific research. Grace Fielding(?) suggested that that world can be divided into three parts. Asking questions about what is happening, which implies descriptions of one sort or another.

Questions about whether there is or is not a systematic effect of an attempt to intervene with what is going on to interfere with the educational world, with the classroom, whatever.

And questions about how and why whatever is happening, or seems to be happening is happening. That gets into thinking about, and doing empirical research on mechanisms, dynamics of what is going on.

With respect to descriptive questions, generally speaking we have endorsed the idea that high quality probability sample surveys are a good idea. We have had an uneven history of that activity in that country. Most recently, the National Center for Education Statistics has done a marvelous job in becoming a world-class producer of statistics in this arena.

We understand and depend on those kinds of data to establish statistical relationships. Certainly, in the case of NAEP, understanding that there is a strong relationship between the level of achievement, and the education of mathematics and the backgrounds of teachers in mathematics and mathematics education is important to understand.

The idea that in different kinds of settings, probability sample surveys may not be appropriate. Rather, ethnographic studies or anthropological studies might be the best way is also clear. And there are lots of cases in which, for example, what we understand about social phenomenon out there is pretty sparse, pretty scant.

I think of a specific example of understanding something about what happens when children who are poor, but who are relatively high achievers get scholarships. What the effects of monetary incentives on these children are. It is an interesting question, at least to some foundations that dole out scholarships to children.

Doing the reconnaissance on how children think about that money, how their parents think about that money by and large has to do with a perspective that one cannot get using probability sample surveys, at least at the front end, because one does not understand enough about the phenomenon to even pick off the right questions that will be addressed in the probability sample survey. And as a consequence, what we know is based in advance of the experiments and of the probability sample surveys. It is based on work that is ethnographic.

With respect to causal questions, we also are reasonably clear in the report, suggesting that the probability sample surveys and the ethnographic work is largely narrative. Its character has lots of provocatory power. It is producing ideas, elevating the level of discussion, but has limited explanatory power for generating answers about what causes what.

Here again, we divided the world up into when randomization is possible, and when it is not possible. We have encouraged the idea that regular science is the best way to achieve fair comparisons. That is to say, generating statistically unbiased estimates of relative effects of interventions in a reasonably legitimate statistical statement which brings confidence in the results.

We have also emphasized the fact that they are used in many disciplines, certainly not only medicine. And the fact that although they are "relatively rare" in education, there is lots of opportunity to do far better work. The claims that experiments are impossible, made by some scholars, including in some university settings, is simply not true.

Even worse, there is a fair amount of ignorance out there about the extent to which trials have been run in a variety of education settings, including large scale trials in which schools and school districts, not to speak of hospitals, geographic regions, police hot spots, and neighborhoods are the units of random allocation in analysis in attempt to intervene in the system to enhance the well being of people who are in the schools, neighborhoods, and so on.

We certainly concluded that experiments are not possible, and would not be desirable activities. Understanding that in the non-experimental settings, one can evaluate or estimate the effect of almost anything, but a lot depends on the assumptions one is willing to make and tolerate in any of those assumptions for economists and statisticians and theoreticians, all heroic, which is one of my favorite lines, which is epochful.

It comes from a Nobel Prize winner in economics, who, at a meeting of the Labor Department, discussing whether labor ought to invest in experiments, opened up with the phrase, "assume for the sake of simplicity that people live forever." For him, that was a tolerable assumption. Others found it stunning.

The bottom line here is that certainly we have to rely on non-randomized trials at times. We have to learn a lot more than we know now empirically about the extent to which the randomized produce the same estimates of relative effects as the non-randomized trials. And there are a number of efforts going on in the area that are identified in the text.

With respect to mechanisms questions, simply because one can generate an unbiased estimate of the relative effect, that doesn't mean we understand exactly how it works. There is still a fair amount of research, for example, re-analyzing data from the Tennessee class size experiments to understand exactly why those reduced class sizes lead to a reasonably discernible effect on those kids.

Here again, we reach to our colleagues in the ethnographic community, to other folks engaging in so-called designed studies. The main objective is to be as careful and open as possible about how things are supposed to work, what the dynamics are, or various dynamics might come into play for example in how small class size might produce those effects on children.

I think that's the end of my tour.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Jack Fletcher, University of Texas, Houston

DR. FLETCHER: Well, I would be remiss if I didn't also acknowledge the joy of working with people that are generally smarter than you. But it would also be unfair if I didn't acknowledge how fun it is to work with people who are just as tired and irritable as you are at the end of a meeting.

And this next section is undoubtedly the part of the report that produced the most fatigue and irritability amongst the members, because was sort of hard to get a handle on exactly what we were supposed to do, and what we were supposed to evaluate. Part of the problem that was actually an advantage was that we were not the first report of this sort. There were actually reports that go back to 1958, that talk about what educational research agencies should look at.

And what we found most helpful was a deliberate decision, because it wasn't in our mandate, not to evaluate the Office of Educational Research Improvement, but ask broader questions that were more forward looking. For example, the question that we really focused on was how could a federal education research agency facilitate the development of a scientific culture of research? Really foster a community of researchers.

If you read Ellen Lagemann's eloquent history of education, it is clear that one reason that educational research is often fragmented is that there isn't a coherent scientific community of researchers who have a firm commitment to science.

So we were really focusing on issues that might involve a culture, and the development of a culture, as opposed to approaches that try to introduce mandates, which is the usual approach taken to develop education research. And we thought in general that you really can't prescribe good science through mandates.

We recognized that there are a number of dilemmas. That education research has to be grounded in practical problems, but too close a relationship to practicality and what works on a day-to-day basis has a significant downside. We also recognized that there are all kinds of people who are involved in educational research through different forms of scholarship, through different disciplines involved. And while our mandate was to focus on science, we stated very clearly in the report that we felt that there should be federal support for other forms of scholarship -- libraries, history, and so on.

We came up with six different design principles that we hoped would foster a culture of educational research, and I'm going to go through these briefly. The first one is to staff the agency with people skilled in science, management, and leadership. And by that, we mean simply that the director of a federal education research agency should be a scientist. It should be somebody who is in themselves, a researcher.

They should have a term that is independent of political terms, to insure continuity of the research program. They should be free to hire the best person possible, and they people all can be researchers independent of political constraints. We noted that you cannot attract people to an underfunded, understaffed agency. There are better jobs out there for solid researchers.

We think that it is very important to have a federal education research agency to facilitate interactions with other federal agencies that engage in educational research, like the NICHD or the National Science Foundation.

The second design principle was to create structures what would guide the agenda, to inform the funding decisions, and monitor the work. And what we mean by that is that the agency itself should be free to interact with the field in developing an educational research agenda. They should not have to work as closely with mandates, but should interact more closely, for example, with the governing board that also has an independent agenda setting committee.

The governing board could be modeled after the National Science Board. You could have an agenda setting committee that was chaired by a distinguished practitioner, for examples. And these sorts of interactions would help the director and the staff set educational research agendas they interact with the field.

We felt that the development of standing, ongoing, long-term peer review panels was absolutely critical. There are many structures of this sort that can work. The key is being able to recruit people to these panels, who are good peers, who see serving on a peer review committee as a service to the field.

And that perhaps the most important thing about peer review is that it facilitates the development of a scientific culture, and a community of scientists by providing feedback to the field through systematic reviews of research, by deliberate attempts to help develop young researchers through the peer review process. Peer review is not perfect. It is highly dependent on developing a stronger field, and it is tricky in education, because the field is bit more deflective.

The third design principle is to insulate the agency from political interference. Again, we feel that it's important to avoid micromanagement of decision-making, and to distort the agenda to be strictly on a short-term position; the use of the agency to promote certain political positions.

The director should have a fixed term that is independent of political terms. Setting priorities, and so on, should be done through an agenda committee, and through annual reports to Congress. There should absolutely be accountability for the research that is produced, to independent reviews by Congress and by the executive branch.

We think that budgetary discretion is particularly important. To use OERI as an example, the bulk of funding decisions right now are largely mandated, and there is very little free discretionary money that could be used to fertilize scientific research in education.

We think that to a certain extent the dissemination functions should be separated. And while there should be interactions between a research component and a dissemination component, the research component should be free to develop as much as possible.

The fourth design principle is to develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice. Short-term for example would be what works. What instructional practices would address a particular problem that is of concern to the country?

And what we might do for example, is to develop empirical syntheses of educational practices that might facilitate the development of a particular group of students that we are concerned about. But you can't develop educational research just on a short-term agenda. For example, the RAND reading study group recently wrote a report emphasizing the importance of a research agenda on reading comprehension.

A research agenda on reading comprehension cannot be short-term, because it is a complex process, where insufficient research has been done. And it can't be done with just a single span asking what's the best way to teach reading comprehension? You need to have multiple disciplines apparent. You need to have multiple strands of research, cognitive mechanisms, instructional research, assessment research, because we don't know how to measure it adequately.

Even things like neuroimaging and genetics would be relevant, and the real issue is how do you integrate across these different strands to develop a coherent model of reading comprehension? That is a long-term agenda.

And then you probably need to balance it, and have both short-term agendas, and long-term agendas. For example, if you think about English language learners, we could certainly synthesize research around what works, but a problem with our body of knowledge about how to teach English language learners to read is that we haven't had a long-term research agenda, and we tend to rely on the short-term to inform the long-term, which may misinform public policy.

We especially would emphasize the need to incorporate syntheses of bodies of research as a way of facilitating the accumulation of knowledge. And that's been done adequately in education.

The fifth design principle is not surprising. It's been said repeatedly, and it is to adequately fund the agency. As we noted in the report, about 1/10th of 1 percent of all educational spending is on research. The 1973 research budget for OERI was about $425 million. In 1991 it was about $100 million, and it increased only slightly in real dollars since then.

Good educational research, like research elsewhere is expensive. The examples that we gave, the Tennessee star study cost $10 million over four years. The National Institute of Child Health and Human Development has spent $100 million on reading research over the past 15 years. Even now there are several projects that are funded by the NICHD on reading research that have budgets between $1-2 million per year. And budgets of that size are necessary to deal with the complexities of educational research for different levels of analysis that you have to have.

The sixth design principle is to invest in the research infrastructure. The agency should have as a goal, a desire and a mission to help develop a strong -- what Ellen Lagemann described as a self-regulating community of scientists. It should emphasize the importance of peer review by requiring it for grants. It should support important components like data development, like an education statistics function. And then finally, it should be linked to practice and policy communities by the development of partnerships and interactions with a separate dissemination function.

Thank you.

DR. FEUER: Well, this was a very speedy tour of what is in this report. I am reminded a little bit of when Woody Allen said he took the Evelyn Wood Speed Reading Course and then was asked what "War and Peace" was about, and he could say was Russia.

I encourage you to read this report more slowly and carefully than we have been able to sort of hit you with, with these slides. To spur our thinking, we have done something which is really quite customary to the Academy, and that is to invite a group of folks who were not part of the original deliberations, who are not on the committee, but to provide some of their independent, thoughtful judgment about what this committee has done.

Those of you who are familiar with our review process will wonder whether it was not designed by that economist that Bob mentioned, who assumed that we can live forever. In this case, actually we got this report done pretty quickly. The people we have asked here today were yet three new people, who haven't been even part of the review process that went into this report. So this is another example of how much we want to engage the community in thinking and talking about this project.

We are very happy to introduce our panel of discussants, Rick Apling from the Congressional Research Service. As you know, CRS is an organization that has an interesting client. It's the United States Congress. And they have been in the business of providing objective and independent knowledge and advice to Congress for a long time. So I'm very much looking forward to hearing Rick's spin on how scientific inquiry in education can contribute to the policy formation and policy thinking.

Also on the panel of discussants, Pat Graham of Harvard University, a distinguished scholar of education, history, and research, and a leader in this field. And actually, quite recently the chair of one of our committees, for which I just want to say thank you again to her for her work with us on that.

And then finally, Dave Grissmer, from the RAND Corporation, a leading contributor in the research on education and related public policy fields.

So I think the batting order is in the way I have introduced you. And then we will look forward to a time for questions and answers. I'm happy to report we are now only six minutes late thanks to the great work of our panel on this side.

Agenda Item: Discussant - Richard Apling, Congressional Research Service

DR. APLING: Thanks, Mike.

I want to talk a little bit about CRS, and a little bit about how we work, because it colors my reaction to the report. And some of you may not be very familiar with our agency.

We are one of three congressional support agencies, but unlike the General Accounting Office, and the Congressional Budget Office our work is much more closely aligned with the legislative process, from the very inception of legislation, to the final conference committee agreement. So several of us have been spending the last six months, lots of time with the conference committee staff for HR-1, and we are glad to see that that's been signed, sealed, and delivered.

Our effectiveness and credibility really depends on our strict nonpartisianship, and this both with regard to party, but almost as important with regards to House versus Senate, and of course on the quality and the relevance of our analysis. I see my role as providing the best possible information to all members and staff and committee of Congress. And although sometimes it's a little unnerving, we really feel like we have done our best work when everybody is using our work to make their arguments on all sides of the policy question.

We almost never collect our own data. So we are very dependent on the work of others. So we have a vested interest in the work of this committee, and the various executive branches for the quality of educational research.

I want to do just kind of a brief case study of a project that we have been working on for the last couple of years just to give you an idea of how we do our work. And this has to do with something called the Rural Education Achievement Program, which actually has been signed into law as part of HR-1.

About three years ago the American Association of School Administrators floated a proposal basically calling for more flexibility and more funding for rural schools. This struck a chord with the Congress, and we were asked to look at that proposal. When we start to do this, one of the first things we often do is to try to assemble a data set from the various disparate sources of data that we have available.

And in this case we put together data from the Department of Agriculture, the NCES common core of data. Later on, poverty statistics also from NCES, with respect to Title I schools, and the Census Bureau with respect to population density.

Early on we raised several questions about the proposal. One being what is a rural school anyway? Is it dependent upon where it is located, whether it's rural? Or is it size? And if it's size, how do you measure size? What is small? And should poverty be part of the equation? And is the problem not enough money, or not enough flexibility?

Now some of these questions that we raised were considered, other were not really paid much attention to. And as the legislation moved through the process, as it often does, it comes down to who is going to get money, and how much money are they going to receive? So using data from other sources, we were able to look at the original AASA proposal, and we discovered that with some states, there would be lots of rural schools, and in other states, especially Southern states, there wouldn't be any.

This led other staff of members, especially from states from the South, to question the proposal, and we started looking at other alternatives. And one of those alternatives was to bring in a poverty factor into the legislation. Well, if you look at the final legislation, you will see that we now have two grant programs, one which is based on ruralness and size, and the other one is based on ruralness and poverty.

Now there were some questions that we weren't able to answer, and that's because of data limitations, and our inability to collect our own data. A key question had to do with how much money districts would get. And part of this had to do with the way the AASA proposal originally was crafted, and which made it into the legislation, which was that there was an offset from antecedent programs that would reduce the amount of the grants.

Well, as some of you know, it's very difficult, surprisingly, to be able to say how much money school districts receive from the federal government. They have requirements to do that in the General Education Provisions Act, and the Department does try to do this, but we don't have very up-to-date data. So we knew we were never able to say for that part of the formula, here is how much XYZ school district was going to receive.

Other questions either we never returned to, or we would never ask in the first place. For example, are the indices of ruralness accurate? And are rural schools really disadvantaged compared to other school districts in terms of their inability to compete for funds in the formula factors? And many even more complex questions of how this money would be spent. Would it be effective? Are there more effective ways of using these funds?

The point I'm trying to make here is that although these are important policy questions, our work is driven by the needs and concerns of Congress, rather than what the really important research or policy questions might be in the abstract. This was a quick overview of a project that we have been working off and on for I guess about three years. I hope it will help explain some of my reactions to the report.

I think the report does a very good job of describing the scientific method, and a particularly good job, in fact perhaps too good a job at laying out the difficulties in making educational research more scientific. Let me just point out a few of these points that I believe the report makes.

We have very few, if any real theories in education I think is one problem. The problem of replication, even if you have a well design random assignment experiment is very problematic. And I think most importantly, which the report acknowledges, there really is not currently an appetite for educational research among policymakers and the public in general.

And I also want to just mention a few things about the report that I think that I originally said were unrealistic, but I think maybe that's a little unkind. I think there are areas that need to be thought about very carefully, because they are very difficult issues. And they are kind of contrasts.

One is the desire for more federal support, but on the other hand, the desire to have less political influence on the process. Another is the recognition that research has to be tied to practice, but on the other hand, the notion that program improvement activities should be removed from the research entity.

And a third area that I think requires a great deal more thought is the notion on one hand -- and this is the way we always write our CRS reports, on the one hand, on the other hand -- there is a desire to hire very skilled people for this educational entity, but the report acknowledges that we really don't have a very good idea of what educational scholars should know and be able to do.

So this all in all, makes me a bit pessimistic about first of all, immediate congressional action. But more importantly, whether the Congress is going to be able to be the main solver of our problems here.

So to conclude, I want to paraphrase an article that was cited in the report, which I would like to read. I guess it's in press, by Carol Weiss, which was entitled, "What Do We Do Until the Random Assignment People Arrive?" My question is what do we do until high quality scientific education research is available?

And my suggestions are first of all, that most research, if not all, has to really be driven by the needs of the stakeholders. I think we pay a lot of lip service to this. I don't think we pay enough attention to it. And these stakeholders, as you well know, are teachers and principals, and in may case, national policymakers and analysts who support them. In the latter case, I think this is particularly true if you are expecting people to produce the money.

From my perspective, what I need now is more and better descriptive research. What are people doing? What are they spending money on? What works? I'm less interested in this point in causal and explanatory information, not so much that that wouldn't be great, but my feeling is that we are not at that point yet. We haven't done the descriptive spade work that we need to do.

And I case my analogy to the physical sciences, I don't see a Newton or a Maxwell or an Einstein on the horizon who is going to come in and give us a grant theory. I think we are more at the stage of alchemy. And I don't mean to be completely facetious about this. I think we have to discover oxygen before we have Mendeleev put together the Periodic Table, and certainly before Linus Pauling can tell us how quantum mechanics expands chemical reactions.

Meanwhile, Congress is going ahead and passing educational legislation, and we try to give the best information that we can. And again, what I need, and what my colleagues are good data that we basically can reanalyze, and make relevant to the policy questions in a very rapidly moving policy environment.

Agenda Item: Discussant - Patricia Graham, Harvard University

DR. GRAHAM: Thank you. That's very helpful to me, as were the earlier remarks.

I want to say that I read this report carefully, and with admiration. I want to go on to raise two questions about it, but I want to start by saying that I think that for those people who have the energy and the inclination, this is a report well worth reading in its detail.

It stresses important questions in education that need investigation and clarification. It points out that not all research in the past has been as good as it should have been in education. It points out a limited constituency for good educational research. Who really wants good educational research? And who wants to insure quality control?

There are many pressures in the field of education that make quality control very difficult. For those of you who have never had the opportunity to attend a meeting of the American Educational Research Association, which is a great, big outfit, and when you are asked why is there not more selectivity in the program about having a limited number of really fine talks, the answer is, well, departments won't pay unless you are on the program, so we want everybody to be on the program so they can get their way paid to the AERA. That is not a good means of quality control.

And this report, if carefully read, I believe would make some real progress on this. I think to me, one of the most important elements of this report is its emphasis on a culture of rigorous inquiry. Indeed, it often uses the phrase, "scientific culture." Well, there's good science and there is bad science. I'm not just interested in a culture of science. I'm interested in a culture of good science. So I think the emphasis in the report on a culture of rigorous inquiry being essential for good education research is absolutely vital.

It also has some great things to say about enabling conditions for good educational research -- time, money, and support. I think it makes another excellent point when it talks about the fact that educational research has been hampered by not wanting to know, or being unwilling to investigate certain questions.

You think for example, about how hard it was to get NAEP established, the National Assessment of Educational Progress. NAEP could not be established within the federal government. The initial funding for the exploration of NAEP came from the Carnegie Corporation. And school leaders in America, known as school superintendents, chief state school officers of various states prohibited the data from NAEP initially being released by state, only by region.

This was an example of not wanting to know how American children were learning in their schools. And this has been a profound hindrance to some educational inquiry. We have been quite reluctant historically in releasing academic achievement data either by race or by income levels, which we have begun to do in the last decade or two.

And finally, we have not been enthusiastic about engaging in widespread research on the relative significance or insignificance of schooling in child's overall educational achievement. We have found it very difficult to talk about in the past, about how important schooling was, as opposed to how important other kinds of factors are. So I think what we have avoided has been a problem.

There have also been successes of educational research. I take the long view. I have made my living teaching the history of education, so I'm looking at the past on a regular basis. And if you look at the kind of educational research, and the stark experimentalism that characterized the field of Edward L. Thorndike from 1925 up until the 1970s, in which we thought that we could learn about learning by what college sophomores did, which was the favorite unit of analysis, because this group could be easily studies.

We paid very little attention to context, to community, or to issues of shared purpose. One of the great contributions of a fine quantitative methodology, Anthony Bryk at the University of Chicago, has been, and it's cited in the report, is to stress the significance of shared purpose in influence in what happens in Catholic schools. But the issue of context, the issue of community we now see must be considered, as well as these types of experimental designs.

When you think about the first test that educational psychologists put together, the Army Alpha, the first large group scale test for World War I, and if you haven't taken the Army Alpha, I really encourage you to do so, because it is a tremendous example of what enormous contributions have been made in test methodology since then.

There is one scale on the Army Alpha which purported to be a culture-free test, which was which of the following makes soups? Well, the answer was Hardshaftner and Marx. My father comes from and Danish immigrant family. My father took the Army Alpha. The Army Alpha was supposed to separate the people who were going into the military from the people who were going to be cannon fodder as enlisted men as opposed to officers.

Now my father is Dane. How would he know that Hardshaftner and Marx made soups? And as we look at the sophistication that has come in testing with the intervening 75 years, I think it's a real tribute to educational research.

I also think that the famous progressive educators in the United States in the midyears of this century who said to us that curriculum doesn't make a difference. Children should just study whatever they like. This is a parody of the worst form of progressive education, part of which was widespread, however.

It made a real mistake, as we would say now, in not thinking that curriculum is important. What we need to know and be able to do. And it is educational research of the last decade that, complete with the standards movement, has made us believe more about this.

So there are some very good things in this report. I have three question about it, three queries. The first is the audience for the report? The second is what's the point of educational research? What is educational research for? And the third deals with OERI.

The first question, who is this report for? We heard a little bit about who this report was for. My question is, does the audience that is the group that is intended to read this report, does that make a difference in what this report says? Should it? And I think the one point that I take away from this question about audience, which I think is important for this report, is that this report needs to be read carefully.

You need to read footnote 4 on page 94 that says history and philosophy are also relevant to understanding education. You need to read on page 52, "science does not necessarily mean good."

Because if you do not read this report carefully, and if you only read it with executive summaries, as some of us have sometimes done in reading reports, you will get a simplistic notion that the report does not, in its complexity state that what we have here is the paradigm for the hard sciences that need to be applied to education. And that it is just scientific, and then it would be all fine. That's not what the report says. But a cursory reading would leave you that impression.

The word science in English has a connotation that it does not have in Washington(?), naruka(?), or German, vichensha(?), which in those cases means scholarly inquire of a rigorous sort. I think what this report is calling for is that kind of rigorous, replicable, scholarly inquiry. The word science in English makes you think of cell biologists and physicists working in a laboratory. They do wonderful work. They have excellent principles. They are outlined here. But this subject is more complicated.

Secondly, what is the purpose of educational research? What is educational research for? Why should we do it? Is it to improve education through knowledge? Or is to improve educational research itself? And I would argue that there are many references here to the disciplines, particularly physics and biology. And research in those fields in intended to help us learn more about those fields.

But many would not characterize education as a discipline, but rather as an applied human activity. And therefore, what is the best way to characterize research about it? Whose interests are served by education research? Is the purpose to increase our research knowledge, or is it to find ways of improving education research?

Agriculture and medicine can be improved, and are improved by research. The argument is that they have been improved more by research than education has. And that education ought to be more like those. Well, I would just like to observe that there is an economic infrastructure in both agriculture and medicine that does not exist in education, that is served through research -- fertilizer companies, pharmaceutical companies, for example.

And that there is an economic incentive for improvement. Doctors have recently discovered productivity. Farmers heard about it some time ago. That is also served by research. Education, except for book publishers, hasn't seen much of that.

And finally, the most significant examples of successful research in either agriculture or medicine have to do with dealing with something which is external to the human. That is changing cow production so that they have a higher butter fat content in their milk. It took 50 years for the University of Wisconsin to figure that out. Changing tomatoes' ripening characteristics so that you could send them on a long trip without their getting mushy.

Major research in medicine has allowed physicians and surgeons much better tools with which to operate. Anesthetists have better skills in putting people to sleep so this can go on. Finding medications that control widespread ailments such as diabetes, hypertension, and coronary disease.

When medicine has turned to changing the way people live their lives -- lose weight, don't drink so much, don't use drugs, don't engage in illegal, unsafe, or socially disapproved sexual activities -- medicine has had much more difficulty changing individual behaviors than it has done in changing the techniques that allow us to learn.

Smoking is perhaps the best example. Smoking in the last 30 years -- more people smoke a lot less than they did. But in point fact it is most prevalent among the most affluent in America, not among the poor. The analogy to education is that the problems that we face that most of us are particularly deeply concerned about are the educational problems of the poor.

Bob Boruch mentioned, I think very effectively, the section of the report that talks about opportunities for evaluation studies. And I just would like to say that I agree with everything he said about that. And I, for example, favor education engaging much more actively than it has in the past in efforts at randomized trials, where selection can be done according to the rules which he outlined, and the report outlines.

But I would also like to encourage Bob and his colleagues to push a little on answering that last question that they talked about, about why and how these changes have taken place, to which Bob alluded already. It's what I call the whether-why question. Policy people want to know whether a program works. Practitioners need to know why it works, so that it can be replicated, and knowing what the essentials of replication are.

That's a much harder question to answer, and often our colleagues in the research community are unenthusiastic about pursuing that, because the research design for answering the why question is not immediately apparent. But it seems to me that if it's essential for those of us who think of ourselves as researchers, and well as for the population with whom we work to be able to address that question.

Let me now turn finally to OERI. What are the implications of this report for OERI, the federal government's only agency solely committed to educational research and improvement? Well, poor old OERI. The tragedy of the federal government's educational research effort in the last half this century is that unlike its experience with research in health through the National Institutes of Health, or in science through the National Science Foundation, the Department of Defense, and several other federal agencies, where much of the nation's finest research in those areas has been federally funded, that has not typically been the case in education.

There certainly has been some fine research funded by the old US Office of Education, by NIE, and by OERI. But the dominant educational research funders in the US I believe, have been the major foundations and the universities, the latter through their support of faculty and graduate students who are expected to do research. Now why is that so?

I think the first is that there has been an absence of a research culture in the US Office of Education, which had dominated the culture of the Department of Education when it was created in 1980. The USDOE was a place that dispensed funds according to congressional mandates. It was not a place that valued research, nor were the school systems in the United States dying for research.

The limited success of the National Institute of Education from 1972-1980 was due in part I think, to its presence within the Department of Health, Education, and Welfare, which did have a strong research culture in its institutes of health, which was reflected to some significant degree in the senior management of agencies.

The only two times that I am aware of when the budget for the National Institute of Education went up significantly, 25 percent the first year of the Carter administration, and 10 percent the following year was when it was in HEW, where there was a strong tradition of research. So I think it would be inaccurate to say there has been a steady drop in decline for educational research funding, but the one time that it went up, it was when there was a relation to the strong research culture.

And I want to say something about the issue of political support and political interference. I'm absolutely in agreement that agencies need to avoid, if they can, political interference, but I am absolutely in disagreement with the idea that you do this through insulation. Rather, you do it through long, close relationships with your colleagues on the Hill, and in the administrative branch.

And what has been most striking has been the inability of educational researchers to figure out how to get along with congressional staff, with members of Congress, and with OMB and the White House. Figuring out how to build those close ties is absolutely essential, so that you are trusted when you say to a congressional staffer or a to a member or to somebody in OMB, this is the way it is. This is what we have to do, and we need your support for it. That's based on close ties, it is not based on stiffing them.

The third, as the report says, we must hire good people to work in the federal research agencies. And there are ways to do that. And we have had good people. But the creation of the Department and followed by the Reagan administration's efforts to eliminate the Department of Education sent people out of the educational research agency who are among the most distinguished educational researchers in universities, in the policy community, and in the world of practice.

I mean I've got a list here, which I will not read at the moment. But it is an absolutely distinguished group of people who used to work at the education research agency, who spread throughout the government, back into academe, and into policy positions.

The final point I want to make is that the biggest problem of OERI is that it doesn't have any money. And this has been alluded to by several speakers, but I just want to go over this one more time. The assistant secretary in the first Bush administration reported a 90 percent decline of federal funding for educational research between 1975 and 1995.

The National Research Council study in 1992, found an 82 percent decline between 1973 and 1989. Another NAS report in constant 1990 dollars found the federal educational expenditures of research and development had gone from 1972 at $1.1 billion, to $300,000 in 1991. That is an incredible drop.

Field initiated studies, OERI does 2 percent, NIH does 56 percent, NSF does 94 percent. If you believe people out there are smart, and you don't fund their ideas, you don't advance education. In terms of basic research, and I am a strong believer in basic research. Nothing is more practical than a good idea. NSF does 94 percent, NIH does 60 percent of basic research, OERI, 5.5 percent.

Federal educational research for research and development, only half of 1 percent of educational spending is committed to research for education. It's three times that for agriculture, 21 times that for space research, 30 times that for health.

The congressman who stimulated this report said that there is a problem with educational research, it is broken. The problem with educational research is not that it is broken. It is that educational research is broke.

Thank you.

Agenda Item: Discussant - David Grissmer, RAND

DR. GRISSMER: Certainly, some of my comments will echo what Rich and Pat have said. First, I would like to congratulate the Department of Education for funding a study like this. It's always risky to have an expert panel look at what you are doing. I have found the Department of Education to be very open to the kinds of advice they get.

Secondly, I would also like to congratulate the National Academy for taking on a study like this. They get handed some of the toughest issues in scientific issues in our society. But this one I think had an absolute peculiar twist. That is, most National Academy studies tend to assess a fairly narrow area of research, go through and assess the evidence, come out with some balanced conclusions.

It's quite a different animal to try to assess whether a whole body of research is scientific or not. And that's much more difficult terrain, especially with the time period they had. I heard the word epistomological earlier. I haven't heard that since my Jesuit educational days. But since we don't have an NAP, a National Academy of Philosophy, I'm glad the National Academy of Sciences took on this task. It was a difficult one.

I think it's important to reiterate what Lisa said what the charge was. They did not take on the task of evaluating the quality of research, of existing researchers, or the existing federal agencies. What they did, what the charge was, review and synthesize the literature on the science and practice of education research and education, and consider how to support high quality science in a federal agency.

So they have described what scientific inquiry is, and the principles for doing scientific research. They have identified distinguishing features of educational research that might make it difficult or different to carry out than in other areas. They suggested research designs to make research scientific, and the design principles for federal agencies.

As such, I think this should be seen as more of a how-to manual, rather than a comparative analysis of what is actually being done with respect to what is being done, which is a different, and I think probably a future study.

I found much to like in the report. Their description of how to do scientific research of always making it sound like a recipe, but captures much of the complexity and uncertainty involved in scientific discovery. Science is perhaps the best way we have found to develop widespread consensus about theories which explain observational data. But it's often circuitous. It's indirect. It's nonlinear. And it is also a very human activity.

The goal of the construct of theories that explain ever expanding and more complex sets of observations is only accomplished in the community of scholars that involves open discussions, strong disagreements, politics, unpleasant personalities. In short, everything that occurs in any community with people. And I think more and more we are beginning to capture that part of science. And it's not a process, it is much more complex, and they have really captured that well.

In describing the practice and environment for educational research, I think the report makes several good points, although I will later disagree with some of these. The first point I think is very important. Educational research is different, because if affects people's lives, and challenges people's beliefs.

In other words, educational researchers could be doing perfect and outstanding scientific research that produces reliable results, but a cloud and fog of controversy from inside and outside the education field will always be present that makes it appear like the education research is inconclusive.

One of the more difficult things, and when I started education research I had to address is what I call this fog of advocacy which surrounds education. It will always make it more difficult not only to achieve consensus, but to make the consensus knowing within the community and outside. And I think one of the major points that we need to discuss later is the consensus panels. And how we know when we achieve consensus. We can have consensus, and not know it very easily in education, whereas, that's not true in a lot of the physical sciences.

Second, the authors state the physical and social sciences have a similar set of principles that guide the inquiry. But they sort of expect the same degree and pace of success. I think that is well said. There are many reasons for this, but primarily physical science continually developed technologies that allows them to observe and manipulate every possible object of study at ever increasing depth and reliability. And the objects do not change over time and space.

What technology allows is not only observation, but extremely high signal to noise ratios. No two children or teachers are the same. They change over time. The source of their behavior and learning takes place within the least explored, and most complex of human organs, the brain. We are still looking through a glass darkly.

So in the short run we are going to make less progress, achieve less consensus, and have more uncertainty than is found in the physical sciences. An argument can be made the brain research will eventually provide the kinds of reliable and producible data that will change all of this, but that still remains to be seen.

I also like the conclusion that the distinction between qualitative and quantitative research is overdone. I agree to much less extent about the distinction between basic and applied research may be unnecessary. One reason the qualitative research can be important is that it basically can improve quantitative research.

Quantitative educational research I think is often characterized by relatively simplistic models of development and behavior. Teachers, social workers, therapists have exceedingly more complex models of what goes on in classrooms, families, and students than do researchers. And qualitative observation should bring new hypotheses and variables to bear in helping quantitative research capture more of this complexity.

I do think a distinction between basic and applied research is absolutely necessary to maintain. While it is true that pursuing applied research can certainly make progress in basic research, basic research is needed because funding is hard to justify because of the basic.

The history of science tells us that one cannot predict where breakthroughs are going to occur, but it also provides confidence that supporting a wide array of basic research is the surest way to insure progress. So I think they make some comment that basic and applied may have outlived its usefulness. I don't think that is the case.

The section on the design of educational research distinguishes between experimental and quasi-experimental. The pitfalls of establishing causal relationships, as well as establishing reliable, generalizable, and reproducible relationships. It points out that selection effects and contextual effects are particularly difficult. I would add the difficulty of separating family, school, and community effects, and substitution effects between families and schools. The design principles for a scientific agency are hard to argue with.

Let me talk now about some of what I would like to challenge in the report, but also make some observations that primarily lie outside the scope of the report. The question we are ultimately trying to answer is why is practice not making more progress? While the report did not directly address this, my reading between the lines is that one answer would be slightly apologetic -- perhaps too harsh a term -- but it would go something like this.

Well, it's harder to make progress for a number of reasons that we're different from the physical sciences. So we shouldn't have the same expectation. Progress can be slow in any science. There are long periods of nonproductive research, often punctuated by great breakthroughs, a lot of dry holes drilled. So lower your expectations.

Emphasize we have made progress in some areas. I think there are examples of assessment, and phonetics and reading are particularly apt. I'll argue later with the example that we made progress in making the effects of resources on education.

And then the final thing is funding has been a problem. So there is sort of an underlying term in this that says we have sort of got excuses for why we haven't made progress. I'm sure that's too harsh a judgment. But let me read a couple of quotes from the report that I think tend to support this.

One is that "Studies that do not start with clear conceptual frameworks and hypotheses may still be scientific, although they are obviously at a more rudimentary level, and would generally require follow-on studies to contribute significantly to scientific knowledge."

Another point is, "In sum, it is clear that there are no bright lines that distinguish science and nonscience, or high quality from low quality research." And I think even the authors felt a little just tainted in the report, because they say at one point, "The broad view taken in this report should not be interpreted to suggest anything goes."

So in writing the report, there was sort of this tension between justifying what we are doing, and yet criticizing it. I think that for my taste, they came down too much on the sort of anything goes kind of realm. Two other examples I think illustrate this. One is that there are two kinds of research, experimental and quasi-experiment. And there is a third, which is called non-experimental.

And the differences in my mind, the quasi-experimental has some prethought as to choosing the control group, but 90 percent of research in education is non-experimental, which is nothing has been done beforehand, and it uses a set of data, and try to figure it out. And I think that distinction is important in terms of understanding what is done.

Secondly, they give an example of an example where progress is being made in the area of the effects of resources on education. Let me paraphrase very simplistically, but I think it goes something like this. We used to think resources mattered. We went through a period when we thought resources didn't matter. And we now we think resources matter, but only if they are applied in certain ways.

I think over 25 years, that's not a whole lot of progress. And the sad thing about this is that we have four literature reviews, three of which use the same set of steps, two come in one direction, one in another. The fourth literature review rejects two-thirds of the articles in the other three, because they don't meet the quality criteria.

The heart of the issue here is not whether the meaning is plus or minus on effects. That's why we have such large variants, and why we can't do a leap with the simple results. A research effort needs to go into answering that question. Why can't we get reproducible results? That's a different kind of research than just picking up a database. And I think we are going to have an opportunity to do that with new state data, where we have similar specifications on lots of different data sets to determine if we have comparable results. So I think that was a particularly bad example to use for progress in education.

Let me wander into some territory not covered by the report, and I think both Pat and Richard have done, and will make some opinions about why we aren't making more progress. Richard mentioned reproducible results. And I think if there is one basis of science that we need, it is that we obtain reproducible results. We cannot have good science without understanding why we are not getting reproducible results.

I think the second area that I think was also mentioned is there is really not a tradition in educational research or social research that theorizes. In hard science you have theorists and experimentalists. And there is a good reason for that. Without good theories, you do not have a scaffolding on which to accumulate knowledge, and it doesn't matter what the next question you ask is.

So that we need a lot more effort to discover and frame theories before we go into research in education. There is an example of this in the class size example that I provide. But I think it's the area that we really suffer, in that our research is not guided by theories, and is not enough tension to develop theories of why things happen.

I think probably it's important that there is no sure tool of knowledge researchers have. Education is as affected as much by what goes on before school, outside of school, as it is in school.

The federal government spends about $3 billion a year on research and development directed toward areas of children or youth. Education research is roughly less than 25 percent of that. I don't think we are going to be able to make great progress in education research unless we are able to solve similar problems which exist in the entire body of research.

And in order to understand education, you have to understand a much wider body of research and human learning than currently is thought. And one approach to this is a development science curriculum that would basically focus on providing the basic knowledge that people doing research on children need -- genetics, statistics, parts of economics, parts of sociology. But there is a base of knowledge developing which researchers in the field should all have. And I think the training issues in this area are fairly severe, and they need some reworking.

The absence of consensus panels in education really makes it tough to even determine whether we have consensus or not. We really need consensus panels. Very simple things, like what is the best measure of educational expenditures over time? Have scores gone up or down? There is the group of experts who have said this is what it shows, and this is where we disagree.

I'm so tired of seeing this NAEP chart that is 0-500 points and it looks flat. Well, I can make anything look flat if I can sort of address the scale. We need consensus panels so as there beings to develop a body of agreement about what we know and what we don't know.

Certainly, experimentation, but experimentation needs to be strategically framed they think. In the long run, we are going to need to depend on non-experimental analysis, because we don't have enough time and resources to experiment with everything. So experimentation should be geared toward understanding what assumptions we need to make in non-experimental analysis.

I think the Tennessee class size experiment begins down that road. It really tells a lot about how to specify non-experimental models. So that we need a body of research that is geared toward testing the assumptions in non-experimental models so eventually experiments agree with non-experimental models. That's the sort of holy grail we are looking for here. But that takes some real strategic planning to carry out.

We certainly need a serious examination of the major data collections, a look at the research centers. All these things should be on the agenda. One of the interesting things that I learned at RAND when I was doing military research is that generals need to be at the table when the research agenda is devised. You get a different research agenda with the generals at the table.

It focuses your research on the most important questions. And the education equivalent I think is having research centers connected with the chief state school officers, the Council of Great City Schools, registering with groups. But it really needs to be more of a defining of the key questions of research centers, that have some accountability with the practitioners.

And finally, this need for a long-term strategy of research is really absolutely needed. That is, the year-to-year kind of things that we do, you can't depend on purely viewing things to sort of keep things moving. There really does need to be a broader strategic plan to help in guiding this research.

Thank you.

DR. FEUER: Thank you to the presenters and to the discussants. I feel like I'm a little bit torn here. There is some momentum here that I'm wondering if we should just sort of capitalize on, or should do what we promised and give you a break? Should we do this in the democratic style, or should we just decide from up here? What do you all think?

Hold all of these thoughts, please, and come back in 15 minutes sharp for questions, answers, and discussion.

[Brief recess.]

Agenda Item: Q and A

DR. FEUER: Okay, what we really want to do here now is hear from you. We have had a chance to look at some of this report, and see some excerpts. Someday I think we will have this on a bumper sticker, but so far we have worked it down from the report to the slides to the presentation. And we think we have given you some sense of what this is about, and we really do want to get into a conversation.

Any first takers?

DR. ACOSTA: Good morning to everyone. My name is Adele Acosta, and I am the principal of a little school in Hyattsville, Maryland. A very poor school, with high mobility and a large Latino community, an African-American community. Our children are illiterate in Spanish and in English. So we have the challenge of performance.

I just took some notes. I am just happy to tell you that as a practitioner, the comments of the discussants were great. Again, I believe that the issue of who is your audience is a very important question. One of the things as a practitioner that I need to know is what works. Can I get it? Does it work? And can we afford it?

Practitioners unfortunately jump too fast in education all the time. And we need something that is solid and reliable, and that is applicable in today, not tomorrow. Because the accountability issue is part of the reform effort is one that weighs heavily on us as well.

If you need partners, you need to include more practitioners in the research community. I am very lucky to be among this august body, but I would like to see more going on with more review, so that we can shoulder the bearing of responsibility and accountability, and really address the issues of education for our nation's children.

And I would like to call for culturally balanced inquiry or purpose, which is what one of our discussants talked to. And just one last thing -- actually two. One, I loved the report, because I understood it. Do you want to whet the appetite of practitioners? Develop reports for us that we can understand. And I know that sounds silly, but it is very intimidating to pick up this weighty stuff.

But if it's too complicated, and we have kids that are banging at our doors that need to be taught, and the parents to deal with, and the superintendents, and we have a whole cadre of people, the culture of education to deal with. So we cannot feel intimidated. We feel less intimidated when a report like yours, which is so well written, so articulate, and so understandable that it was a pleasure -- I don't feel like I was wading through it.

I would suggest that one of us take the task of sharing this work with the board of supervisors, with superintendents, because that is where the leadership is going to come. If principals and educators are going to take educational research, take the base of your work and apply it, it has to go from top, down. We know that.

Principals and superintendents, they go, research, please. Get that away. That's do so and so. Things are not going to happen in the classroom. So the stakeholder's table has to get broadened as well. And I would hope that at some point someone would stand and address those people that make decisions that affect not only the school house, but for the educational research community as well.

Thank you.

DR. FEUER: Thank you. Motion to adjourn? No.

Well, does anybody on the panel or among the discussants care to respond to that, or shall we just go on?

DR. DE HAAN: Well, first of all, thank you very much for those comments. For any author or partial author of any document to have a reader say that it is readable and understandable warms the cockles of our respective hearts.

But me also ask as a newcomer to the NRC, don't we have dissemination programs to do exactly what was just asked? That is to get this out into school districts and other organizations?

MS. TOWNE: I'll just make one comment on that. The National Educational Research Policy and Priorities Board, our sponsor, was wise enough to give us a little bit more money after the report was done, we are happy to report, to actually try and do those sorts of things, to be very strategic and think hard about who the audiences are, how we might have to repackage this, or talk with people in different forums to pull out messages and ideas that might useful for different groups.

So it is something that we are staring to work on, starting today. So any great ideas, I would be more than happy to hear about.

DR. FEUER: Miron, with your indulgence, I want to make sure I get the people who aren't on the NRC staff first, and then we will take you.

DR. WEINGARTNER: Herb Weingartner from NIH.

This was really a pleasure, sort of reading through the report, and listening to some of the comments. And I had a bit of deja vu experience, and I wanted to share a piece of this to put in context what I think is happening. We have very much the same kinds of problems in the divide between clinical work and basic work in the life sciences. And the struggle to change culture, which is really what you are about, is an ominous thing to do. It's very, very hard.

So when I listen to comments like is it broken versus broke, the two are linked. If you have something that is broken, you don't put a lot of money into making it continue to be broken. And NIH goes through this struggle all the time.

We have programs for example in prevention which are really struggling. We are not making very much progress in that area of research. We are reluctant then to put a lot of dollars into it. And unless you can figure out some way of nurturing that culture in terms of making it more really based on science, as opposed to pseudo-science, the money isn't going to go there.

The issue of translating from basic science into clinical work is part of the struggle that you have. And also in trying to convince folks. I was listening to the comment that was made, making something understandable to teachers, to principals, to administrators. That's a very tough thing to do, to make a compelling argument that this is going to have some payoff.

So my only comment really is this. That the struggles that you have are very, very similar and parallel to the struggles that other folks have in other areas such as the distinction between clinical and basic research. And how do you improve the climate of clinical research in for example medicine? And that was one of the things that Harold Varmus, before he left NIH, struggled with, and it's still a struggle that we have.

So it may be useful to also then look at what are the kinds of solutions or partial solutions that people have struggled with, and perhaps tried that make that possible, as maybe it has been played out at NSF or at NIH, and make use of that in thinking through the logistics of how you implement some of the things that you would like to see out there.

DR. GRISSMER: A couple of comments on that. I think the gap between the clinical and research community is a very serious issue, not only in education, but elsewhere. And I know in medicine, I think one of the things that helps this is the central structure of a teaching hospital, that brings together people being trained, the clients that you are serving, and research. It at least makes an effort to sort of develop that mix of clinical work, research, and training.

There is virtually no education laboratory that does that. We don't have an education laboratory devoted around an experimental school or a community. The research tends to be very distant from the place where the education occurs, which doesn't help at all.

It is also a cultural issue, as you indicated. The lady that spoke earlier, in her head she's got a much better model of what happens in a school, and what works and what doesn't work than any research we know about. She mentioned top, down for research down. There needs to be a lot more floor, up in terms of understanding what is in social work, therapists. They are out there daily in the nitty-gritty of things.

I have a favorite story on this. My daughter is a fourth grade teacher. And so in my family, I'm the one that knows nothing about education. But she was talking about class size, and how much difference would two less students -- she has 24 or 22, not a lot. And she said, but I could choose the two students. What that indicates is we had the wrong variable. It's the burden on the teacher, not the number of students that is the relevant variable.

There are all kinds of insights like that. The research community, sometimes I think we are even not interested in it. But there needs to be more floor, up.

DR. FEUER: Valerie?

DR. REYNA: Hello, I'm Valerie Reyna from OERI, which has been mentioned a few times today.

But I want to thank you for this report. It is extremely timely as far as we are concerned. And I want to state something that I think many of the many of the people in this room probably know, but is worth restating, and that is the leadership at OERI is committed to the use of scientific evidence in education. And we are committed especially to the use of high quality scientific evidence in education. And this report will be tremendously useful as a means to achieve, and I want to thank you all for your hard work.

If you look at our Web site, we have some new initiatives that not coincidentally embody some of the principles contained in this report. And I would be delighted to get your feedback on that.

So I think we are moving in the same direction. I would add a little quibble to some of the remarks that have been made, however, in an otherwise extremely useful report. And that is the notion that non-experimental techniques are required to study issues of complexity, issues of underlying process, or issues of culture, and so on.

This assumption is a common one, and I am not saying that for example, qualitative or ethnographic techniques are not useful in studying such things. But experimental techniques of the sort that Thorndike used in fact can be applied to these sorts of issues. And systematic scientific experiments can be applied to the study of culture, to complex systems, and so on, and can be very fruitful if the research is hypothesis-driven.

I mean one of the messages that I continue to try to send wherever I go speak is that the scientific method, for the most part, applied to education, is very, very similar. There are other complex systems that are studied scientifically. So that would be the only footnote I would add to the report.

Probably I have some insight into that, because as a psychologist, the notion of introspection and self-report has a long history in psychology. And what we found as a result of research is that self-report about the nature of underlying process is not always reliable. And that has been shown in the scientific research that has been done over many, many decades.

So it is not a direct pipeline to the mind, or to causality to simply ask people to introspect about what the nature of what they are doing is. And that is not to say that that's not valuable data, and that in fact the study of consciousness can be studied scientifically.

But thank you very much for this report, and I can't wait to see the transcript and be able to put it into reality as soon as possible.

DR. KELLY: Hello, my name is Anthony Kelly. I'm at George Mason University. I was for three years, a program officer at the National Science Foundation in the educational research field. I see my current life's goal is to make sure that Michael Feuer has enough work to do.

DR. FEUER: Thank you. I want to just see it now.

DR. KELLY: I want to comment on the assumption behind the report, because it seems to me that the National Academy only gets involved when there is a problem, like global warming for example. But the question wouldn't be asked unless there was a deficit behind it, what's wrong with educational research, and can we make it more scientific? Meaning, can we fix it by making it more scientific?

And it's a fine question, and it would be interesting to see how it gets played out in regional programs at OERI. Whether or not making it more scientific by one definition of science is the way to go.

One nice thing to add might be some history of the science perspectives. For example, Gerald Holton's work at Harvard, who talks about science being as much a retrospective process as a prospective process. So the processes that are going into making scientific discovery is often quite heuristic, fallible, based on luck.

And then after the fact, the scientific report is written not as if everything was inevitable, when it isn't always the case that it is inevitable. And it's very hard to be inevitable before you are going. That is to say, basic research is what you do when you don't know what you are doing.

And the randomized experimental trial is a very appropriate tool akin to sinking a well when looking for oil. So you sink the well. Is there oil or not? But it's not a good exploratory tool, because it is very expensive to sink a well. So what you have got to do is a lot of descriptive work, where you go out and look at the terrain, and decide is it even worth -- there aren't any oil wells in Ireland where I come from. They don't sink them, because there is no oil there in the first place. So it's better to sink your well in Saudi Arabia.

So you get a better idea of whether or not you should put the money into a randomized, once you've got a good idea that the variables that you would like to assess really are relevant. Also, it would be interesting to see if this plays out, if we do get more scientific research in education by this definition, whether or not it will make any difference. We will end up wishing for what we shouldn't wish for.

I want to follow the comment from the teacher that stood up. If it's a thing that policymakers, administrators, students, parents have this available to them, strong scientific trials in that sense, would it change their perspective on what they do today? Will it be realistic in any sense that they can use it on Monday morning, or will they have to overqualify to the point where they won't find it useful?

That's a set of interesting questions, in addition to is educational research sufficiently scientific? Is scientific research sufficiently educational? We don't put in the time to get the consensus, and then in turn what the findings are, or even what it means to do educational research.

If you look at the product of science, it's not just propositional statements about the right questions with the right warrant. A lot of progress in science itself has been made in coming up with not just propositional findings, but new theories, new methods, new tools, new instruments, and growing new researchers, and bringing new researchers in from different fields. Each one of those is just as important as making sure the propositions at the end of the day will fulfill a number of criteria.

Other questions, certainly as a program officer, we often wondered whether educational research was sufficiently economical. So you could get an effect if you put a tremendous amount of money into the very well supported faculty doing the research. But would that carry on once you went to the reality of education?

I currently have a project from the National Science Foundation looking at design experimentation, which got some mention in the report. And it wouldn't be there so much as educational research that is sufficiently scientific, as educational research sufficiently creative, because you can ask what works, but that assumes a certain condition that you are happy to live with, and within that infrastructure, what works?

But if you look at the work that has been done in science that is new and creative and interesting, complexity of theory, chaos theory, the new applications of adaptive optics in astronomy that are leading to discoveries of new planets. The idea about how you get that into the curriculum is a very different question than works. It's a question of what could work, and what should work, and what might work in new and interesting ways, and ways we don't understand.

And again, the tools for that will be maybe at the end of the day, will involve randomized trials. But in the way there is going to be much more like the day-to-day work of science in the discovery phase, as in the verification phase. So it would be Michael to look at other projects like that, is educational research sufficiently other things?

Finally, it would seem that there is sort of an academic kin to the report, is educational research scientific enough, as if it were an academic interest to the scientists on the panel. And I know it's probably more than that. But insofar as the success of science as an enterprise, it involves an educated citizenry, and new scientists, and politicians who understand science. An equally powerful question is, is science research sufficiently educational?

Thank you.

DR. GARCIA: Gill Garcia, the US Department of Education.

Again, I would like to reiterate the very positive comments that have been made about the report and the panel. And I would like to build a little bit more on what Ms. Acosta, some of the comments that she was making at the beginning of this question and answer session.

She started her comments by making reference to her school, and the fact that her school had large populations of Latinos and African Americans, and that in fact is the gist of my comment.

I think the principles lend themselves to any number of different very positive and very powerful applications. But one of the tensions that I always face as a manager of research in the US Department of Education is working with the researchers and grantees and contractors that we fund over time, to insure that ultimately there is a balance between a very, very careful, very comprehensive, very explicit understanding of the subjects of the research, whether it be youth and children, and for schools, or community settings, or whatever the subjects are that being inquired, and the results.

What I find lacking more times than not are very sketchy profiles of the youth and children that are being studied in any number of different research studies. So that for example, this is not a criticism of what Ms. Acosta was saying, but it's just a departure from the very point she is making. If a research report makes reference to Latinos, for me, as a research manager, it's not sufficient to know that in fact the subjects were Latino. I want to know a lot more than that.

If the research report is referencing English language learners, I want to know very explicitly what are the key features, what are the explicit and unique features of this subsample, this subgroup of English language learners that were the subject of this report.

If the research report tells me that there are certain assertions to be made about low income schools or low performing schools, that is one very superficial indicator and guidance point for me to follow to the end. But I want to know a lot more about a low performance school in Washington, DC, and its unique characteristics, versus a low performing school in Arlington County, and its unique characteristics, so that I can in fact ultimately be able to attach the assertions, the comments, the findings, or the recommendations that are being made to a particular subgroup of kids or children.

So I would encourage the panel, as well as the Academy, because I know that it is in fact a challenge that some of us in the US Department of Education have accepted, to continue to force the issue beyond principles, beyond research methodology discussions, beyond data analysis discussions to a very fundamental and practical question, who is it that this research is talking about, so that we can fully understand how to make use of these results.

Thank you.

MS. TOWNE: I just wanted to respond with one point, just as a way to point to a part of the report that we talk about some of these issues, and that we didn't actually have a chance to talk about in this discussion. It is Chapter 4. But Chapter 4 is the place where we talk specifically about education, and the ways in which the principles that apply broadly, play out in education, and how that application is manifest.

And in doing so, talking about some of the very strong and very influential contextual factors in school settings for example, that for example have very strong implications for replication, which is one of our principles. Replicate for whom, and under what conditions?

The kinds of things you were talking about in terms of the population of students that are being studied is one good example of that. Just in some cases knowing Latino doesn't help you really understand the conditions under which the findings at the end of the day will replicate in different settings. And that is a particular challenge in education.

So I just wanted to make the point that there is at least a preliminary discussion of the types of things that you brought up in Chapter 4.

DR. BERLINER: Hi, my name is David Berliner from Arizona State University.

My concern follows a little bit on Gill Garcia's statement, so I thought I would follow now. The concern is that in the report a lot is made of education research as like all of the sciences. And I have the feeling that you overstated it in a way, and wanted to comment on that.

I often make the distinction between the hard and soft sciences as the wrong distinction. That the real distinction is between the hard and easy sciences. And that the easy science is what physicists and geologists and chemists do. And the really hard science is what education researchers do.

And two examples of this come to mind. The first is the decade by treatment interactions we get in educational research. Geologists don't deal with these kinds of things, as I understand it. For example, everything we did, good science in the sixties, on women's motivation to achieve is completely irrelevant in the eighties. It's all wrong. Was the science bad? I don't think so. But those kinds of interactions occurred to us in ways that make our science more difficult than let's say some of physics and some of geology, for example.

The second issue, and this is really I think part of what Gill was saying, is that I don't think enough is made in this report of individual differences and interactions because of that. While small group work appears to be really good for many kids, it isn't good for all.

The attention rate is bad for most kids, but isn't bad for all of them. Phonetics instruction seems good for many kids, but apparently there are some kids that don't need it all. Taking notes during lectures is good for some people. It actually interferes with learning for some other people.

These are the kinds of interactions that make our science a little bit different than some other sciences. It's a little harder to do. I think because of decade by treatment distractions, because of interactions associated with individual differences, they limit severely our ability to generalize. And they make to some, look weak or soft, when instead our science is pretty good. It's just much harder to do. I would like to see that notion discussed more. We are a little bit different than other sciences, because of these things.

The second issue I would have maybe stressed a little more in the report has to do with the issue of values in educational research. Let's take a topic where we have good research. I think we did a meta-analysis of research on attention and grade. We would find out that most researchers, a consensus panel would say that attention and grade is probably the wrong decision for 950,000 of the 100,000 kids we leave behind this year.

But we are going to leave behind 1 million kids this year, when the preponderance of the research says it's the wrong decision to make. Why? It has to do with the fact that you are going to be punished for not working hard enough. It's a values issue, not a research issue.

Shared values and purposes as a school culture make for better schools. Well, we have known that since Rod Edmunds. That's 20 or 30 years now on effective schools literature. How come principals don't spend their time doing that? What are the values they uphold that keep them from doing that?

We've got research now, year after year after year that says reading begets reading. The more you read, the better readers you are. And yet the school curriculum doesn't seem to be fostering just plain old reading. How come? What other values are interfering with that?

Certified teachers outperform non-certified teachers. If we did a meta-analysis now, we would get an effect like 5 percentiles, 10 percentile ranks a year on standardized tests for the certified university teacher versus the uncertified teacher. Well, how come we have in many districts, over 50 percent of the teachers uncertified?

Our research and our science is pretty good. The summer drop off is another case. Reciprocal teaching -- here 18 studies in a row tell you get a big effect for reciprocal teaching, and you never see it in a school. What is it that is going on in the values issue is that when we do have good research, it doesn't get implemented?

And I don't know the answer to that, but it seems to me that that's a pretty important question. Of course, the better our research gets, the more it keeps hitting up against other issues, like the values that are either rejected in the research, or find it less important than the other values that people hold. We have to somehow start looking at that.

Thank you.

DR. GRAHAM: I could just add to David Berliner's very eloquent statement that Brian Edmund's additional work was funded by the National Institute of Education.

DR. DE HAAN: One other comment? We did a lot of discussing about the business of applying the medical model to the educational model. And that is one of the most -- I used the term "passionate" a couple of times. But anyway, clearly it doesn't apply in many cases, but in this particular aspect there is a certain applicability.

By that I mean that conceive of the situation in which the medical and biomedical research was showing all the things that we all know that it shows, but your physician was still using leeches and scalpels. The difference that I think is applying there, and the reason that isn't the case is that there is this wonderful culture within the biomedical and science community that continually upgrades the quality of the practice, as well and to a large extent based upon the current and past research.

And that, it seems to me, is one of the areas that we kept coming back to, that seems to be not as strong in the educational community. When the physicians go to their -- even the ordinary doctors, I don't even mean research-oriented physicians -- but when any doctor goes to his or her annual meeting of his or her specialty, there are all kinds of things going on to upgrade in the latest research in his or her area.

I have now for the last five or six years been going to NAST meetings, and the AAIA meetings, and I have been struck quite frankly, by the difference in the quality of the material that is given, in the effort that goes on to bring in young teachers -- what I would have expected, to bring in young teachers, and bring them up-to-date in the latest research in their areas, and so on. There is much less of that than I frankly expected.

DR. FEUER: I just want to follow-up, David, and ask you something about the business of individual differences that you raised. Is the point that the preponderance of research according to what you said, suggests that retention and grade is not the right to do? Is that the principle thing that we should be focused on? Or is it the fact that for the million kids who would be retained, there are 50,000 who should be?

Because I think what we've got to come to terms with is whether one of the goals of science is to achieve some at least reasonable basis for policy, even in the absence of certainty and sort of perfection in the analysis.

DR. BERLINER: The retention and grade I was using as an example of other values, like the need to punish kids. In light of research, the preponderance of which I would argue says it's the wrong thing to do, retaining in grade. You also increase drop out rates 30 percent in later life, et cetera.

But there are children for whom it will work. That's the individual difference thing, and we don't know who they are. So the policy decision is pretty easy. Since we can't predict in advance for whom it might be beneficial, don't retain in grade. So that's an easy one in a way.

I would argue that other kinds of treatments, when a teacher says, but it's not working in my class, even though you have research that says it works, that's a legitimate bit of feedback, because it may not be working there. I think the evidence is that context is important. And you do very well with context in the report I think, talking about the importance of context.

But you don't talk at the micro level of individual differences, motivation of kids, second language learners, new immigrants, poverty and its effects. The effects of poverty are pretty debilitating. So will it work in a low income school of second language learners? You know, a lot of times things won't, even though they are working quite well in other kinds of places.

What I think that does is weaken us in the eyes of many congressmen, in the eyes of the public, when in fact it's just a fact of life that our science is harder to do. It's not that our methods are bad. We did good science originally to get a finding we trust. It's that it doesn't apply everywhere. And Ms. Jones is perfectly right to say it's not working in my classroom.

Now do you want to believe Ms. Jones right off the bat? No, the self-reports have to be checked too. But it is quite possible things will not work as well as a broad public policy about wash your hands before you eat. That's a good public policy. That does work. We don't quite have the same power.

And even randomized designs, and I agree, I was trained that way. I like randomized designs. I could ferret information out, but each time you start adding in the interactive elements to try to figure out where the individual differences are, who is profiting, and who is not, you start slicing your sample size right down. And you can maybe make one or two cuts at some hypotheses about who this might work for or not work for, but as Cronbach pointed out, there are a thousand interactions going on in human life, and you can't get them all.

And I just think it makes our science harder, that's all. It is a difference that matters I think, as to the perception of that research, that I didn't think enough was made of in the report's desire to make us look like other kinds of scientists. I don't think we really are.

DR. FEUER: Okay, let's go on. Yes, I think you're next.

DR. ERICKSON: Yes, one of the reasons I'm here is that I've been -- I'm Frederick Erickson from UCLA. I've been pondering these issues of hard and easy. I like that a lot, Dave. And also clinical and basic, because I do both.

And that's especially poignant for me right now, because I have just taken on a new job as the research director in UCLA's laboratory elementary school. I'm an anthropologist of education. I finally went native, and now I'm living inside the school with the natives.

And we are trying to wrestle with how to do research in the way that takes account of the kinds of things that Dave was talking about, and other have here. And it's really not easy at all. So I'm hoping to ponder this report some more to get maybe some light on that.

I think you do overall quite a good job on the qualitative/quantitative issue, which is what people have asked me what I think about, because I'm a qualitative researcher primarily. And so I like that. And there is a lot I like about the report.

But I also find it puzzling, and this is a real question, this isn't a rhetorical ploy, this is a real question. I have a feeling, and maybe it's partly because I have been so paying attention inside my school for the last couple of years now, that I'm coming in on a movie fairly late in the movie, and I don't know what happened earlier. I really don't know what happened in Congress around the issue of science and educational research.

And I don't know what's happening at OERI around that, but judging from what the person from OERI said, something has been going on there. And I was puzzled that the report sort of didn't frame that very explicitly at the beginning, because it seems to me that it's terribly important that we think carefully, and do careful inquiry about the ends and means of education. I think all of us would agree on that.

But why science is necessary or desirable for that is I think, still an open question. And that wasn't addressed directly. So the very assignment the committee was given wasn't, for me, fully fleshed out, and I wonder about that. And that's not just a trick question. I really mean it. This is clearly an answer, this is clearly a rhetorical statement intended to address a set of issues, but what are those antecedent issues and happenings? I could understand the report better if I knew more about that.

DR. FEUER: Do any of the committee members want to try a short answer to that long and important question? Do you recall from the early deliberations how we framed it for you, for example? Why we convened the committee, and what the charge was as we understood it?

DR. BORUCH: Well, there is some material in the report about the backdrop, the variations in budgets, for example, drops in budgets. The fact that that is a signal that something is wrong with this picture, or something is wrong with the research, or something needs to be different about the perceptions of the research. That is certainly one good empirical base for our being concerned about the state of support for educational research in the United States, and the congressional interest in that, and other people's interest in that.

Second, the Castle bill was mentioned in I think the preparatory material. That bill was controversial, partly because it was so explicit in talking about standards of evidence or requiring standards of evidence in different kinds of research, including qualitative research. Not only the mentioning randomized trials.

Well, as some of you realize, it's a relatively blunt instrument, assuring that the Congress and the legislative staffers understand what the implications of this blunt instrument in that explicit new law seems like a good idea. One cannot just demand a controlled experiment. It's kind of akin to asking people to levitate. You have sort of do some planning first, and so on.

One cannot just demand qualitative studies, and expect them to produce estimates of effects that are defensible in a harsh scientific or political forum, because they don't produce that, again.

So part of the object here is to assure that the people on the Hill and in the agencies have enough in the way of relatively simple, but plain spoken principles to understand what is meant, what is intended by the phrase scientific research in the context of education. And to assure that we at least, as a committee, are in agreement about what those principles are, and how they might be applied in an education context.

There are lots of nuances to this stuff out there, but I'm actually not an ethnographer, as you may know, Fred, so I don't have a real good grip on all the inside stuff.

DR. FEUER: I would just add as a footnote that when we were given this assignment, one of the things we did, as we always do, was talk to people in the field. And I remember one very poignant conversation I had with a distinguished scholar to whom I said that we were about to launch a study of the scientific quality of education research. And his answer was, well, finally a short report will come out of the Academy.

That, along with other things that I heard confirmed a general sense that there is a perception, right or wrong, that the scientific quality, whatever that means, of education research is not as good as it could be, number one. Number two, that the problems in education are so fundamental to American society and to the future of the American way of life, that we better apply the best tools we have, whether they be scientific or other, to addressing them. And that there is a perception that we haven't done a good enough job of that.

Now I'm not endorsing that perception. I'm reporting my understanding of what that perception has been. And third, that there is a longstanding, historical traditional in American society to turn to science, technology, and rational thinking as a way to address very fundamental complex problems.

So I put all that together, and it sort of made sense that the question about what is scientific research in education, and what could be done to make it better would come to a body that has been thinking about that in other ways for a long time.

One of the things that happened in our first committee meeting was a discussion about evidence-based and you can fill in the blank. And this is largely about evidence-based education policymaking. And so the analogy was quickly made to evidence-based medicine, which for me, was a completely new and very scary concept, because I thought it always had been evidence-based, until we found out that in fact evidence-based medicine is a kind of new thing in the world of medical research.

So suddenly the education researchers on this committee felt like they were in even better company than they thought, because even medical research, there is this concern that it isn't enough evidence-based, or that the practice of medicine isn't enough evidence-based. So that gives you I think a little bit more of the part of this conversation that we didn't report on explicitly enough in this document. But that's where we got into it.

DR. DE HAAN: And also at not time, I think, was the sense on the committee that we were unaware, or not acknowledging the complexity of education research, which among all of the social sciences probably has the biggest row to hoe, as it were. Although it is, as with all of the social sciences, more than the physical and biological sciences, beset with these problems that were just mentioned of individual differences, and so on.

But the three questions that we started with, sort of assumed as Michael just said, assumed this history, which is briefly outlined in the first section of the report, and simply said, okay, given this history, given this perception of lack of confidence in educational research for whatever reason, what can we do? How can we look at the situation in order to better understand it?

And so we started with the three questions that are outlined on page 15, as I think, of what ought to be or what are scientific principles that would apply to education research, and do they apply across the board? How can the kinds of evidence that is provided by such research, accumulated? Not only accumulated, but also be translated from the research venue into the practice venue.

And then finally, how does one go about producing a political situation, an agency, that might be able to foster these kinds of progress in educational research? If we didn't make that clear, if we didn't make those characterizations clear enough, then obviously we have to go back and do a little more work.

DR. FEUER: Let's go on. We are going to extend this conversation another 10 or 15 minutes, because I think we are getting into some good stuff here. And then we'll have lunch.

DR. CHELEMER: Carol Chelemer from OERI. I'm really glad to see the conversation heating up a bit. I'm always interested in the dogs that didn't bark. I understand that it wasn't the charge of the National Academy or the National Research Council to actually analyze the quality of education research. But if had been, my question is this, how would you have undertaken such an analysis?

Dr. Graham made the point that in addition to the Department of Education, there are other federal agencies, foundations, universities that fund research. And I wonder if you might have hypothesized anything about the nature of the research, the quality of the research in terms of the funding agency, given the discussion in the report about the necessary culture to really promote good quality, scientifically-based research?

DR. FLETCHER: Well, frankly, I don't think we could, nor do I think anybody could take something that is defined as broadly as education research, and have a discussion of quality. I think it would have to be broken down into for example, content areas or around a particular question or something like that.

That was why the committee chose to focus on general principles, because I think you can talk about principles, but to talk about quality, you have to get into the content itself. How good is the retention research, for example? What do we know about learning and reading, do math for example. I mean I think that's the way that you would have to do it.

DR. NODDINGS: I was mentioned here that you will get an opportunity to talk about this topic, along with several others this afternoon.

DR. CHELEMER: Well, actually I won't.

DR. NODDINGS: It's too bad that you won't be here, because one purpose for the focus groups is to discuss issues like this. We are going to share with you something that the National Academy is thinking about doing, the National Academy of Education that is, is thinking about doing. We are very, very aware that we could use some help in thinking it through better.

And this question that you have raised, what criteria will we use, and can we come up with general criteria, or do we have to come up with separate criteria for different questions. That's what we were going to invite you to talk about this afternoon. It's a temptation to start right now, but I think we shouldn't.

MS. TOWNE: I think the other thing about taking on that kind of question is that to answer it, you would have to then put it in the context of compared to what? Simply coming up with an answer that says along a Likert scale that education research is a 2.5 on a 1-5 scale -- I'm oversimplifying, but even if you were to be able to come up with something like that, it still wouldn't be meaningful, especially given the context that some of the people here have tried to describe would be meaningful.

Could the same be said for example in medical research? Could the same be said for other types of research? One of the points we make in the report is that we think there is good and bad research in all fields. And that it's really not constructive to try and come down hard and fast with a real analysis of the quality of such a broad number of studies, and number of approaches, and disciplines, and that sort of thing.

So in addition to just the sheer complexity of your exact question, how would you even go about designing something like that. So we punted on it, just because we didn't think we could do it. But I think we also thought that even if we could, it wouldn't be the most constructive thing for us to do anyway.

DR. GRISSMER: Let me make one comment. I kind of disagree with that. You have to do it. You really don't have a choice. And it's not that the judgments are always going to be right, but there needs to be a built in self-evaluating into the research system.

Part of that happens in peer review, but that assumes a peer group who has the common shared knowledge of what goes on, and where we are going. And NIH is an example. They bring together panels every year that tries to define priorities, what is good, what is bad. I mean judgment gets made anyway. It's not that it doesn't get evaluated. It's that things like consensus panels are part of the process.

There is not a self-evaluating process built into our system now that allows that to happen. And it does need to happen. It is something that needs to be done. It's difficult, but it can't be done perfectly. But I think there are ways in which we can do it much better than the current ways we have.

DR. GRAHAM: I just want to agree with what David has said. Another way we do it in one sector that does educational research, by no means the only sector that does educational research is through tenure reviews in universities that have very high standards for who gets tenure. Places where not everybody gets tenure, but only some get tenure.

It's not a perfect system. Peer review isn't perfect either. But that is one way in which this kind of thing gets done. And that's why from my perspective the alumni group, the National Institute of Education and OERI in its early days is a very interesting group to look at, to see what many of those people have done subsequently.

DR. BORUCH: One kind of response to your question is to remind you what's in the report. The quality of the evidence, its appropriateness depends heavily on the question that is addressed. So for each of those questions, what has happened, what is the effect, what are the mechanisms, different standards of evidence might kick in.

Within any given area, another part of the report says sort do a literature review before jumping in on this stuff. Well, it turned out to be the case that the Cochrane Collaboration in health care, which focuses mainly on randomized trials since 1993, has produced something like 2,500 systematic reviews of randomized trials in a variety of areas.

So if you are interested in behavioral approaches to left-handed children with asthma, who are between the ages of 11 and 14, you hit the button, and you get a systematic review of the half a dozen or two dozen randomized trials in that arena. Thirty or 40 have been screened out, because they are imperfect on a variety of accounts.

The standards used in screening these studies in and out are part and parcel of the business of evaluating the quality of the evidence. Now even there, there is a fair amount of ambiguity, because in some cases you cannot achieve the same kinds of control that you would like in a sort of laboratory setting versus a field setting.

In the social sector, the Campbell collaboration I hope will be emulating that Cochrane effort. That is to say, generating systematic reviews in which the standards of evidence are made transparent, they are more or less uniform. They may change over time as we increase our knowledge about what constitutes reasonable evidence in particular cases. And it will do in education, as well in justice and welfare.

We can take another step and say, well, what about evidence relative to the uses to which the information is put? Some of you may be familiar with the book entitled, "Probability of Statistical Inference in Ancient and Medieval Jewish Literature." There are good lessons in it about this connectedness between the quality of evidence and the character of the decision.

One of the stories concerns a debate among the rabbis in the 13th century about how one ought to draw a sample of an olive crop in the interest of tithing to the temple. That is to say, establishing the value of the crops, take 10 percent off the top and so on.

Well, one can, for example, hire someone who is sworn to a vow of poverty and honesty, such as a graduate student, send them out to the olive grove, sample the olives, bring it back, declare the value of the crop based on that convenience sample.

Or one can bring together a bunch of experts inside government, as well as outside, draft the RFP. Get it vetted in wider places, issue the thing, and have our for-profit and not-for-profit colleagues bid on this thing and generate a much higher quality estimate.

The rabbi debated which way to go for something like 20 years. These are profound issues. The resolution that they reached was that in this kind of case, the resources that you invest in generating the information should depend on the origins of the demand for the information. If the demand is merely the benefit of bureaucratic in origin, you just hire a graduate student to just go out there, grab a handful, and bring it back.

If on the other hand, it is biblical in origin, if God wants it, you then better get a better estimate. Where the Congress falls and secretary falls in this is not exactly clear. But that connectedness is another sort of arena for another kind of research, and another kind of conversation.

DR. FEUER: Does anybody really want to ask another question?

DR. MAXWELL: I want to go back to David Berliner's comment that it's really values rather than research results that primarily drive practice. And as a methodologist, I want to comment on some methodological implications of that.

The first is that this is not something that is sort of a separate issue from educational research proper. This is an empirically researchable phenomenon. That is, just how do values influence the implementation of research-based or nonresearch-based education practices? It's not a phenomenon that is particularly amendable to randomized experiments.

There are lots of ways you can study it, but it's a way that particularly lends itself to qualitative research, because it is contextually complex, it deals with issues of meaning, and it is situationally variable. The values aren't the same in every situation. Different schools are all different in terms of what things are influencing the implementation of a particular practice.

And it's a kind of phenomenon that is often studied in educational research. That is, what are the contextual factors, and motivational, cognitive factors that influence educational practice? And I'm thinking of a study by Mary Lee Smith and Lorrie Shepherd some years ago of how kindergarten teachers' beliefs and values influence their decisions to retain or advance a student to first grade.

Now one of the things that I wasn't completely happy about in the report is that I felt that there was a kind of an implicit and subtle devaluing of this kind of qualitative research in terms of its explanatory ability. That studies like this can in fact explain why kindergarten teachers do or do not retain a student.

That there is a tendency to treat this kind of research as descriptive. To not give it full value as having an explanatory power that is equal to, although different from the value of randomized experiments. In the interest of time, I won't go into what those differences are. But I have written a paper on that if anybody is interested.

DR. FEUER: Can you just say who you are?

DR. MAXWELL: Joe Maxwell, George Mason University.

DR. FEUER: We need to offer someone the last comment. Miron, since I asked your indulgence, you do get the last question.

DR. STRAF: Thank you. I wanted to follow-up with one of the comments made earlier, and also David Grissmer's remarks to a question to the panel. The report argues that the scientific principles apply broadly to all disciplines in field, that knowledge is accumulated in similar ways. When putting it together for determining whether something works, or why it works, are the standards of evidence for education different than those for the other sciences?

DR. BORUCH: Miron, could you rephrase the question? I didn't get it either.

DR. STRAF: This is particularly relevant to us at the National Research Council, trying to infer whether something works, a particular policy. And we draw upon evidence of a wide variety of things. Some people have argued that in the behavioral and social sciences, mostly in social sciences, the standards that we use are different from those in the natural and physical sciences. Others argue they are really quite the same, but that we have different contexts, different situations.

And I wondered if any of the panel members had addressed this issue of trying to pull together the evidence from education studies. Do we weigh them differently? Do we combine them differently than would be done in other sciences?

DR. FLETCHER: Well, I think that what the report would say is that what all branches of science have in common is the formulation of warrants and the development of a chain of reasoning. And that what we know, regardless of whether it's social and behavioral versus physical still depends on the chain of reasoning, and the number of lines of investigation that you can integrate into a coherent explanation of a phenomenon.

In that sense, I don't think the report would say that the standards of evidence are really different. It's still a matter of how a group of people integrate information to come to a consensus. And the idea that that process might be different, I personally would find difficult to see.

DR. GRAHAM: I think this is a means-ends question. And I think that education, and research about education deals both with the means of achieving an educational goal, but it also deals with what the goal is.

How do we define right? Does it mean that it is students who are retained, do not score as well the subsequent year, on a reading test if they were not retained? Does retaining students mean that they will develop a better sense of discipline because they have learned that they have to work hard in order to make progress?

Does it mean -- what is the right thing to do? And I think that the ordinary modalities with which the cursory reader would consider scientific research, do not yield smoothly with figuring out the value question of what the right thing to do is. And education is at least as concerned about figuring out what the right thing to do is, as it is in figuring out the means to do the right thing.

DR. DE HAAN: But it is also true, and Miron, I think that's a wonderful question. It reminds me of a two or three hour discussion on that very question that we had at one of several meetings of the panel in which we concluded, I think, at least after several more discussions, concluded that partly it depends on how you define the "standards of evidence."

I think what we would say is that the standard of evidence is not different from other sciences, but the application of the extraction of evidence, the recognition of the noise level versus signal is very different, and therefore, has to be taken into consideration.

So I think it's whether you are using the term standard to be sort of synonymous with principle, or whether you are using the term standard to be a statistical technical term.

DR. GRISSMER: My take on this is partly the same. In the physical sciences you can trust measurements better. I mean both the technology and the fact that hydrogen molecules are the same whether they are here or other places. So when the measurement is made, you have a much better signal to noise ratio, and you are trusting that measurement is more precise. And when you do your theories then, your theories can be more refined, because your measurements are better.

In education, and I think other social sciences, there is not much as trust in the measurement. It's just the way it is. It's because of the contextual effects, or it's because we never have sample sizes. It's because of a lot of good reasons why no individual measurement is ever decisive.

That's why in one of the Brookings books I think, there was this whole discussion of the gold standard measurement. Well, there is no gold standard measurement. Experimentation would be the gold standard. But in social science and education, what you are looking for is a convergence of different evidence that seems to show the same kind of thing.

And I think it stems from the fact that no measurement is as trustworthy as we would like. And my example of this is this business of do resources have more effect on minority and disadvantages than higher ability kids? And I think there is no single measurement that is going to tell you that, but I think there is a wonderful convergent to that from a number of sources which come to that conclusion.

So I think the standard has to be a more convergence of different measurements. But ultimately what gives you confidence of the standard is in the theory, the accuracy of the theory that it can forget. Ultimately, that's the standard that needs to be used. And I think a weakness of theory here is a factor in fact that our standards seem to be a little slippery.

DR. APLING: I just wanted to kind of bring us back to some of the political realities. I was reading an article in the Hill, which is an alternative publication. Congress is rethinking that maybe they shouldn't have abolished the OTA, and a lot of us agree.

And they were interviewing several members of Congress who are trained as scientists. And one of the members was Rep. Germet, I guess who is trained as a psychologist, who was making the point that Congress will pay attention to science sometimes, and sometimes not. They are selective.

And his example was they will pay attention, and they need advice on anthrax, but they won't pay attention on cloning. And I think this is another dimension of the values. And it becomes David Berliner's point about how this is really hard. This is another dimension of it. It's hard because we all went to school. And I think that was in some of the previous research on this. It's the we all went to fourth grade phenomenon.

And I run into that in briefings, where a congress person has said, I know all about education. After all, I went to school, and left the briefing. I proceeded to brief his dog. But I think that's another thing that makes this very hard. And even if we are talking about "hard science," medical science or whatever, there is this other political value aspect of it.

And this brings me to another point, which is are we really going to pin down the Congress to fix this for us? I'm not very sanguine about that. I think it's up to the profession to improve the quality of professional education. I don't think you want the Congress doing this for you, because what you get is dictation about what you should be doing, and how you should be doing it.

And I think the report makes a very good point that it's not an algorithm, but legislation sometimes tends to be algorithmic. And I think we need to kind of keep that in the back of our minds while we have these other discussions.

DR. FEUER: I think we are ready to pause here. The alternative would be that we convene ourselves as a new committee, and try this all over again. And I don't think we could do that this afternoon. But we are going to try to actually move this discussion into some practical application. And to tell us more about that, Lisa.

MS. TOWNE: [Administrative remarks.]

[Whereupon, the meeting was recessed for lunch at 12:20 p.m., to reconvene at 1:00 p.m.]

A F T E R N O O N S E S S I O N (3:30 pm)

DR. FEUER: We're down to a somewhat cozier group, which is fine. I'm glad to see people still smiling after this kind of a day of what I consider to be tough intellectual gymnastics and some hard thinking. I know that in my group it was quite compelling from beginning to end.

So what we are going to do now is just spend a few minutes sharing the highlights from our respective focus groups. And then we'll have the great pleasure of hearing from that man, who I think almost everybody here knows, but if not you'll get to know him real soon. So I'll save my introduction for him until we are done with our little focus group wrap-ups.

I think we are the four rapporteurs, so I'll start with Nel and your group, if that's okay.

Agenda Item: Report Out From Focus Group Moderators - Research: Nel Noddings

DR. NODDINGS: Well, from my perspective, our session was just terrific. I mean people were enormously helpful, and they talked to each other. If there are people here who were in that session, and I miss some important things, please feel free to add to it. But it seems to me these are the things that came out as really important.

The first was the great need for information for practitioners. Now we all know that's important. But coming from the research group, this is really quite dramatic. That was really the emphasis, that people needed to be able to sort through things, and figure out what was worth trying, and what wasn't. So the idea of high quality synthesis came out.

As part of that, there wasn't great enthusiasm for the National Academy's idea of an annual report on research. I think by the end of the session people said, well, if that's part of the project, that would be okay, but it better not be the whole project, because nobody will read it, and it just won't work.

The second thing, and is part of the first really, but it raises the question of how do we reach people? So first of all, who is the audience, and somehow or other we ought to be sure the practitioners are the audience. And related to that is the question of barriers to dissemination. Why is it that things don't seem to get through? Even those things that we're pretty sure of don't seem to be picked up. And the group felt that we better be very sure to study that problem before we got too far into this.

Closely related to this, if we are going to do a report, some kind of annual report, and one thing people did like was the idea of choosing one important topic, and concentrating on that, rather than trying to do a synthesis of a whole range of topics of research.

The group pointed out that there is a teaching function to this kind of project, and that we ought to take that seriously. That somehow or other, we ought to teach people how to use this research. And it raises questions about conducting focus groups around the country involving superintendents, and professional organizations, and what not to get groups of this sort set up.

Then from my perspective as a philosopher -- but I didn't raise this question, someone else, a very important one -- is we should not just ask the question what works, but why we would even want it to work. In other words, there ought to be more attention to ends. What sort of things should be seeking, and not simply seeking this to get it. That seems to be important.

And I think we agreed on this too, that we need to work hard on ways of identifying good and bad research, and do something about the problem of popularization. So people mentioned some pseudo-research that is out there that appears in US Today and Parade magazine and other places, and people seem to pick up on that, and they don't pick up on the bona fide research that we would like them to read. So that's the other problem to study.

One or two topics a year, one preferably done right. I already mentioned that.

And then finally, and again I think an extremely important comment from a group of researchers was that we should be careful not to let scientific research bury other forms of inquiry. The National Academy, what it stands for is the promotion of scholarly inquiry. And that covers that range of ways of studying education. It isn't simply limited to scientific research.

People pointed out again that if a claim is made that the researcher is doing scientific research, then all of the material from this very good report becomes relevant. But if that is not the claim made, it doesn't mean that the research is of no use, no value.

That I think pretty much covers the territory of what we talked about. Anything to add, to correct me? No, okay.

DR. FEUER: Good, thank you. I guess we'll just go through all of these, and then if there are questions or comments, we can go back open this up a bit.

So, Kenji.

Agenda Item: Report Out From Focus Group Moderators - Federal Policy: Kenji Hakuta, Stanford University

DR. HAKUTA: Thanks. I won't add redundancy. There are a lot of things that you said, that I think are issues that we may resonant to.

The first thing I'll say is that to the extent that there are transcripts made of that session, I have permanently disqualified myself from applying for a job as focus group leader. I tried to stick to the script, but it didn't work very well, and part of why that didn't work very well was because it was very clear from the beginning that this idea of synthesis, while the idea of doing synthesis is good, there are people who are doing it already. And that the National Academy either of Education or Sciences, there may be value to it, but there may not be value to it either.

So then there are questions downstream from there like should this be a committee of how many people? Or how many papers might there be on this particular topic became less relevant. And at an abstract level this idea of having consensus and so forth, and having it be good, and consistent with the principles of the report is well supported, but the eighth actor of that function remained unclear.

There was a lot of emphasis on issues of the question that is being asked in the synthesis. By whom? Who frames it? The principal who happens to be on our board made a real impression on a number of our group saying the most are the kinds of questions that would speak to the sets of issues that she raises. So asked the question, as well as from a scientific point of view, let the question drive the process.

And since our task is really framed around process, the way in which a question might play itself out would be different, depending on the kind of area, and particularly in the public domain and so forth. And since my group was the federal office people, people come with that perspective as well. I think that's why some of that was highlighted. So from the office's point of view, you've got your own set of programmatic issues. And the questions are best framed in that context.

Again, we did talk about the value of the National Academy of Sciences, and there are some important status issues involved that give it appeal. On the other hand, there are some issues having to do with people questioning its record in the area of dissemination, and connection with practice is less established. And so whether that's the right venue or not.

There was a lot of discussion about audience. While our directions for discussion was to think in terms of them, the people in the focus group as the audience, we kept drifting back to how are we going to get the practitioner as the audience? How are we going to get policymakers as an audience?

Also, the intermediaries who deliver technical assistance, the labs and the technical assistance centers, would they be the appropriate people in that the kinds of reports that we issue would really have to be tailored to the audience. So we weren't very clear about who the audience would be.

The idea if you are going to do a synthesis, to have several papers in it was roundly rejected. It you are going to try to do a synthesis and a consensus, then it ought to be a consensus.

DR. NODDINGS: Our group just said that was an old fashioned model.

DR. HAKUTA: Finally, there was some sense as to what kind of topics, if you were going to do a synthesis or an activity around it, there are some areas like early reading, where there is clearly potential for synthesis, but others that are more low hanging fruit, where there isn't much, and that might be a different kind of report that could come out, that is not a synthesis.

But really saying here is work that is really important, for which there is good ground work already established, and therefore we ought to do research in this area. An example of that being the area of reading comprehension that the RAND panel on reading has come out with recently. So that's another activity that might have the potential for building a constituency around specific research areas.

I'm sure I missed a lot more from the group. Does anybody else from that group want to add?

DR. FEUER: Okay. This is David Berliner.

Agenda Item: Report Out From Focus Group Moderators - Associations - David Berliner, Arizona State University

DR. BERLINER: We too had a good committee. And again, I hope at the end you will correct me or add something. Our committee was made up of associations primarily. They thought the reports were a good idea, but they did suggest also checking with OERI on their possible new program called What Works? Some sort of clearinghouse that will also put an imprimatur on reports, and might duplicate some of this effort. But I know nothing about that other than there is something else going on.

It was important to the members of the associations that the imprimatur of the Academies be on such reports. And they pointed out something out, how important it is to have the National Academies, both. That imprimatur gives a certain kind of status and legitimacy to research in ways that an article in Education Leadership doesn't.

They thought the reports can go two ways. That one way they can go is that there is something to take seriously. That is, in the synthesis and the analysis there is some signal to take seriously.

But the other way the report could go is that the synthesis leads to a statement that there is no evidence here. And that is equally good, because in a government that seems to be very concerned about evidence-based education, it's a chance to say why don't you spend some money for some evidence. Because if we visit it five years from now, it would be nice to have half a dozen or a dozen studies on this topic, because the topic is barren right now. So the idea being that either way the synthesis went might end up good for the field, for the profession.

The report gives education research permanence. Thus, some things happen. The teachers would need more research training, so they too can not just be told things. They can actually read them and understand, so they need the research training.

They need some time for discussions of research built into their professional lives so they can digest it, and talk about adaptations to their own environments.

The teacher education courses need to teach these finds. They are too often forgotten in the dissemination issue, but the teacher education programs also need to be up-to-date, so that this is a continuous set of issues here.

Their concern for audience was also clear. And in our group one scenario we talked about was that the academics, the committees write the report, and they didn't like the report process either in my group. They weren't sure that commissioning three viewpoints and bringing it together was sensible. But a process has to be worked on.

But if the academics write the report, then the associations representing all the constituent groups, PTA and school administrators, and teachers, they might be able to rewrite and disseminate. Because the feeling was that the National Academies don't have those skills, and as a member in both camps, I think they are right.

So that it might be wise to think of right from the start, having practitioners, association members helping in this process, setting up the dissemination links from day one about what will happen next. And let them do that, because they know how to do that better than perhaps we do. So start to talk about dissemination on day one, and not leave it until the end.

Something about the follow-ups came up, and made a lot of sense to us that you don't just write a report, find ways to disseminate it, and leave it. You have to study the process. And is it being used? Who is using it? Did this information get out? Has it changed practice? Is there any evidence that the practice that has been changed has led to results that what you would like, et cetera? And we're not good at that. We need to do that too.

The feeling was that topics were not hard to come by. That they bubble up. It's just that not all of them may be researchable. The term used was to pick the enduring topics, not necessarily the popular topics. And there are some enduring topics. What do we know about homework policy? What do we know about parent involvement? These are pretty enduring. They are not just popular, even though they have some popularity at the moment.

I suggested that it would take two years to get a report to the Academies from the day you pick a topic, to the day we disseminate something. I was just guessing, because we needed to talk about what happens when hot topics come along. And the answer is that's not what the Academies can do probably. That's why we have to look for enduring topics, rather than hot topics, if that distinction makes sense.

The issue of release came up. The thinking was that if reports are going to be released, they ought to come out right after January, or at the latest late winter, because that is the time when people would plan their summer and fall activities. It's also a time to try to get the funds for summer and fall activities from many districts. So a report that something that needs commitment of funds at the local or state levels needs to come out at a time when that is still possible for the following year.

The group was quite clear about if a report came out on some topic, that people be brought together to talk about applications of the findings, tips about using the findings, concerns about using the findings. That is, perhaps as the report went out for review, a panel could be brought in of people who would deal with applications, tips, concerns.

The transcript of that could be synthesized, and that could go out with the report. Because practitioners are going to see problems and prospects much easier than perhaps the people who wrote the reports, and that could be amended to the report, and then the whole package could go out together.

The group was also a little concerned about once you start packaging these reports together with the practitioners' views, that the reports lose some status, which is interesting, because of the way that practitioners are viewed in this world. And that's a funny problem of our field, that in a sense everything we do is to influence the practitioner, but the minute we do, some status is lost. And I don't think that's a minor problem.

If the imprimatur of the Academies is to mean something, then you don't want to take away from that. But if it's to result in any utility, it has to be linked to the practitioners, which takes away something from the shine of the imprimatur. This is a problem of our profession that we have to at least think about.

That's my report, except as amended by people from my committee. Anybody? Thank you.

Agenda Item: Report Out From Focus Group Moderators - Schools: Michael Feuer

DR. FEUER: Well, I just want to cover a few of the themes that came up in our focus group, which consisted primarily of people who were either teachers or working in schools, or at the school level.

First of all, I want to say that there is a sort of implicit hypothesis out there about there being low demand for educational research as an explanation of why educational research is hard to get funded and all of that. Now without getting into too much of this, based on the limited sample size of my focus group, I am prepared to suggest that that hypothesis needs to be rewritten.

Because the teachers and school people who were in our group expressed an incredible appetite for serious, scientifically-based, evidence-based advice on how to solve problems that they are confronting in their classrooms, and in their schools. So I find that very heartening, and it actually tells me that not surprisingly perhaps, people who have been looking at this from the vantage point of research, the research community, the scholarly community, have missed something here.

And that there really is a demand side that we need to learn more about. I don't want to go further than that based on this small sample size, because I will get in trouble with my scientist friends here, but you get the idea.

Among the themes that emerged in our conversation, certainly the reputation, the credibility, what has been referred to here as the imprimatur of the Academies is considered to be significant. And we discussed this in terms of how people in schools who are confronted with a barrage of good ideas, could very well benefit from a process that filters those ideas, and tells them which ones have the strongest evidentiary base, and thereby the most reasonable thing to pursue.

And that if this were done by a reputable organization that has a minimum of its own kind of interests in any particular type of intervention or strategy, but is more committed to the idea of science providing a neutral view of these things, that that would really be very useful to this community.

I prodded the group a little bit to think about what other audiences there might be for this kind of product in addition to the school people. And there were all quite ecumenical about recognizing that this could be a value to the research community. This could certainly be of value to the policy community. And then we got back to thinking about our own interests as school people in this group.

But I think lingering there was the thought that if a product like this is going to be valuable to all of those audiences, that's quite a challenge. Because maybe some kind of a mixing and matching would be necessary to get the right product to specific audiences.

Another theme that came up was indeed that this kind of report could become a "yardstick" for the research community. That is, it would be value to researchers to see what a group of people whose job it is to synthesize, and to try to find the highest quality research, what they are coming up with. This could create incentives to the research community to focus on topics and methods and the culture of science most effectively.

I want to try to pick out the things that I think are in some ways special vis-a-vis what the other groups have reported. It's interesting that one of the criteria that this group suggests for making this kind of report successful is that it be able to clarify the how generalizable the knowledge is from some body of research, and in what contexts it is most likely to be applicable.

So you can imagine that there are lots of studies that lend themselves to being misunderstood or misappropriated. And the results of which are used in ways that either go well beyond what the study was able to demonstrate, or somehow misses the context in which the study results were relevant to.

Another theme, and this came up in other groups I see also is that this report could point out what is missing in research. That is, what are the important questions for which we don't yet have good research? That would be valuable and a contribution. And indeed, I would amplify that by suggesting that researchers themselves have an obvious incentive to argue that the idea they have to study is worthy of funding.

The question is whether the community as a whole would agree that these are the priority areas that are worthy of funding, and so this could actually be quite interesting from the point of view of filling those gaps.

Other issues that came up in our discussions have to do with the practicality of this kind of product, making it accessible, readable, clear, frequent enough to be really amendable to the lives of professionals in schools. Another challenge again was about this business of whether getting three people to write independent reviews was a sufficient, or even a necessary basis for this kind of synthesis.

One question that came up was whether this would involve people who are working in the field that is being reviewed, actually reviewing that field. And we get into all kinds of hang ups about whether that upholds the reputation of neutrality and independence that we think is so important to this kind of effort.

And finally, someone asked whether there would be opportunity for dissent once the report comes out, which was a reminder to me at least of the fundamentally democratic instincts that pervade all of our work, characterized by the mural that I remember when I was in college, where the students were all invited to write whatever graffiti we wanted on this one wall.

This way, by the way, the antidote to having graffiti everywhere. They said, okay, you can have this wall. And one day somebody had come and written, "Challenge Authority," which was very popular in those days, by the way. And the next day somebody had come and crossed that out, and written underneath it, "Says Who?"

I keep thinking about that from the point of view of every time we come up with something that is intended in some way to be authoritative, the next question is where is the big opportunity for dissent. And I welcome that, at least conceptually. But it was clear there are some logistical questions that we're going to have to deal with.

In any event, I think those are the key points. And again, I want to just say how terrific it is to speak with people who are actually doing the work of education. First of all, because I come out of these conversations with new ideas that I think would be worthy of research. And secondly, because as I said, this kind of demonstrates to me that there is a real appetite for science-based, evidence-based knowledge to contribute to their world. So I thank them for enduring my focus group lack of skill.

Anybody want to chime in or add something, or comment on any of this? Great. Well, thank you for helping us with these focus groups.

We do have a challenge.

DR. HAMMOND: I'm Peirce Hammond from OERI.

A couple of thoughts, one of which came out of the kind of general conversation as we were milling in here. But before that, two ideas that didn't quite reach the level of being commented on Kenji that came out of our group I think.

One had to do with the very notion of a paper, a synthesis as a paper, and whether that was the right model, especially for practitioners. The second is linked to it. That is, we live in a technological age. Why aren't we talking more about technology and its role in all of this? I'm not sure I heard that from any of you as commentators, though it may have come up.

And the third point that came out of the sort of milling around that we did is that there is a body of work on dissemination, that is research into dissemination. That possibly needs to be looked at again. It has not been as active as perhaps it was 20 years ago or so, when there even a unit within NIE that had that as its principal focus. But there might be a lesson or two that we ought to take back out.

DR. FEUER: Very well taken, Peirce, thank you.

Indeed, in our group we did talk about the use of the Internet and computers as a way to communicate the results of this kind of synthetic process. So thanks for that reminder.

No further comments, questions?

Then I think we are ready for our closing presentation. It is a great pleasure to introduce Joe Conaty, the acting deputy assistant secretary in the Office of Elementary and Secondary Education, a long time contributor to the knowledge about education in America, a great supporter of the development of more knowledge about education in America, and somebody who ever since I have been in Washington, has been one of the most reliable fonts of knowledge and wisdom on any of these topics. So it's great to have Joe with us today.

Agenda Item: Invited Closing Remarks - Joseph Conaty, Acting Deputy Assistant Secretary for Elementary and Secondary Education, US Department of Education

DR. CONATY: I want to first of all, congratulate Lisa and Michael on the report. It could have been more timely given the legislation that was signed two days ago by the president. As I read the report, and as I talk to people about your meeting today, a lot of the conversation has focused on the supply side, and the generation of knowledge, and the generation of reliable and valid knowledge about the improvement of education.

What I want to talk to you about today a little bit is the new legislation that passed, and the profound impact I believe that's going to have on the demand side. Both David and Michael in their summary comments used the word "demand," and I think people are dramatically underestimating both the nature of the structural demand that the legislation has created. But also the nature of the structural demand that systemic education reform as being driven by the states had created.

And I did this slide presentation last night, so if there are typos in some, I apologize. I don't expect you to read all of the words on the slide. They done in part to demonstrate the dramatic nature of what has happened in the legislation.

The first thing I want to talk about is the definition of scientifically-based research that's in the legislation. Then I want to show you just a few instances of the more than 150 times it or one of its synonyms is used in the legislation; what the government has done before in response to this demand; and what I think some of the implications for those of you who produce this knowledge, synthesize it and promulgate it in its various forms.

The word "scientifically-based" research is literally all over the bill. It is essentially a central component of the modern accountability conversation. The bill has four principles really: accountability, flexibility, increased options, and then this last one, what works, or if you will, evidence-based education.

The reason it is part of the accountability conversation is a very simple one. If you look at federal legislation, since there is a federal role in education, since Brown v. the Board of Education, you will see a transformation over the last decade or so of the nature of federal legislation.

In the early part of the mid-sixties and the seventies, the issue was essentially a federal role that was to create access to education, and to undo the injustice of segregated education in this country. In some ways, the federal role was to level the playing field. So what you saw at that time was the development of programs like Title I, that was directed at categories of kids, that is a category called children in poverty.

You saw other components of the legislation directed toward other categories of kids, migrant children, children who were of Native American origin. You see the categories were categories of children. And so that the federal legislation in many ways kept hitting categorical groups with increasing refinement -- migrant education, bilingual education. All of those are very much characteristics of the recipients of the funds.

Now, in the nineties, and particularly the last bill in 1994, and now the most recent one, if you look deeply into the words in the bill, you will see a subtle transformation. The titles are still there with those same labels. But what they are talking about now are interventions.

Underneath all of this it is not so much about poor kids. It's about good interventions, or effective interventions for poor kids. Class size never used the words kids at all. It was about an intervention. An improvement of reading just talks about K-3 reading, an improvement to reading. So the legislation starts to manifest an intervention type mood.

So what happens? If the accountability requirements go up, then you are going to use, from a policymaker's perspective, intervention with a higher likelihood of success. And where do you find those? You look in the research base.

Now, we could spend hours talking about this research definition, but I want to give you some sense of it. In the legislation the word "scientifically-based" research means regular, systematic, and objective procedures to obtain reliable and valid knowledge relevant to educational activities and programs. And it includes the following: systematic empirical methods that draw on observational experiment, with a standard analysis that is adequate to test the hypotheses.

It relies on measurements of observational methods that provide reliable and valid data. It is evaluated using experimental or quasi-experimental designs, in which individuals, entities, program level activities are assigned to different conditions to evaluate the effects of the condition and choice, with a preference for random assignment experiments or other designs, to the extent that those designs contain the condition across the condition of controls.

This was not written by Campbell and Stanley. This was written by legislators. It insures that experimental studies are present in sufficient detail and authority to allow for replication, or at a minimum offer the opportunity to build systematically. And finally, it's been accepted or reviewed by a higher quality review process. Now, in other words, there is federal legislation that now defines the term "scientifically-based" research, and says that this is the standard that has to be aimed for.

Now, how is it used in the bill? The bill is on the consensus of what we have learned. It is less likely to be subject to fad and fashion. It is a way to accumulate knowledge. And as I mentioned earlier, it is a central part of accountability systems.

In Title I, which is the largest federal program, I just want to give you some illustrations of how it is used. Title I is roughly $10 billion these days. It is slated to get an increase. It reaches approximately 48,000 schools in this country.

School plans have to incorporate scientifically-based research strategies. Technical assistance shall be based on scientifically-based research. Corrective action, including what they will later define as reconstitution, has to include professional development that is tied to scientifically-based research.

It complements school reform. We all know what this is. That's the part of the legislation that funds whole school reform models. It allows schools to get $50,000 for the implementation of research-based, whole school models. The models have to be based on promising and effective practices, and scientifically-based research.

The state applications have to describe how whatever whole school reform would be based on scientifically-based research. Local applications, that is the districts that apply to the state for the CSRD money, also have to describe their plans in research-based terms.

There is a new part of the legislation about teacher quality. It combines what the existing Eisenhower money is with the existing class size money, so this is approximately $3.6 billion. Professional development has to be based on the best available research on teaching and learning.

This one is extremely interesting. There is a tremendous interest in the use of research-based strategies around safe and drug-free schools. The prevention programs are most effective within a complex of scientifically-based research. Now remember that definition that I gave you earlier applies here. It applies to the state plans. It applies to the local plans.

At the other end of the continuum it recognizes while in some areas of education, research has to be done, because the research is relevant. However, to the extent possible, they will engage in dissemination. It means dissemination procedures that have some basis in research. And I think David's subgroup talked about the need for research and dissemination in the adoption of innovations.

On the other extreme you have the new leading first legislation which requires the use of the exploitation if you will, of the profoundly strained research base in reading instruction. But to implement the program, the program has to be based on scientifically-based research. When you select the instruction materials, the instructional materials need to be grounded in scientifically-based research.

The professional development has to be grounded in scientifically-based research. To provide training to reading specialists, even that has to be based on scientific research. We are given a unique definition of scientifically-based research, because the research base is stronger there, they felt they could use an even more rigorous definition of what constituted scientifically based research.

I know you can't read all of this at this speed, but the point of this is to give you some sense of all of the things that it modifies. If you look at the legislation, it modifies instructional methods, strategies, practices, how school should be organized, professional development, teacher preparation, curriculum, materials, technology, remedial programs, services, assessments and screening instruments, and all sorts of people get modified.

Peer reviewers have to look for a research-base. It impacts plans. It impacts technical assistance, curriculum implementation, schools, professions, and supplemental services, materials, reporting, and so on.

We can see from this illustration that there is built into this legislation, a pent up demand for research-based evidence. And I'll just spend a couple of minutes of the beginning of the conversation telling you how we have responded to this in the past.

The first real push on this was complements of school reform. And what we did with the help of some of the people in this audience, we had the reading education lab, and subsequently American Institutes for Research produce catalogs of whole school reform models. And basically, those weren't so much judgmental as they were summaries of characteristics. It said this model has these two points.

They produced an educator's guide to schoolwide reform, and they also produced something called the continuum of effectiveness, in which we tried to draw people to use more rigorous research-based questions when they examined models.

Then when the next act passed, we had stronger tools. We had preventing reading difficulties. But as some of the focus groups mentioned, such documents aren't always as accessible as they might be to the variety of audiences that you want to reach. So a subsequent volume, Starting Out Right, in some ways was a more popular version of Preventing Reading Difficulties. But the existence of those two documents allowed us to tell the story. We know a great deal about early reading. We know a great deal about what people think about early reading.

Then came the National Reading Panel report. And I think people underestimate the profound difference between preventing reading difficulties, and the National Reading Panel report. People focused on the National Reading Panel's methodology. I think that is less important than the distinction being made that what they were looking for were interventions that were ready to put into practice.

It wasn't the knowledge synthesis in the sense of saying what is everything you know about reading? It was never designed to be that. If you look ahead, what it really was trying to say was what does it mean to school districts, to state school officers, principals, and teachers, as well as the classroom implementation. And that's a very different than the synthesis question that was what about preventing reading difficulties?

So in the focus group that I sat in, it was quite clear that the topic has been well covered. OERI has a competition announced to follow-up on the report to do work in reading comprehension in the middle grades.

The legislation is going to hold states accountable, districts accountable, schools accountable for performance in both reading and mathematics. So there is an enormous demand for a similar synthesis in the area of mathematics. The mail is going to come from the chief state school officers, from the districts, from all of the people who are going to be held accountable in the reading scores.

Unless you read the report again as a document, it's rather difficult for the reader to read well. So we put out another book called Put Reading First, which took what the National Reading Panel said was ready for classroom implementation, and gave illustrations of the kinds of practices that were concordant with what the reading panel said was important.

What will all of this mean for educational practice? There is a summary question on a couple of these. For us, it is going to mean we are going to have to work very closely with chief state school officers, district heads, everybody through all of the layers of government to give them some sense of what this means, and how we can go about meeting this requirement of the law.

We're going to have to develop some guidance on these topics. We're going to have to develop some selection criteria that embody them. And then when we go to look at how people are spending the federal money, we are going to have to pay attention to whether or not they are spending it in ways that are consistent with what we know from research.

And all of this is driving to what Assistant Secretary Russ Whitehurst is calling evidence-based education and evidence-based educational practice. We all know that in the last two years, medicine have moved dramatically in this direction, and these are the first steps of education moving in the same direction. And this bill embodies an awful lot of that.

Now what will it do for you? I thought Nel's point was very interesting, in which she started us with, and I think we always have to be reminded of it. There is federal legislation, and federal legislation has certain things. It has certain requirements that have some impact. However, you shouldn't be taking any of this to be the equivalent of academic inquiry or academic scholarship.

And I think it's important to say okay, for the purposes of the law, this is what evidence-based educational research means. And for the purposes of perhaps federal and state investment in education, this is what it means. But I think it would be a terrible tragedy to overlook things like the history of education, case study work, and all of that, because that is all part of an important part of theory formation, which is essential to actually generating hypotheses that can be tested with more rigorous methods.

But I do think that this demand will encourage the development of high quality syntheses, not only syntheses of the knowledge base, but syntheses of the knowledge base that is ready for classroom implementation. I do think you have to pay close attention to the research design features that are built into that language.

It is quite clear that experimental designs are preferred by the legislature, that where possible, random assignment to insure pretreatment equivalence. It is incredibly important to measure the effectiveness of particular interventions. So I think research design will be an important part of any synthesis activity that is underway.

Also, we have to think about the evaluation designs. Can we design evaluations that implicitly or explicitly correlate to experimental? I think it should impact the kind of research training that goes on, as David mentioned, in schools of education for teachers. But also in the social science fields.

When we went to graduate school there was a fair amount of quantitative training. If you look at the curriculum now for a lot of the social sciences, a lot of that quantitative part have fallen away.

There is the clearinghouse function that Barry or someone else might want to elaborate on.

Then there is another part that I think we have to pay attention to, is the collection of practical wisdom or effective practice from people who are actually doing it. One of the things that we learned over the years is the federal role in knowledge utilization and dissemination was that we haven't exploited places that are actually having success, and use those as knowledge generators.

We have always taken the view that somehow a centralized center knows best. In many ways, I have come believe that part of the federal role is a need for the systematic gathering of this sort of practical wisdom of what people are doing.

And I think finally, that it's going to have profound impacts on teacher training and craft knowledge in the area of reading. And the first goal, states are allowed to examine the curriculum of teacher training programs.

That's the end of the formal part of the talk. I would be happy to entertain any questions or comments or suggestions. And Cathy is first to the mike.

Agenda Item: Q & A

PARTICIPANT: My question is how did you say that the definition for the Reading First and Early Reading First was more rigorous? When I looked at the two definitions, they looked pretty similar, except that the part about experimental and quasi-experimental and building knowledge don't appear in the Reading First part.

DR. CONATY: You may be right, Cathy. I may have misspoken. The definition that applies to everything is more rigorous than the definition that applies to reading, even though the reading has the stronger knowledge base.

The last time I saw the legislation, the definition that applied to professional development and technical assistance, and so on was a much more general definition. I think what happened in the last few days of the crafting of the legislation, was as the requirements for the accountability system were clarified, and the requirements of the expectations for adequate, really progress were clarified, I think people understood the central role that effective practice was going to have to play if people were going to meet the accountability expectations.

I think it was deliberately made as rigorous as possible to again, increase the likelihood that people will ask the right kinds of questions, and address them with rigorous enough methodology.

We briefed the chief state school officers this morning -- not brief them. We met with them in a meeting that was hosted by Sec. Page, and the assistant secretaries. And while the chiefs expressed an enormous support for evidence-based intervention strategies, they saw a large federal role in doing what they said they couldn't do themselves. That is the synthesis function.

They also a federal role in things like developing technology-based assessment. But one of the clear things that they felt that we could do was help put an FDA-type approval on practice where in fact there was credible evidence. But they also expressed some impatience. Don't just leave us hanging out here with nothing. Make some judgments, even if the understanding is these judgments may change as more evidence rolls in.

The conversation about this has gotten much more sophisticated in the last half dozen years, and particularly in the last three or four. All of the states when through the Reading Excellence Act, all of the states have gone through the CSRD process. And it has made them quite sophisticated about research, research technology, research design, and so on, at least in the vocabulary sense.

DR. MOSHER: Well, I hope you've got a good bank, or at least the Social Security funds that you are supposed to put our money in, because if you had to meet any reasonable criterion of what research-base means, you are going to have a lot of money you can't spend, or you are going to have to put it all in early reading.

And if you do spend it, it's apropos of the conversation you had with the chiefs. If you do spend it, you are likely to devalue the meaning of scientifically-based research, which is a little bit redundant. Does that worry you?

DR. CONATY: It worries all of us quite profoundly. I think, Fritz, there are some areas where we more than we do others. I do think that David started to enumerate a few homework policy, retention policy, parental involvement, perhaps school change. There are probably a dozen areas that are quite ripe -- safe schools research, violence prevention research, peer group influence on adolescent behavior. There are large bodies of literature withstand some really close scrutiny.

I think, Fritz, what we have to do is we have to create a mindset of inquiry among people who are implementing the law. I think I would feel comfortable if in the first go round we were confident that people were asking the right sorts of questions of people who were advocating particular interventions.

If they started to ask under what conditions? For what kids? What measures did you use to claim that it was efficacious? We may move along that path. And that's why I was saying there is going to be a tremendous demand. The field probably isn't ready to meet that demand, either as a field or as an enterprise. But I do think that that demand will in fact generate more demand on the supply side. So I think all the time people are going to say, hey, how come you don't know more about this?

PARTICIPANT: [Question off mike.]

DR. CONATY: I think Prof. Whitehurst has a better answer, Fritz. I think we are going to try and talk to them about designing their implementation in ways that are knowledge-generating. So in other words, let's say you're going to start to use curriculum A and curriculum B. Let's not just use curriculum A. Let's in fact use curriculum A and curriculum B. To the extent that is practical, randomly assign them. And then use the intervention itself to generate the evidence and knowledge base. And that's what I implied as a different sense of evaluation.

In the Reading Excellence context, I had gotten one of the states to agree to randomly assign the money to schools. That is before the governor heard about it. He wasn't too keen on random assignment of funds to schools, so they didn't do it. But I think that you will see some movement toward that, and that's why I think the implementation itself has to be a knowledge and learning activity. People's vocabulary aren't totally exact on this, but that's the logic that has to work.

And then I think the issue of whose endorsement. Whose knowledge is this anyway? Who supports it? A lot of this you will see the sense of independent evaluation by non-developers. What the chiefs wanted in part was armament against product peddlers. That they felt extremely pressured. And what they wanted was some protection from that pressure, some set of questions, some set of evidence that they could ask when people came to them pushing a product.

So I do think the sense of the Academy of Education, Michael's sense of the OERI working together. What we never did, Fritz, was make judgments. I never stood up in front of 2,000 people and said, I know this reading program is better than that one. That was never seen as a federal role. I think that the events here will transform that role, and transform that legitimacy.

I do think there is a danger. The danger is it would be difficult to gain a reputation for soundness, and easy to lose it. So I would encourage agencies to err on the side of cautions about claims, because in fact maybe we don't know quite as much about fluency as everybody thinks we do. Maybe this intervention, because it doesn't pay much attention to fluency, won't produce the kind of results.

It's easy to lose a reputation, and once it's gone, you can't rebuilt it. So I would urge some caution about overclaiming initially when the agencies get together to do these things.

Please, David.

PARTICIPANT: I have a question, and it's related to some colleagues' concerns, not just my own, which is when you do treatment studies, treatment A, treatment B, the general rule is one doesn't swamp the other. That is if you look at subunits, there are classes of treatment A that outperform other classes in treatment B, and vice versa, and the means by differ significantly.

In other words, there are practitioners very often who are excellent at whole language, and others who are lousy. There are excellent phonics people, and others who are lousy. And because teaching is such a human endeavor, you get this mix of teacher and manner with curriculum, et cetera.

When you find that a mean different exists, so that treatment A is better than treatment B, but there are teachers who use treatment B efficaciously, are you going to order them to stop doing what they do well?

DR. CONATY: I could make a joke and just say, no, I'll be retired by that time. But I know you are serious, David, and this is a dramatic and important issue.

In some ways, I think the answer you will get when you raise that question now will center on the tools that teachers use. You remember well there was a time when there was a lot of emphasis on developing teacher-proof tools. I think there is a lot of attention these days to the development of curriculum and materials, and to the extent possible embodying the scientific findings in those tools and materials.

So that a high quality and a whole language teacher using these tools, you know I think it's without dispute that a big problem in high poverty, low reading performance is from poor teacher knowledge. That tool, that part of the tool, that part of intelligence can be built into the tool in a high quality. The whole language teacher using those tools will cover that topic as well.

So I think to a certain extent it is going to be teacher knowledge and teacher tools. And I'm using the word tools to be the most generic term I can think of. So I think that the science can be both embodied in the teacher knowledge, and in the tools they use to conduct their craft. So nobody is going to say no, she shouldn't be -- which literature, but they will say, she should pay attention. If that kid isn't paying attention to sounds and words, because we know those children are at a higher likelihood of risk.

The other part, David, is that it is quite serious, this business of No Child Left Behind. Taking again the reading component of the legislation as an illustration. It talks about the use of diagnostic assessments. Let me do it in the right order.

It talks about the use of screening instruments to say let's give everybody a quick screen to see how they are doing. Then for children that are flagged by that screen, let's do a diagnostic to see what their issues are, and let's design an educational intervention that is suitable for that individual child issue.

Given that, now there is much more demand on teacher knowledge and teacher skill and teacher tools. And that's why I said earlier it is much less about categories of kids than it is about the nature of the effective intervention.

PARTICIPANT: [Question off mike.]

DR. CONATY: For children instructed in that school and by that teacher, nobody will say you have to change your practice. Because the bottom line in all of this is performance on the accountability measures.

Now, we can go onto the dimension of those measures, and all of that, and the adequate of those as a measure. But you and I both know from having done this so long that teachers hit plateaus. Even the best teacher hits a plateau where she is successful let's say with 70 percent of the kids. This is not going to be an acceptable an outcome anymore.

This is going to say, you have to pay attention to that other 30 percent, or that other 20 percent, or that other 5 percent. And so that the standard of what constitutes a high performing teacher is going to be driven up by the logic that is embedded in this federal law, given all of the constraints. It's only the federal law. It's not the state law, and so on.

PARTICIPANT: It seems to me that the implications of this could be a sea change for, to use your term, the product peddlers. And I was surprised when I first started attending teacher meetings, that textbook manufacturers will talk about the number of color pictures or the free CD-ROMs or the great overheads; everything except whether they have any evidence about whether kids learn from those materials. So I would be interested to hear you expand on that.

DR. CONATY: This is really a state issue, and it is being driven by the states that have statewide adoption policies. Probably the biggest impact that there will be in the next two years on reading materials is the decision California is about to make in its adoption policies.

What materials will be on California's approved list of reading and curriculum materials? Given the size of California, given the size of that market, given the number of smaller states that follow California's lead, I think that that sort of decision is very, very critical to what you describe.

And I think what's going to happen is these states have gotten quite sophisticated about things like evidence-based education, effective materials, what do we know about reading, what should be in those materials. Now will that lead to a narrowing of the kinds of literature children read? Probably not. Will it lead to a narrowing of the kinds of basil readers they will use? Perhaps.

DR. NODDINGS: The model you describe is in many ways so attractive. When you talking about screening and diagnosis, and intervention based on it. But it is nothing like reality. We have just never seen this. How are you going to get people to do it? You know as well as I do that is operating in most places, in most states now is that there is a standard that has something to do with the mean of reading.

And if you get enough kids up to that, you're off the hook, so to speak. So it will have exactly the opposite effect to what you were talking about. It's a triage sort of thing, where you can forget a certain percentage of the kids, as long as you get up to whatever that standard is for the reading. How are you going to handle that?

DR. CONATY: I'm going to answer that in two ways. I don't have it on a slide, but there is a component of the legislation that describes the reporting requirements for the assessment. And it sets what it constitutes adequate yearly progress.

The subcategory of reporting requirements includes social-economic status. I want to first of all say this isn't multi-variate. I'm going to give you sets of independent reports, so we don't have to generate all of the cells on this multiplicity.

But socio-economic status, English language proficiency, reading, math, disability. And you have to report by all of those subcategories. So you have to report by race, ethnicity. You have to report by disability. And you have to make adequate progress in all of those subcategories.

Now I don't mean that you have to report black, disabled females all at once. No. You have to report them separately within the limits of individual privacy. And therefore, you won't be able to write off pieces of the performance. They will have to make adequate yearly progress in each of those cells, or be driven toward school improvement.

And school improvement is not a punishment system at first. It is a system where the state is required to spend additional resource to pull those schools along. So that's how the accountability reporting part is to work.

The other part is for some of these activities, like the reading, that does require those early assessments. States will have to describe to us how they are going to monitor the implementation of these activities in districts and schools. We won't do the monitoring, but the states and districts are supposed to.

We know from our experience with some of the Reading Excellence and CSRD -- CSRD, probably a third of the schools, absolutely nothing happened. They adopted -- and the RAND report is pretty clear on that -- they adopted the name of a program, but no classroom practice. No behavior changed. They just said they were doing this, when in fact they weren't even doing it.

But there was a continuum. I won't describe it as a normal curve of implementation, but there was more or less effective implementation. And as time went by, more coherent implementation. The same is true in the Reading Excellence Act, given what we know. Some of the states really have started to change classroom practice.

Others took the money and did what they were always doing, and supported their activities and very little changed. But we have new monitoring and reporting requirements that will be imposed on the states for them to pay attention to it. Whether or not it will work, you know.

DR. NODDINGS: [Questions off mike.]

DR. CONATY: That ongoing coaching by specialists, where they were regularly in the school on an almost continuous basis was the most effective changer of the teachers' instructional practice.

But also it goes to the point about the materials. With increasing frequencies, the textbook market and publishers are also selling professional development around the use of their materials as part of a package. I'm just reporting that as a trend. It could be a very expensive training process. Given some of the rates I've seen them charge schools, it's quite high. And I don't know the efficacy of those.

DR. FEUER: You have obviously stimulated a lot of very interesting late afternoon conversation.

DR. CONATY: The legislation did it, not me.

DR. FEUER: If could go back to David's point. As I understand this, and I have a feeling David and I will continue to talk about this later, tomorrow perhaps, the problem here is that there is a risk that a policy that insists on evidence as the criterion for implementation will crowd out the more intuitive, or the more individualistic and successful efforts that some teachers have used and continue to use with their kids.

In other words, as David put it, that there will be teachers who do not use a method that seems to sort of follow from the evidence, but has good effects. My question is to what extent do you think that that is different in education than it is in almost any other endeavor in which treatments that are sometimes unconventional work for some people, and yet there seems to be a demand for some kind of federal or public knowledge base that guides people as to what the most likely effective interventions area?

In other words, to what extent is this any different from the physician who has some good luck treating back injuries with manipulation, even though the preponderance of evidence suggests that in certain cases surgery is called for. And I'm just taking something off the top of my head.

DR. CONATY: That's David's question to answer. In some ways, if you think about it in the aggregate, that is going to be a phenomenon at the margins. It is just not going to happen very often. The thing that we know from studying instruction is the worst thing for a child is incoherent jumping from topic to topic. And an unstructured instructional practice is the worst sort of outcome.

And what has happened to a certain extent is attempts to be balanced have created such heterogeneous strategies for instructional practice that in fact they doom a kid to failure. So I took David's question to mean somebody who had a coherent, systematic instructional practice as sort of what would happen to that person, not somebody who I assume that they have behind that, some instructional strategies that they are using. It's not just a gift.

PARTICIPANT: Two quick things. In the area of technology, the ESEA bill has a different term that is used again, the review of relevant research. And that was recognition of Congress, with some work done by myself and others. That if part of the legislation that is encouraging the use of innovative technology, waiting for control group studies to come around on a particular new technology is really going to be restricted to the kinds of technologies schools could use. So we ended up with a looser definition, maybe a little too loose. But again, considering the alternative, it was the best we could get.

The second thing is while again, scientifically-based research is pushing pretty hard for control group studies, there is another provision in the ESEA law which makes it harder for researchers to do that. The grantee part provision for parental rights has a whole new level of challenges for researchers to go into schools, and to work with classrooms and students in controlled group studies. That's another provision in there that is going to have some far reaching ramifications.

DR. CONATY: I agree it has ramifications. I don't agree with your assumption that it's going to make it more difficult. Institutional review boards, human subjects approvals, a lot of that has existed for many years.

PARTICIPANT: I'm very glad to see this legislation, but I haven't been able to read it. What parts of this is prospective in terms of teacher preparation? How is teacher preparation being looked at in terms of research-based?

DR. CONATY: The few times it appears, it is largely an exhortation to do it better. The professional development funds are really post-graduation funds. There is a little bit in the reading section that I mentioned that talks explicitly about teacher training.

I think if a school were clever, or a district were clever, they could figure out -- this is not a legal opinion, this is just an idea from somebody who has read the law. It might be possible to have the last full year of teaching practice, the practicuum part be covered by some of the professional development funds, because the payment of teacher salaries is an approved activity.

So we could get an implementation part between a clever district and a clever institution of higher education, where they could develop a partnership that allowed a teacher to participate in research at the university, at the same time they were in their last year of trying to implement it in a school.

We have a program for the improvement of American history education, where the funding is going from $50 million to $100 million. And that program has very much structured into it, that sort of partnership model to improve teacher knowledge of American history before they enter the classroom.

Just so you know, if you go to the department's Web page at ed.gov, there is a summary document that was prepared. It's probably 100 or so pages that summarized each of the major features of the law, under each of the titles. And the real law is 1,000 plus, right? So a 100 page summary is not bad.

And it summarizes the major changes to each of the Titles in the legislation. And I would encourage you to look at the area that you are interested in, because the pages each is about a page or two.

DR. HAKUTA: Do you have the ability to put that slide up again, the one that defines scientifically-based research?

DR. CONATY: Yes, it runs across two now, Kenji.

DR. HAKUTA: There is one that has the definition written out. I was just wondering, as an experimental psychologist as my background, and therefore seeing the word "experiment" there, there are different kinds of experimental assignments you can make. One of which is a very popular design used by cognitive psychologists or psycholingists with is experimental assignment of let's say items or treatments within an individual school. You don't really have the students assigned to a treatment.

I wondered if this definition -- "individual entities, programs, or activities are assigned to different conditions." So you could actually have under experimental research, it could be research that let's say compares nouns versus adjectives within an individual student. It doesn't have to involve the kind of logistical difficulties of having to assign stuff to students into different types of programs. Would that be a reading of that?

DR. CONATY: Let me not answer that on the fly, Kenji. Let me think about that. The basic principle here is one to measure the value-added by the intervention. The clearest way to do that is to have large groups randomly assigned, an intervention that differs by only a fraction from the alternative.

I think that the conceptual difficult, Kenji, in this is not the where you are heading. I think the conceptual difficulty of the transfer of the medical model to education is the notion of a treatment. In medicine, the treatment or the intervention is pretty clearly well circumscribed event of set of activities, circumscribed both by time and by delivery method.

That is less true in education than in almost any other enterprise. So I think when people pay attention to his definition, you can talk about sort of a medical model, but it's important to pay attention to the difference in what consists a treatment in medicine, and what constitutes a treatment or an intervention in education.

And I think that's why some of these even well designed studies, people are surprised by the results. In fact, it is because the nature of the treatment is so poorly understood.

PARTICIPANT: And that's one of the places where qualitative research becomes essential. Because if you don't know about the treatment, or it's very crazy across 17 different instantiations, you don't know what to interpret the effects information. I was in a meeting about six months ago and somebody said, and you also need to pay the same close attention to the control group for the very same reason.

So I am at a loss to see why qualitative research is being marginalized by the edicts of OERI, when that kind of documented implementation is absolutely essentially to do what you say you want to do.

DR. CONATY: Remember that the federal legislation, although it has changed, it is designed to impact children in poverty. So I think in addition to the treatment issue, there is the issue of compositional issues in the sampling. But I think as we get more sophisticated about these questions, I think we will develop better answers.

And again, this is not to narrow the scope of inquiry. I was just trying to describe what I think the consequences of the legislation are going to be.

PARTICIPANT: With the emphasis on experimental and control groups, I'm assuming that that does not leave out the powerful design of single subjects research. I'm reminded of the research that is done at the University of Kansas over the last number of years on the strategies intervention model, where primarily, if I understand correctly, they have relied heavily upon single subject design. And how do you see that fitting into this legislation?

DR. CONATY: I must be candid with you, the National Reading Panel essentially excluded single subjects design from its review of the literature. It set a set criteria for which studies were included, and where were excluded. And I don't have the detailed list of those criteria on me, but my sense is that they excluded those.

And I think you should look to that National Reading Panel report as a model, maybe not the last model, but as a model of the kind of synthesis that people envisioned when they talk about research practice. Again, when they are trying to generate knowledge, what they are trying to do is say what knowledge is ready for implementation in the classroom, and that for which we have sufficient evidence to reassure the public that that's a wise investment of funds.

That's a different question than understanding. That's a different question than explanation. That's a different question than history. As stewards of public funds, our stewardship involves trying to maximize the returns on these investments by producing the best student achievement we can for the most children, regardless of their circumstances.

And in that circumstance then, not all studies will be created equal. I just think that we have to face that as a fact. That doesn't mean that a study is any less valuable for other purposes, or won't in fact have a profound impact on the standard of some problem. In terms of this sort of investment notion that is implicit in this, I don't think that all studies are, or would be considered equal.

DR. FEUER: It's been great. Thank you so much.

I think we are adjourned. Thank me in helping Lisa in particular for organizing this meeting.

[Whereupon, the meeting was adjourned at 5:05 pm]