THE NATIONAL RESEARCH COUNCIL

AND

THE NATIONAL ACADEMY OF EDUCATION

WORKSHOP ON

SCIENTIFIC INQUIRY IN EDUCATION:

IMPLICATIONS & NEXT STEPS

January 10, 2002

National Academy of Sciences Auditorium

2101 Constitution Avenue, NW

Washington, DC

Proceedings By:

CASET Associates, Ltd.

10201 Lee Highway, Suite 160

Fairfax, VA 22030

(703) 352-0091

TABLE OF CONTENTS

Page

Welcome & Overview:

Dr. Feuer, NRC 1

Dr. Noddings, NAE 2

NRC Committee on Scientific Principles in Education

Research, Findings and Conclusions of Scientific

Inquiry in Education:

Ms. Towne, NRC 16

Dr. DeHaan, Emory 22

Dr. Boruch, University of Pennsylvania 30

Dr. Fletcher, University of Texas, Houston 36

Discussants:

Dr. Apling, CRS 45

Dr. Graham, Harvard University 52

Dr. Grissmer, RAND 65

Q & A 78

Report Out From Focus Group Moderators:

Research - Dr. Noddings 124

Federal Policy - Dr. Hakuta 127

Associations - Dr. Berliner 131

Schools - Dr. Feuer 136

Invited Closing Remarks - Dr. Conaty 143

Q & A 155

P R O C E E D I N G S (8:25 am)

Agenda Item: Welcome & Overview - Michael Feuer, Director, Center for Education, National Research Council

DR. FEUER: Welcome to the National Academy of Sciences, to our cozy little Academy. I'm very grateful to all of you for joining us today. This is, as all of you know, a big week in education policy in America. The president, the day before yesterday, signed a landmark legislation called No Child Left Behind. And we are meeting already today, just 48 hours later, to discuss an issue which, as most of you probably know, is in some ways quite central to the current wave of thinking about education policy and reform in America.

I understand -- I have not read the entire law line by line yet, but I understand the phrase, "scientifically-based research" appears over 100 times. And so a report on scientific inquiry in education is indeed timely. And I'm very gratified that with the foresight with some of our friends in the federal government, and with the hard work of our volunteers and staff, we have been able to produce a report that I think is really in many ways, significant to the current debate.

This morning I just doing to say a few brief words, and then I mainly want to introduce Nel Noddings, the president of the National Academy of Education, who is here with me. And then I will say a few words about the NRC, and about what we are doing today.

But again, let me start by thanking in particular, Kenji Hakuta, who, in his capacity as chair of the policy board, had the foresight to encourage the NRC to take on this kind of study. As he knows, the time between the policy board approved the study, and when we released it, the policy board itself didn't meet. So we were very proud to be able to provide meeting-to-meeting service with our report. This one was about something like 14 months in the making. It took the work of a number of distinguished scholars and superb staff. I will mention their names in a moment.

So with that, let me subside, and I want to turn this over to our special guest, Nel Noddings.

Agenda Item: Welcome & Overview - Nel Noddings, President, National Academy of Education

DR. NODDINGS: Well, thank you, Michael. I am happy to add my welcome to his, and say a little bit about what the National Academy is, and what we are hoping to accomplish.

The National Academy is now almost 40 years old. It was established to promote inquiry. And in the original document it said, "inquiry into the means and ends of education in all its form in the US and abroad." And I think that holds true. We are interested in education in schools of course, but education takes place in places other than schools, and we are well aware of that.

Some of the things that the National Academy has been involved in are well, first of all, I guess the thing we are best known for is the post-doctoral fellowship program that we administer with the generous funding of the Spencer Foundation. Now this is an important way really of promoting educational research, because we identify and nurture -- the word that we prefer to use -- young scholars. People who are particularly talented in doing work associated with educational research. So we have been doing that post-doctoral program for about 16 years now.

We also run some projects. That is, we sponsor or co-sponsor some projects. And we have a major project going now in teacher education. And our question is, how can we better fulfill this original mission, which is to promote scholarly inquiry in education? At this stage we are very much looking forward to collaboration with other academies, with the National Research Council, and with practitioners and policymakers.

I want to point out a sort of systemic problem that we have in research. People sometimes look at educational research and they say, this is terrible stuff. Educational research is bad. Well, there is bad educational research. There is bad research I think in almost every line of scientific enterprise.

There is also some very good research. But, if we are going to make it better, it requires something other than just concentrating on educational research as a separate enterprise. That's one of the reasons that we are so excited about collaboration, because the connection among practitioners, policymakers, and researchers is extremely important. It influences the whole enterprise.

Now let me give you an example of the kind of thing that I'm talking about. It is sometimes said, and you hear this complaint a lot, that practitioners don't read educational research, they don't know educational research, they don't apply educational research. Well, that may be true. It's at least partly true.

But on the opposite side of the picture, there is an even more troubling problem. And that is that policymakers often seize upon research before that research is ready to be used. That too, is an extremely important problem. And we can see now how all of these things are connected, because if researchers are going to take their time and do a really thorough job before releasing results, it means a sort of different way of funding research. It's trying to ease up that pressure for immediate results and immediate application. Take the time to really look at things.

Let me finish up here by just using a medical analogy. When we do drug tests, it's wonderful to find out that a particular drug has an important effect on a particular condition. But see, that's not enough to know that. We then also have to know whether this drug has other kinds of effects on particular people -- on the very young, the very old, pregnant women, people with other ailments, people who are taking other medications. And sometimes we only find out about these things through tragedies.

Now in education we are not facing the same kind of physical tragedies, but we are facing exactly the same kind of problem. That is, hasty implement of results that are not as solid as we would like them to be, can be a very great mistake. So envisioning something for the long-term, what I would like to see is long-term funding for projects that would involve monitoring and evaluating and revising and changing and reporting. And meetings of this sort, where we have an opportunity to talk with one another, and correct one another's error.

So I very much appreciate your being here, and I hope we have a very productive day.

Thanks.

DR. FEUER: Thank you, Nel. It's exactly the kind of background that helps me launch into three minutes worth about this Academy, that is the National Academy of Sciences, where you are sitting this morning. The story of this institution is really quite germane to the purpose of today's meeting. So if you will indulge me in two minutes worth of this, I promise not to tell all of the same jokes that I tell about the history of the Academy, so that some of you will keep coming back.

This was an institution chartered in 1863, by an act of Congress, and articles of incorporation, the charter was signed quite enthusiastically by President Abraham Lincoln. This was an innovation in its time. It was a variation on a concept that already had existed in at least some other countries such as the Swedish Royal Academy of Sciences, which has been established in 1739, I believe.

Lincoln's innovation was in some ways, a reflection of the peculiar genius of American democracy. And that is that in granting a charter to the National Academy of Sciences, he attached a proviso that required public service. In essence, a requirement that on request, this Academy would advise the federal government on matters of science and technology and public policy. And furthermore, do it pro bono.

And since then, this has been a somewhat radical experiment, I think, in the application of scientific thinking, and in the scientific enterprise to the betterment of public policy that is indeed the mission.

Now another very interesting variation that was reflected in the charter of the NAS was that from the beginning, the behavioral and social sciences were included. And this was again, a departure from the European tradition. So anthropology and psychology were really part of this enterprise from the get-go in a way that in the European academies, the separation was much more stark between the natural sciences and the behavioral and social sciences.

There are some very interesting archives and histories of the work that the Academy did in the early days. Some of it was on aspects of human behavior -- training in the workplace, the psychology of sexual behavior. Of course in the beginning, a lot of the work was what we call sort of more kick the tires kind of technology and analysis.

And in fact, one of the first -- not the first -- very interesting problems that this Academy had to solve had to do with an issue in the Civil War, which had to do with the ironclad ship. Which, as you may recall, was an invention that ultimately enabled the North to actually win the war. The problem with the ironclad ship, however, was that they couldn't get the compass to work because of the magnetic field. And so this was one of the first projects turned over to this young Academy.

And as you can imagine, it was a quite example of the intersection of technology and public policy, because in a situation such as the Civil War, you do want to get north and south right. And getting the compass to work on the ironclad ship was no trivial matter of public policy. I am happy to report that the Academy's results on that study are now cleared through review, and we will be releasing them shortly.

Education became a focal point of the Academy, and more specifically of the National Research Council when it was established by executive order in 1918, or right in that period. This was during the days of the first world war. And President Wilson came to the realization that the Academy had provided some extraordinary support for the federal government, and encouraged an expansion of the franchise to include people from business, and experts who were not elected to the honorific society of the Academy.

Today, this enterprise, the National Research Council and the National Academies produces a report just about every working day. Last year we had 230 reports released from the National Academies. We work with a group of anywhere from 7,000-10,000 volunteers who serve on our boards and committees, many of you, many of these people, and a staff of about 1,000, who are altogether quite committed to the idea of science and the betterment of public policy.

It is in the last eight years, perhaps most powerfully manifest under the leadership of Bruce Alberts, the president of the National Academy of Sciences. We have released just in the last eight years, over 150 reports just having to do with education. And this reflects Dr. Albert's abiding passion and commitment for the application of science, and the humanitarian and honest and ethical and rational ethos of science to the improvement of education.

So this report on scientific inquiry in education is really the result of questions that were asked about the basis of scientific method as it applies to education research. And as I indicated, it was a request from the Office of Educational Research and Improvement Policy Board, the NERPPB, National Education Research Policy and Priorities Board, that we undertake a kind of inquiry into the scientific basis of education research, or more precisely, aspects of scientific education research. That's a distinction that I hope comes across in the report, and that I think my colleagues here today will have a chance to say more about.

So this is a great pleasure to able to go more public with this report, which was just released about a month ago, and to have a group of folks who are from the worlds of research and policy and practice to explore some of the issues in this report, and to think about next steps.

I want to get right into our program now, and first say a few words of thanks to some people who are not here. First of all, to Rich Shavelson, a great friend of the NRC, and a great scholar of education and public policy, who, in another weak moment, answered a call from me and agreed to chair yet another NRC panel. And this was a daunting challenge, because of how little time we had, and how complicated and tough the questions were.

Rich couldn't be here with us today, but as I told him on the phone yesterday, he is certainly with us here in spirit, and his footprint is quite clear in the report, and in a lot of what you will hear today.

I also want to thank in addition to Nel, who, working with the National Academy of Education, has agreed to try to collaborate with us on some of this. Kerith Gardner, who has been so ably helping us with staff work on this. And then people on our staff, Tina Winters, and Teresa Williams, and Linda DePugh, who were an extraordinary team putting together this event today. Just managing the logistics of this is no small thing. We are deeply indebted to them for their hard work.

Adam Burns, who helped us sort of in the final hours, set up this afternoon's program. You will see later on what an impressive bit of work that is. So we thank Adam.

And finally, let me just introduce the person who really was -- well, first let me introduce the panel for today, and then at the end of that I will introduce Lisa. This report that is the basis of today's conversations, like all NRC reports, was written by a committee of scientists and experts who worked as volunteers on this effort. And some of them are here with us to share the wit and wisdom of this report: Bob Boruch from the University of Pennsylvania; Robert DeHaan, who is actually now on the staff of the National Research Council, but was on this committee in most recent prior capacity as a professor at Emory University; and Jack Fletcher of the University of Texas are here on the panel, and we're very grateful to them.

Unfortunately, Ellen Lagemann could not be with us today. I'm sorry to tell you that Ellen lost her mother yesterday. We extend our condolences to Ellen, and wish her long life and good luck.

And finally, let me introduce Lisa Towne, who was the study director for this project. And if any of you are wondering how you get a committee of experts and scientists to agree both to the concept of writing a report like this by consensus, and then actually doing all of the work, it's because of the glue provided by the staff. In Lisa's case this was an extraordinary actually first effort here at the NRC for her. And so it's a great pleasure to introduce her. Lisa will tell you more about the logistics of today, and other things.

MS. TOWNE: Thank you. Hi, everybody. Welcome again. I'm going to just briefly walk through a couple of the key pieces of our agenda, just so we are square on the program before we get started. And then I'm going to start off the presentation that my esteemed subpanel of the committee that actually authored this report will then deliver.

So let me start just briefly with the agenda. You probably have looked through your packet of materials already, and just want to make a couple of key points that I wanted to make sure that you saw. First of all, we are recording this event. That's why we have got microphones. We will making a transcript of it, and posting it on the Web. That's just something that we like people to know in advance, so they temper their remarks -- no, you can say whatever you want, but it will be public.

And that also means that when we have question and answer sessions at various points in the agenda, if you could come up to one of the microphones that are in the audience and state your name, that would help the transcription process tremendously.

Just a few things on where places are in terms of various agenda items. You will see that there is a map in your packet for some of the room locations for later this afternoon. You should familiarize yourself with that. We'll also be trying to help direct you, and have arrows on walls and such to help you make your way through what is somewhat of a labyrinth back here.

[Administrative remarks.]

That is about it. What I wanted to spend a few minutes on in terms of logistics is the afternoon program that Michael mentioned. This is something a little different that we are trying out at the Academy. We are actually putting together some focus groups. There was a sheet in your packet that was called, "Focus Group Methods and Instructions." It has a little bit of information on this. I just want to go over this briefly, because it is important. You all individually play an important role in this.

We are setting up these very small groups to try and gather information about how you think the National Academy of Education and the National Research Council could collaborate and continue work on this important topic of discussion today, and have a specific idea that we actually want to run by you and get your input on.

So we have structured these focus groups to be small group discussions. We had to do some selection, some not so random sampling. It's actually very purposefully selected. You will notice that some of you have a colored dot on your name tag. If you have a colored dot on your name tag, that means that through selection criteria according to your background, and the size of the group that you are most like, we made decisions about putting you in these groups.

If you have a C also on the back on your agenda, there is a categorization by color of what group we have put you in if you do in fact have this dot. If have a dot, we have then put you in a focus group, and we are expecting you to attend. So if you actually were not planning on behind here in the afternoon, and you can't be in this focus group, if you could let someone at the registration desk know that, so we can make any last minute changes to get these groups to be a good size and representation. That would help us tremendously.

If you do not have a dot on your name tag, you are most likely someone from the Department of Education. We have a lot of you here today. We're very pleased you are here with us today, but we had more people here than we could accommodate in these small groups, so what we have done is in each one of these rooms is put together some chairs where people can sit around and listen in on the discussion. You are more than welcome to do, and to sit in on any group that you choose to.

If you do want to provide formal input to us, you are more than welcome to do that in writing at the end of the day, and leave that with us. So we thank you for your help in trying to pull these off, and to let us know if you can't be here, or if you think your categorization is off. Many of you could fall into a lot of different categories, and we do have some space constraints, but if you think you have been grossly miscategorized, you can also let someone at the registration desk know, and we'll try to do some midcourse shuffling.

I think that's it on the focus groups for now. With that, why don't we go ahead and get to the meat of all of this. With this first slide I just want to say that I am aware that it is 2002, but I didn't actually notice that until we just got up here this afternoon.

What I'm going to do is just start off this presentation, and then my colleagues will pick up were we left off. I'm going to be basically covering the material that Ellen would have covered, had she been here. So I will do my best to fill those very big shoes.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Lisa Towne, National Research Council, for Ellen Condliffe Lagemann, Spencer Foundation and New York University

MS. TOWNE: I'll just give you a brief bit of background that Michael has already given you. This is sponsored by the NERPPB, the National Educational Research Policy and Priority Board, and our work began in the fall of 2000. In fact, the committee first met in December 2000. So the report was produced in less than a year's time, which by NRC standards is lightning speed.

The other thing I will mention about this particular slide is this is sponsored not only by a bill that is pending in Congress to reauthorize OERI, that many of you are aware of. This bill in part defines what scientifically-based educational research is, as does the new No Child Left Behind legislation, which prompted a discussion that actually has been going on in the educational research community, and related communities for a very long time, about 100 years. So this is not new, it had just kind of a new policy spin to it when we took this up.

Dovetailing from that, there are a couple of goals that we hope this report will serve. One is in part, to inform the pending reauthorization of OERI. There is a chapter in the report that takes up specifically the implications of the committee's work for the federal research agency in education, and Jack is going to talk more about that later.

We also hope that it informs more generally, this ongoing trend, as Michael mentioned, for scientifically-based education research. This has been a priority of the Bush administration since they have come to town. And we know that it is something that they want to take very seriously, and we hope that this report is a timely contribution to helping make that a reality.

And finally, we hope this report starts a bit of self-reflection within the field of education research and related communities. It's certainly noteworthy that the Academy was even asked to write this report, and related that the legislative body tried to define what education research was in a political arena. So we hope that this report, while obviously not the last word, can help spark some debate and some self-reflection about what that means, and how to move forward from here.

This is probably not readable. I won't go through the committee membership in much detail. I just want to point out that most NRC committees, it's also in your report, so you can actually read it in 12.5 font if you would like to do that.

I would just point out the diverse perspectives that we had on this panel. We have people who are historians, anthropologists, psychologists, sociologists. We have people who are experts in statistics, economics, philosophy. Bob DeHaan actually is a cell biologist by training. We also had a research chemist on this panel. So as you imagine, putting 16 people of these different walks of life into a room and asked them to tackle these questions was quite an interesting and challenging thing we did.

I'll just give you a little bit more specifics about our charge. We were specifically asked to consider the nature of scientific education research. And the word "science" here is important. We weren't asked to talk about education research or scholarship writ large, which is obviously a larger body of work. We were asked to talk about science, and given the fact that this was a panel of the National Academy of Sciences, that made sense to us. And in particular, as I said, how a federal agency could take these general ideas and build them into their own work.

An important part about the way this committee went about their job is that we did not, in addressing these questions, evaluate existing literature in education research for its quality. We didn't evaluate existing research areas, I'm sure a lot of you in the room would be happy to know. And we didn't evaluate comprehensively, the existing chief federal education research agency, OERI.

We didn't think that actually we could pull all of that off in time that we had. We also didn't think it would be particularly constructive. And so what we have done instead, the committee decided to take a more forward looking approach, to take as a given that there is skepticism and doubt about the quality of education research, and try to set out some principles that fine that, as a way again, to spark some discussion and move forward.

And the same is true for the federal agency. We did do some background work on OERI and related social science agencies in the federal government to try and get a comparative sense of how these agencies do their job, but we did not comprehensively evaluate the work of OERI.

What we are going to do is walk through chronologically, the chapters of the report. I'm going to briefly talk about Chapters 1 and 2, which again, Ellen would have covered would she have been here today. I'll actually not talk too much about this. I'm going to just skip through a lot of this.

This is very important and interesting context, however, that I would encourage you to read, because it shows that in fact there has been a tremendous amount of progress in the last 100 years in thinking about the epistemology of education research, and the nature of scientific research in education. And then in fact, the epistemology and philosophical work on the nature of science in general goes back for centuries.

So these questions are not new. They have been talked about for a very long time, and this introductory chapter tries to summarize very briefly, some of that thinking, and some of the evolution in our understanding of the nature of science, and the nature of social science, and education in particular as well.

That historical and philosophical context led very directly to some assumptions that the committee made when it set out on its work. Those assumptions are described here. One is that there is no one definition of science, and we don't intend to have anyone thing that this report actually offers any definition of science per se, but it is our characterization of that.

We have made a number of other assumptions, one of which is important to point out, and that is that the quality of education research is one aspect of the overall value of education research. In other words, we are not making any kind of assumption that the quality of the science in the research is perfect. That for example, research would be used in an effective way in schools, for example. So that there are other related questions in this larger set of issues related to educational research that we did not take up.

And finally, the last point may seem odd, but it says, science can uniquely contribute to understanding. We wanted to be clear that the committee feels very strongly that science can be a powerful tool in the improvement of education and teaching and learning and schooling. But again, that it is one part of a larger body of knowledge and understanding and insight, that when combined with scientific understanding, can be particularly powerful in improving education for all learners.

So that's just sort of a brief overview of the context and the assumptions that the committee had when it started its work.

If I could have the next slide on Chapter 2. I'm not going to spend a lot of time on this, because the real beauty of this chapter, in my humble opinion, is in the reading. This chapter just goes over a bit -- the report itself starts with the general, and ends with the specific. So at the beginning of the report we are tackling the broader question about how science as an enterprise moves. How knowledge is generated, and accumulated, and understood, and integrated over time.

And what we do is, through a number of examples from a range of fields, education included among them, is show that the way that science progresses is actually very common across fields, despite the fact that the objects of inquiry, and the type of questions that are asked can often be quite different.

And we talk about some of the conditions that we saw that tended to enable progress in the examples that we gave, as well as some of the common characteristics of the way that scientific understanding has moved over time. And so I won't spend any more time on that, but just point out that they did show some examples in a range of fields that show this, and it is quite an interesting read.

And from that broad prospective of how science as a whole moves, we then moved into a discussion of the core principles of science. We call them the guiding principles in the report, that we think apply broadly again, to all sciences. And to talk about that in a little bit more detail, I will turn it over to my colleague and friend and new person whom I get to see on a regular in the hallway and at the coffee machine, I'm happy to say, Bob DeHaan.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Robert DeHaan, Emory University

DR. DE HAAN: Thanks, Lisa.

Let me say that this experience of being on this panel with the Rich Shavelson, Lisa, and the other members has been an enormously gratifying experience, because the diversity of the group, the range of subjects of expertise from philosophy and cultural anthropology and hard science and so on, and the kinds of discussions that we entered into, some very deeply and very passionately, has been an enormously valuable learning experience for me, and a great way to make friends.

The question that we were dealing with, as Lisa said, is there, among the various fields of intellectual endeavor, a body of knowledge, a body of principles that can identify what is scientific from what is not scientific? And the answer, as you can gather from the title of the document that we finally produced is yes, there is.

Now the question is, how does one find those? How does one get to them? And so the guiding principles -- that's what is on this slide -- tell us about the sorts of discussions that we started with. First of all, it became apparent from discussions that started with the philosophy of science and the nature of science, that science is based on the concept of warrant. The concept of evidentiary base. What kinds of evidence warrant a given truth claim?

And any endeavor that bases itself upon that set of assumptions, if there is a body of information out there that we can investigate, and that we can provide evidence to give answers to, has the beginnings at least of a scientific endeavor.

Secondly, however, we sadly decided that there was nothing algorithmic about the principles that we eventually came up with. In other words, you can't simply say, okay, here are six principles. Now we are going to plug those in, and anybody who follows those principles is doing things that are scientific. That turns out not to be the case, as you would imagine, and we'll talk a lot more about the obverse of that, that is, what it is that really does define science.

One of the things that we decided does define science, and we talked a great deal about this, is in addition to the cognitive aspects, is the ethical aspects. There is a body, a core of community norms within what we call the scientific community that is absolutely essentially to the scientific endeavor. And unfortunately, I can't spend a lot of time on that this morning, or else I'll spend the whole hour that we have.

And so let's quickly go into what were the scientific principles as we defined them. And again, partly because we started so late, and partly because we don't have much time anyway, I'll just mention these, and you'll have to read the document in order to get the full flavor of what it is we are talking about.

Any scientific investigate, irrespective of whether it is dealing with being cardiac cells, which happens to be my field, or the star evolution, or social groups in deepest Africa, has to start with an answerable question. That is, one of the things that characterizes good science in every field is the ability of the investigator to select a question which is both powerful, incisive, and also empirically answerable.

And that is one of the critical factors, the questions that are selected for scientific investigation in any field are empirically answerable, are highly significant to the field, and where there is a body of theory from which to draw on that question. And that in fact leads us to principle two.

Effectively all, with very few exceptions, effectively all research questions are, all investigations arise out of a body of previous theory, a theoretical framework. Which in fact, since we are talking about in science, trying to get a generalizable response, a generalizable picture of some aspect of nature, the theory that lies behind the specific investigation that an investigator engages in is of course crucial.

Now it's not only true that theory generates investigations, questions. It's also true of course that questions generate theory, and in fact, that's a completely circular, complete feedback loop kind of situation.

Science uses methods that are variable at the time. Those are the only kind of methods that can be used. Either that the investigator invents on the spot, or that are well documented, investigations that can directly ask that question. So that if an investigator asks a question which is powerful, comes out of a great theoretical framework, but in which there is no way of asking the question, it's not science. It's not even a good way of spending one's time. So we've have got to have legitimate methods for doing science at any particular time, in any particular area.

A very important aspect is evaluating whether a scientific investigation is indeed science or of setting up an investigation. Is the rigor of the logic of the chain of logic that the investigator and his or her colleagues used in order to get from point 1, asking the question, to point X, which is providing some kind of answer to that question.

And that coherent chain of rigorous reasoning, as we called it, has to be something which is public, has to be something which is sharable with other colleagues, and with readers of whatever is going to be published later. The assumptions have to be clearly stated, estimates of error usually in most cases can be provided, and so on.

Starting back with the very earliest history of science was the concept that any valid investigation has to be replicable. If I do it in my laboratory, or in my classroom or wherever, I have to be able to do it again and get something similar in the way of an answer, and I have to be able to have colleagues who are equally knowledgeable, able to do the same experiment and get the same kind of answer.

And to the extent that that is true, we have wonderful science going on, or we have successful science going on. To the extent that that is false doesn't necessarily mean that we don't have science. It just means that there are variables that we haven't controlled. And those are the kinds of considerations that have be thought through very carefully in evaluating the scientific value of any particular investigation.

As I mentioned before, there is a set of ethical norms, community norms that defines the scientific community within any given discipline. It is that open society of researchers that is key to the validity of scientific investigation usually, because whenever one investigator does a piece of research, not only does that research have to be replicable, it has to be in effect debatable.

It has to be open to, and accessible to other investigators who are going to say, well, you have done this or that, and you've gotten this result. And you are attributing that to a particular set of variables, but the fact is that there is this ultimate explanation that would be just as good. And I, as a scientist, have to be open to those alternative suggestions. I should have thought of them myself, but at least if they are suggested by my colleagues, that is a very important part of the whole issue of what is scientific investigation.

Having said all of that, the fact is that we understand that there is no single set of principles, these or any others, that can as I said, algorithmically distinguish real science from pseudo-science, or from bad science. This is a very, very complicated issue that we are dealing with.

And I think we spend a lot of time in the dark, not shilly-shallying, but trying to examine as carefully as we can, what are those complexities that really need to be discussed and talked about and though through very carefully in order to distinguish what is good science, and what is bad science from what is non-science.

Now one of the most important questions that obviously we dealt with, and we had to deal with, especially in education research is, is what I have been talking about, valid for the life sciences, the physical sciences, the earth sciences, but somehow different for the social sciences? And I can tell you that with this very wonderful group, we had some extremely interesting discussions, sometimes passionate discussions about this subject.

And the conclusion that we finally came to, which was embodied in this document, is that these principles, and probably others, apply absolutely across the board in physical science, in natural science, and in the social sciences, and since education is a social science, in education research as well.

What differs is the vast, vast diversity of application that can be made of these principles. As you may have noticed, they are very general as principles ought to be. That's what differs. Social science has different guiding modes, obviously different methodologies, different approaches, bracketing and a lot of other kind of things that one has to do. The investigator as intervener is a concept in social science which is really quite foreign to many of the physical and biological sciences.

But the fact is what the research is able to control in terms of variables differs enormously between what I can have in a test tube full of chemicals, versus what a teacher has in a classroom with 25 or 30 kids. But the fact is that the principles are the same, the applications are very different.

And with that, I will turn the discussion over to my colleague, Bob Boruch, who will talk about Chapter 4 and Chapter 5.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Robert Boruch, University of Pennsylvania

DR. BORUCH: Thank you.

Let me echo Bob's earlier remarks about the value of working with such committees. It also reminds me about Kurt Vonnegut's sort about advice that Kurt's father gave to him. His father said to Kurt, always hang around with people who are smarter than you are. And then he added with a note of regret in his voice: For you, that ought to be easy. It is great fun to find out how smart people are in their own areas, and to understand how to link what they know to what one knows, or thinks one knows.

I was asked to discuss Chapter 5 mainly, and I'll do so. One basic principle here is that the methods used to produce the principal science evidence in the social, educational, and physical sectors evolve over time. The idea of probability sample surveys, although we can trace it back to the fifteenth century, including the history of the probability of statistical ideas in ancient Arabic and Hebrew literature, the fact of the matter is that that technology and its development is less than 100 years old.

The same is true of controlled trials, and a lot of other methods. So that the one major premise here is that these methods are going to evolve over time. That we ought to exploit the ones that we find trustworthy. And their is use is encouraged in this document.

The second major premise is that studies are scientific when they meet those principles of science that Bob described. And understanding the methods of generating the evidence is only feature of that series of principles.

The third basic notion is that the specific questions that are addressed ought to drive the methods that are used to produce the answers. This may seem kind of obvious once said, but the fact of the matter is that there is a fair amount of confusion out there about when this ought to be used and the like. And part of that confusion hinges directly on the fact that the question itself is not specified well enough, and may not be even empirically answerable to yield the scientific evidence.

Rather than get into the endless discussions about the character of what's valuable about which methods, we decided to break up the world in terms of the questions that are addressed in scientific research. Grace Fielding(?) suggested that that world can be divided into three parts. Asking questions about what is happening, which implies descriptions of one sort or another.

Questions about whether there is or is not a systematic effect of an attempt to intervene with what is going on to interfere with the educational world, with the classroom, whatever.

And questions about how and why whatever is happening, or seems to be happening is happening. That gets into thinking about, and doing empirical research on mechanisms, dynamics of what is going on.

With respect to descriptive questions, generally speaking we have endorsed the idea that high quality probability sample surveys are a good idea. We have had an uneven history of that activity in that country. Most recently, the National Center for Education Statistics has done a marvelous job in becoming a world-class producer of statistics in this arena.

We understand and depend on those kinds of data to establish statistical relationships. Certainly, in the case of NAEP, understanding that there is a strong relationship between the level of achievement, and the education of mathematics and the backgrounds of teachers in mathematics and mathematics education is important to understand.

The idea that in different kinds of settings, probability sample surveys may not be appropriate. Rather, ethnographic studies or anthropological studies might be the best way is also clear. And there are lots of cases in which, for example, what we understand about social phenomenon out there is pretty sparse, pretty scant.

I think of a specific example of understanding something about what happens when children who are poor, but who are relatively high achievers get scholarships. What the effects of monetary incentives on these children are. It is an interesting question, at least to some foundations that dole out scholarships to children.

Doing the reconnaissance on how children think about that money, how their parents think about that money by and large has to do with a perspective that one cannot get using probability sample surveys, at least at the front end, because one does not understand enough about the phenomenon to even pick off the right questions that will be addressed in the probability sample survey. And as a consequence, what we know is based in advance of the experiments and of the probability sample surveys. It is based on work that is ethnographic.

With respect to causal questions, we also are reasonably clear in the report, suggesting that the probability sample surveys and the ethnographic work is largely narrative. Its character has lots of provocatory power. It is producing ideas, elevating the level of discussion, but has limited explanatory power for generating answers about what causes what.

Here again, we divided the world up into when randomization is possible, and when it is not possible. We have encouraged the idea that regular science is the best way to achieve fair comparisons. That is to say, generating statistically unbiased estimates of relative effects of interventions in a reasonably legitimate statistical statement which brings confidence in the results.

We have also emphasized the fact that they are used in many disciplines, certainly not only medicine. And the fact that although they are "relatively rare" in education, there is lots of opportunity to do far better work. The claims that experiments are impossible, made by some scholars, including in some university settings, is simply not true.

Even worse, there is a fair amount of ignorance out there about the extent to which trials have been run in a variety of education settings, including large scale trials in which schools and school districts, not to speak of hospitals, geographic regions, police hot spots, and neighborhoods are the units of random allocation in analysis in attempt to intervene in the system to enhance the well being of people who are in the schools, neighborhoods, and so on.

We certainly concluded that experiments are not possible, and would not be desirable activities. Understanding that in the non-experimental settings, one can evaluate or estimate the effect of almost anything, but a lot depends on the assumptions one is willing to make and tolerate in any of those assumptions for economists and statisticians and theoreticians, all heroic, which is one of my favorite lines, which is epochful.

It comes from a Nobel Prize winner in economics, who, at a meeting of the Labor Department, discussing whether labor ought to invest in experiments, opened up with the phrase, "assume for the sake of simplicity that people live forever." For him, that was a tolerable assumption. Others found it stunning.

The bottom line here is that certainly we have to rely on non-randomized trials at times. We have to learn a lot more than we know now empirically about the extent to which the randomized produce the same estimates of relative effects as the non-randomized trials. And there are a number of efforts going on in the area that are identified in the text.

With respect to mechanisms questions, simply because one can generate an unbiased estimate of the relative effect, that doesn't mean we understand exactly how it works. There is still a fair amount of research, for example, re-analyzing data from the Tennessee class size experiments to understand exactly why those reduced class sizes lead to a reasonably discernible effect on those kids.

Here again, we reach to our colleagues in the ethnographic community, to other folks engaging in so-called designed studies. The main objective is to be as careful and open as possible about how things are supposed to work, what the dynamics are, or various dynamics might come into play for example in how small class size might produce those effects on children.

I think that's the end of my tour.

Agenda Item: NRC Committee on Scientific Principles in Education Research - Findings and Conclusions of Scientific Inquiry in Education - Jack Fletcher, University of Texas, Houston

DR. FLETCHER: Well, I would be remiss if I didn't also acknowledge the joy of working with people that are generally smarter than you. But it would also be unfair if I didn't acknowledge how fun it is to work with people who are just as tired and irritable as you are at the end of a meeting.

And this next section is undoubtedly the part of the report that produced the most fatigue and irritability amongst the members, because was sort of hard to get a handle on exactly what we were supposed to do, and what we were supposed to evaluate. Part of the problem that was actually an advantage was that we were not the first report of this sort. There were actually reports that go back to 1958, that talk about what educational research agencies should look at.

And what we found most helpful was a deliberate decision, because it wasn't in our mandate, not to evaluate the Office of Educational Research Improvement, but ask broader questions that were more forward looking. For example, the question that we really focused on was how could a federal education research agency facilitate the development of a scientific culture of research? Really foster a community of researchers.

If you read Ellen Lagemann's eloquent history of education, it is clear that one reason that educational research is often fragmented is that there isn't a coherent scientific community of researchers who have a firm commitment to science.

So we were really focusing on issues that might involve a culture, and the development of a culture, as opposed to approaches that try to introduce mandates, which is the usual approach taken to develop education research. And we thought in general that you really can't prescribe good science through mandates.

We recognized that there are a number of dilemmas. That education research has to be grounded in practical problems, but too close a relationship to practicality and what works on a day-to-day basis has a significant downside. We also recognized that there are all kinds of people who are involved in educational research through different forms of scholarship, through different disciplines involved. And while our mandate was to focus on science, we stated very clearly in the report that we felt that there should be federal support for other forms of scholarship -- libraries, history, and so on.

We came up with six different design principles that we hoped would foster a culture of educational research, and I'm going to go through these briefly. The first one is to staff the agency with people skilled in science, management, and leadership. And by that, we mean simply that the director of a federal education research agency should be a scientist. It should be somebody who is in themselves, a researcher.

They should have a term that is independent of political terms, to insure continuity of the research program. They should be free to hire the best person possible, and they people all can be researchers independent of political constraints. We noted that you cannot attract people to an underfunded, understaffed agency. There are better jobs out there for solid researchers.

We think that it is very important to have a federal education research agency to facilitate interactions with other federal agencies that engage in educational research, like the NICHD or the National Science Foundation.

The second design principle was to create structures what would guide the agenda, to inform the funding decisions, and monitor the work. And what we mean by that is that the agency itself should be free to interact with the field in developing an educational research agenda. They should not have to work as closely with mandates, but should interact more closely, for example, with the governing board that also has an independent agenda setting committee.

The governing board could be modeled after the National Science Board. You could have an agenda setting committee that was chaired by a distinguished practitioner, for examples. And these sorts of interactions would help the director and the staff set educational research agendas they interact with the field.

We felt that the development of standing, ongoing, long-term peer review panels was absolutely critical. There are many structures of this sort that can work. The key is being able to recruit people to these panels, who are good peers, who see serving on a peer review committee as a service to the field.

And that perhaps the most important thing about peer review is that it facilitates the development of a scientific culture, and a community of scientists by providing feedback to the field through systematic reviews of research, by deliberate attempts to help develop young researchers through the peer review process. Peer review is not perfect. It is highly dependent on developing a stronger field, and it is tricky in education, because the field is bit more deflective.

The third design principle is to insulate the agency from political interference. Again, we feel that it's important to avoid micromanagement of decision-making, and to distort the agenda to be strictly on a short-term position; the use of the agency to promote certain political positions.

The director should have a fixed term that is independent of political terms. Setting priorities, and so on, should be done through an agenda committee, and through annual reports to Congress. There should absolutely be accountability for the research that is produced, to independent reviews by Congress and by the executive branch.

We think that budgetary discretion is particularly important. To use OERI as an example, the bulk of funding decisions right now are largely mandated, and there is very little free discretionary money that could be used to fertilize scientific research in education.

We think that to a certain extent the dissemination functions should be separated. And while there should be interactions between a research component and a dissemination component, the research component should be free to develop as much as possible.

The fourth design principle is to develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice. Short-term for example would be what works. What instructional practices would address a particular problem that is of concern to the country?

And what we might do for example, is to develop empirical syntheses of educational practices that might facilitate the development of a particular group of students that we are concerned about. But you can't develop educational research just on a short-term agenda. For example, the RAND reading study group recently wrote a report emphasizing the importance of a research agenda on reading comprehension.

A research agenda on reading comprehension cannot be short-term, because it is a complex process, where insufficient research has been done. And it can't be done with just a single span asking what's the best way to teach reading comprehension? You need to have multiple disciplines apparent. You need to have multiple strands of research, cognitive mechanisms, instructional research, assessment research, because we don't know how to measure it adequately.

Even things like neuroimaging and genetics would be relevant, and the real issue is how do you integrate across these different strands to develop a coherent model of reading comprehension? That is a long-term agenda.

And then you probably need to balance it, and have both short-term agendas, and long-term agendas. For example, if you think about English language learners, we could certainly synthesize research around what works, but a problem with our body of knowledge about how to teach English language learners to read is that we haven't had a long-term research agenda, and we tend to rely on the short-term to inform the long-term, which may misinform public policy.

We especially would emphasize the need to incorporate syntheses of bodies of research as a way of facilitating the accumulation of knowledge. And that's been done adequately in education.

The fifth design principle is not surprising. It's been said repeatedly, and it is to adequately fund the agency. As we noted in the report, about 1/10th of 1 percent of all educational spending is on research. The 1973 research budget for OERI was about $425 million. In 1991 it was about $100 million, and it increased only slightly in real dollars since then.

Good educational research, like research elsewhere is expensive. The examples that we gave, the Tennessee star study cost $10 million over four years. The National Institute of Child Health and Human Development has spent $100 million on reading research over the past 15 years. Even now there are several projects that are funded by the NICHD on reading research that have budgets between $1-2 million per year. And budgets of that size are necessary to deal with the complexities of educational research for different levels of analysis that you have to have.

The sixth design principle is to invest in the research infrastructure. The agency should have as a goal, a desire and a mission to help develop a strong -- what Ellen Lagemann described as a self-regulating community of scientists. It should emphasize the importance of peer review by requiring it for grants. It should support important components like data development, like an education statistics function. And then finally, it should be linked to practice and policy communities by the development of partnerships and interactions with a separate dissemination function.

Thank you.

DR. FEUER: Well, this was a very speedy tour of what is in this report. I am reminded a little bit of when Woody Allen said he took the Evelyn Wood Speed Reading Course and then was asked what "War and Peace" was about, and he could say was Russia.

I encourage you to read this report more slowly and carefully than we have been able to sort of hit you with, with these slides. To spur our thinking, we have done something which is really quite customary to the Academy, and that is to invite a group of folks who were not part of the original deliberations, who are not on the committee, but to provide some of their independent, thoughtful judgment about what this committee has done.

Those of you who are familiar with our review process will wonder whether it was not designed by that economist that Bob mentioned, who assumed that we can live forever. In this case, actually we got this report done pretty quickly. The people we have asked here today were yet three new people, who haven't been even part of the review process that went into this report. So this is another example of how much we want to engage the community in thinking and talking about this project.

We are very happy to introduce our panel of discussants, Rick Apling from the Congressional Research Service. As you know, CRS is an organization that has an interesting client. It's the United States Congress. And they have been in the business of providing objective and independent knowledge and advice to Congress for a long time. So I'm very much looking forward to hearing Rick's spin on how scientific inquiry in education can contribute to the policy formation and policy thinking.

Also on the panel of discussants, Pat Graham of Harvard University, a distinguished scholar of education, history, and research, and a leader in this field. And actually, quite recently the chair of one of our committees, for which I just want to say thank you again to her for her work with us on that.

And then finally, Dave Grissmer, from the RAND Corporation, a leading contributor in the research on education and related public policy fields.

So I think the batting order is in the way I have introduced you. And then we will look forward to a time for questions and answers. I'm happy to report we are now only six minutes late thanks to the great work of our panel on this side.

Agenda Item: Discussant - Richard Apling, Congressional Research Service

DR. APLING: Thanks, Mike.

I want to talk a little bit about CRS, and a little bit about how we work, because it colors my reaction to the report. And some of you may not be very familiar with our agency.

We are one of three congressional support agencies, but unlike the General Accounting Office, and the Congressional Budget Office our work is much more closely aligned with the legislative process, from the very inception of legislation, to the final conference committee agreement. So several of us have been spending the last six months, lots of time with the conference committee staff for HR-1, and we are glad to see that that's been signed, sealed, and delivered.

Our effectiveness and credibility really depends on our strict nonpartisianship, and this both with regard to party, but almost as important with regards to House versus Senate, and of course on the quality and the relevance of our analysis. I see my role as providing the best possible information to all members and staff and committee of Congress. And although sometimes it's a little unnerving, we really feel like we have done our best work when everybody is using our work to make their arguments on all sides of the policy question.

We almost never collect our own data. So we are very dependent on the work of others. So we have a vested interest in the work of this committee, and the various executive branches for the quality of educational research.

I want to do just kind of a brief case study of a project that we have been working on for the last couple of years just to give you an idea of how we do our work. And this has to do with something called the Rural Education Achievement Program, which actually has been signed into law as part of HR-1.

About three years ago the American Association of School Administrators floated a proposal basically calling for more flexibility and more funding for rural schools. This struck a chord with the Congress, and we were asked to look at that proposal. When we start to do this, one of the first things we often do is to try to assemble a data set from the various disparate sources of data that we have available.

And in this case we put together data from the Department of Agriculture, the NCES common core of data. Later on, poverty statistics also from NCES, with respect to Title I schools, and the Census Bureau with respect to population density.

Early on we raised several questions about the proposal. One being what is a rural school anyway? Is it dependent upon where it is located, whether it's rural? Or is it size? And if it's size, how do you measure size? What is small? And should poverty be part of the equation? And is the problem not enough money, or not enough flexibility?

Now some of these questions that we raised were considered, other were not really paid much attention to. And as the legislation moved through the process, as it often does, it comes down to who is going to get money, and how much money are they going to receive? So using data from other sources, we were able to look at the original AASA proposal, and we discovered that with some states, there would be lots of rural schools, and in other states, especially Southern states, there wouldn't be any.

This led other staff of members, especially from states from the South, to question the proposal, and we started looking at other alternatives. And one of those alternatives was to bring in a poverty factor into the legislation. Well, if you look at the final legislation, you will see that we now have two grant programs, one which is based on ruralness and size, and the other one is based on ruralness and poverty.

Now there were some questions that we weren't able to answer, and that's because of data limitations, and our inability to collect our own data. A key question had to do with how much money districts would get. And part of this had to do with the way the AASA proposal originally was crafted, and which made it into the legislation, which was that there was an offset from antecedent programs that would reduce the amount of the grants.

Well, as some of you know, it's very difficult, surprisingly, to be able to say how much money school districts receive from the federal government. They have requirements to do that in the General Education Provisions Act, and the Department does try to do this, but we don't have very up-to-date data. So we knew we were never able to say for that part of the formula, here is how much XYZ school district was going to receive.

Other questions either we never returned to, or we would never ask in the first place. For example, are the indices of ruralness accurate? And are rural schools really disadvantaged compared to other school districts in terms of their inability to compete for funds in the formula factors? And many even more complex questions of how this money would be spent. Would it be effective? Are there more effective ways of using these funds?

The point I'm trying to make here is that although these are important policy questions, our work is driven by the needs and concerns of Congress, rather than what the really important research or policy questions might be in the abstract. This was a quick overview of a project that we have been working off and on for I guess about three years. I hope it will help explain some of my reactions to the report.

I think the report does a very good job of describing the scientific method, and a particularly good job, in fact perhaps too good a job at laying out the difficulties in making educational research more scientific. Let me just point out a few of these points that I believe the report makes.

We have very few, if any real theories in education I think is one problem. The problem of replication, even if you have a well design random assignment experiment is very problematic. And I think most importantly, which the report acknowledges, there really is not currently an appetite for educational research among policymakers and the public in general.

And I also want to just mention a few things about the report that I think that I originally said were unrealistic, but I think maybe that's a little unkind. I think there are areas that need to be thought about very carefully, because they are very difficult issues. And they are kind of contrasts.

One is the desire for more federal support, but on the other hand, the desire to have less political influence on the process. Another is the recognition that research has to be tied to practice, but on the other hand, the notion that program improvement activities should be removed from the research entity.

And a third area that I think requires a great deal more thought is the notion on one hand -- and this is the way we always write our CRS reports, on the one hand, on the other hand -- there is a desire to hire very skilled people for this educational entity, but the report acknowledges that we really don't have a very good idea of what educational scholars should know and be able to do.

So this all in all, makes me a bit pessimistic about first of all, immediate congressional action. But more importantly, whether the Congress is going to be able to be the main solver of our problems here.

So to conclude, I want to paraphrase an article that was cited in the report, which I would like to read. I guess it's in press, by Carol Weiss, which was entitled, "What Do We Do Until the Random Assignment People Arrive?" My question is what do we do until high quality scientific education research is available?

And my suggestions are first of all, that most research, if not all, has to really be driven by the needs of the stakeholders. I think we pay a lot of lip service to this. I don't think we pay enough attention to it. And these stakeholders, as you well know, are teachers and principals, and in may case, national policymakers and analysts who support them. In the latter case, I think this is particularly true if you are expecting people to produce the money.

From my perspective, what I need now is more and better descriptive research. What are people doing? What are they spending money on? What works? I'm less interested in this point in causal and explanatory information, not so much that that wouldn't be great, but my feeling is that we are not at that point yet. We haven't done the descriptive spade work that we need to do.

And I case my analogy to the physical sciences, I don't see a Newton or a Maxwell or an Einstein on the horizon who is going to come in and give us a grant theory. I think we are more at the stage of alchemy. And I don't mean to be completely facetious about this. I think we have to discover oxygen before we have Mendeleev put together the Periodic Table, and certainly before Linus Pauling can tell us how quantum mechanics expands chemical reactions.

Meanwhile, Congress is going ahead and passing educational legislation, and we try to give the best information that we can. And again, what I need, and what my colleagues are good data that we basically can reanalyze, and make relevant to the policy questions in a very rapidly moving policy environment.

Agenda Item: Discussant - Patricia Graham, Harvard University

DR. GRAHAM: Thank you. That's very helpful to me, as were the earlier remarks.

I want to say that I read this report carefully, and with admiration. I want to go on to raise two questions about it, but I want to start by saying that I think that for those people who have the energy and the inclination, this is a report well worth reading in its detail.

It stresses important questions in education that need investigation and clarification. It points out that not all research in the past has been as good as it should have been in education. It points out a limited constituency for good educational research. Who really wants good educational research? And who wants to insure quality control?

There are many pressures in the field of education that make quality control very difficult. For those of you who have never had the opportunity to attend a meeting of the American Educational Research Association, which is a great, big outfit, and when you are asked why is there not more selectivity in the program about having a limited number of really fine talks, the answer is, well, departments won't pay unless you are on the program, so we want everybody to be on the program so they can get their way paid to the AERA. That is not a good means of quality control.

And this report, if carefully read, I believe would make some real progress on this. I think to me, one of the most important elements of this report is its emphasis on a culture of rigorous inquiry. Indeed, it often uses the phrase, "scientific culture." Well, there's good science and there is bad science. I'm not just interested in a culture of science. I'm interested in a culture of good science. So I think the emphasis in the report on a culture of rigorous inquiry being essential for good education research is absolutely vital.

It also has some great things to say about enabling conditions for good educational research -- time, money, and support. I think it makes another excellent point when it talks about the fact that educational research has been hampered by not wanting to know, or being unwilling to investigate certain questions.

You think for example, about how hard it was to get NAEP established, the National Assessment of Educational Progress. NAEP could not be established within the federal government. The initial funding for the exploration of NAEP came from the Carnegie Corporation. And school leaders in America, known as school superintendents, chief state school officers of various states prohibited the data from NAEP initially being released by state, only by region.

This was an example of not wanting to know how American children were learning in their schools. And this has been a profound hindrance to some educational inquiry. We have been quite reluctant historically in releasing academic achievement data either by race or by income levels, which we have begun to do in the last decade or two.

And finally, we have not been enthusiastic about engaging in widespread research on the relative significance or insignificance of schooling in child's overall educational achievement. We have found it very difficult to talk about in the past, about how important schooling was, as opposed to how important other kinds of factors are. So I think what we have avoided has been a problem.

There have also been successes of educational research. I take the long view. I have made my living teaching the history of education, so I'm looking at the past on a regular basis. And if you look at the kind of educational research, and the stark experimentalism that characterized the field of Edward L. Thorndike from 1925 up until the 1970s, in which we thought that we could learn about learning by what college sophomores did, which was the favorite unit of analysis, because this group could be easily studies.

We paid very little attention to context, to community, or to issues of shared purpose. One of the great contributions of a fine quantitative methodology, Anthony Bryk at the University of Chicago, has been, and it's cited in the report, is to stress the significance of shared purpose in influence in what happens in Catholic schools. But the issue of context, the issue of community we now see must be considered, as well as these types of experimental designs.

When you think about the first test that educational psychologists put together, the Army Alpha, the first large group scale test for World War I, and if you haven't taken the Army Alpha, I really encourage you to do so, because it is a tremendous example of what enormous contributions have been made in test methodology since then.

There is one scale on the Army Alpha which purported to be a culture-free test, which was which of the following makes soups? Well, the answer was Hardshaftner and Marx. My father comes from and Danish immigrant family. My father took the Army Alpha. The Army Alpha was supposed to separate the people who were going into the military from the people who were going to be cannon fodder as enlisted men as opposed to officers.

Now my father is Dane. How would he know that Hardshaftner and Marx made soups? And as we look at the sophistication that has come in testing with the intervening 75 years, I think it's a real tribute to educational research.

I also think that the famous progressive educators in the United States in the midyears of this century who said to us that curriculum doesn't make a difference. Children should just study whatever they like. This is a parody of the worst form of progressive education, part of which was widespread, however.

It made a real mistake, as we would say now, in not thinking that curriculum is important. What we need to know and be able to do. And it is educational research of the last decade that, complete with the standards movement, has made us believe more about this.

So there are some very good things in this report. I have three question about it, three queries. The first is the audience for the report? The second is what's the point of educational research? What is educational research for? And the third deals with OERI.

The first question, who is this report for? We heard a little bit about who this report was for. My question is, does the audience that is the group that is intended to read this report, does that make a difference in what this report says? Should it? And I think the one point that I take away from this question about audience, which I think is important for this report, is that this report needs to be read carefully.

You need to read footnote 4 on page 94 that says history and philosophy are also relevant to understanding education. You need to read on page 52, "science does not necessarily mean good."

Because if you do not read this report carefully, and if you only read it with executive summaries, as some of us have sometimes done in reading reports, you will get a simplistic notion that the report does not, in its complexity state that what we have here is the paradigm for the hard sciences that need to be applied to education. And that it is just scientific, and then it would be all fine. That's not what the report says. But a cursory reading would leave you that impression.

The word science in English has a connotation that it does not have in Washington(?), naruka(?), or German, vichensha(?), which in those cases means scholarly inquire of a rigorous sort. I think what this report is calling for is that kind of rigorous, replicable, scholarly inquiry. The word science in English makes you think of cell biologists and physicists working in a laboratory. They do wonderful work. They have excellent principles. They are outlined here. But this subject is more complicated.

Secondly, what is the purpose of educational research? What is educational research for? Why should we do it? Is it to improve education through knowledge? Or is to improve educational research itself? And I would argue that there are many references here to the disciplines, particularly physics and biology. And research in those fields in intended to help us learn more about those fields.

But many would not characterize education as a discipline, but rather as an applied human activity. And therefore, what is the best way to characterize research about it? Whose interests are served by education research? Is the purpose to increase our research knowledge, or is it to find ways of improving education research?

Agriculture and medicine can be improved, and are improved by research. The argument is that they have been improved more by research than education has. And that education ought to be more like those. Well, I would just like to observe that there is an economic infrastructure in both agriculture and medicine that does not exist in education, that is served through research -- fertilizer companies, pharmaceutical companies, for example.

And that there is an economic incentive for improvement. Doctors have recently discovered productivity. Farmers heard about it some time ago. That is also served by research. Education, except for book publishers, hasn't seen much of that.

And finally, the most significant examples of successful research in either agriculture or medicine have to do with dealing with something which is external to the human. That is changing cow production so that they have a higher butter fat content in their milk. It took 50 years for the University of Wisconsin to figure that out. Changing tomatoes' ripening characteristics so that you could send them on a long trip without their getting mushy.

Major research in medicine has allowed physicians and surgeons much better tools with which to operate. Anesthetists have better skills in putting people to sleep so this can go on. Finding medications that control widespread ailments such as diabetes, hypertension, and coronary disease.

When medicine has turned to changing the way people live their lives -- lose weight, don't drink so much, don't use drugs, don't engage in illegal, unsafe, or socially disapproved sexual activities -- medicine has had much more difficulty changing individual behaviors than it has done in changing the techniques that allow us to learn.

Smoking is perhaps the best example. Smoking in the last 30 years -- more people smoke a lot less than they did. But in point fact it is most prevalent among the most affluent in America, not among the poor. The analogy to education is that the problems that we face that most of us are particularly deeply concerned about are the educational problems of the poor.

Bob Boruch mentioned, I think very effectively, the section of the report that talks about opportunities for evaluation studies. And I just would like to say that I agree with everything he said about that. And I, for example, favor education engaging much more actively than it has in the past in efforts at randomized trials, where selection can be done according to the rules which he outlined, and the report outlines.

But I would also like to encourage Bob and his colleagues to push a little on answering that last question that they talked about, about why and how these changes have taken place, to which Bob alluded already. It's what I call the whether-why question. Policy people want to know whether a program works. Practitioners need to know why it works, so that it can be replicated, and knowing what the essentials of replication are.

That's a much harder question to answer, and often our colleagues in the research community are unenthusiastic about pursuing that, because the research design for answering the why question is not immediately apparent. But it seems to me that if it's essential for those of us who think of ourselves as researchers, and well as for the population with whom we work to be able to address that question.

Let me now turn finally to OERI. What are the implications of this report for OERI, the federal government's only agency solely committed to educational research and improvement? Well, poor old OERI. The tragedy of the federal government's educational research effort in the last half this century is that unlike its experience with research in health through the National Institutes of Health, or in science through the National Science Foundation, the Department of Defense, and several other federal agencies, where much of the nation's finest research in those areas has been federally funded, that has not typically been the case in education.

There certainly has been some fine research funded by the old US Office of Education, by NIE, and by OERI. But the dominant educational research funders in the US I believe, have been the major foundations and the universities, the latter through their support of faculty and graduate students who are expected to do research. Now why is that so?

I think the first is that there has been an absence of a research culture in the US Office of Education, which had dominated the culture of the Department of Education when it was created in 1980. The USDOE was a place that dispensed funds according to congressional mandates. It was not a place that valued research, nor were the school systems in the United States dying for research.

The limited success of the National Institute of Education from 1972-1980 was due in part I think, to its presence within the Department of Health, Education, and Welfare, which did have a strong research culture in its institutes of health, which was reflected to some significant degree in the senior management of agencies.

The only two times that I am aware of when the budget for the National Institute of Education went up significantly, 25 percent the first year of the Carter administration, and 10 percent the following year was when it was in HEW, where there was a strong tradition of research. So I think it would be inaccurate to say there has been a steady drop in decline for educational research funding, but the one time that it went up, it was when there was a relation to the strong research culture.

And I want to say something about the issue of political support and political interference. I'm absolutely in agreement that agencies need to avoid, if they can, political interference, but I am absolutely in disagreement with the idea that you do this through insulation. Rather, you do it through long, close relationships with your colleagues on the Hill, and in the administrative branch.

And what has been most striking has been the inability of educational researchers to figure out how to get along with congressional staff, with members of Congress, and with OMB and the White House. Figuring out how to build those close ties is absolutely essential, so that you are trusted when you say to a congressional staffer or a to a member or to somebody in OMB, this is the way it is. This is what we have to do, and we need your support for it. That's based on close ties, it is not based on stiffing them.

The third, as the report says, we must hire good people to work in the federal research agencies. And there are ways to do that. And we have had good people. But the creation of the Department and followed by the Reagan administration's efforts to eliminate the Department of Education sent people out of the educational research agency who are among the most distinguished educational researchers in universities, in the policy community, and in the world of practice.

I mean I've got a list here, which I will not read at the moment. But it is an absolutely distinguished group of people who used to work at the education research agency, who spread throughout the government, back into academe, and into policy positions.

The final point I want to make is that the biggest problem of OERI is that it doesn't have any money. And this has been alluded to by several speakers, but I just want to go over this one more time. The assistant secretary in the first Bush administration reported a 90 percent decline of federal funding for educational research between 1975 and 1995.

The National Research Council study in 1992, found an 82 percent decline between 1973 and 1989. Another NAS report in constant 1990 dollars found the federal educational expenditures of research and development had gone from 1972 at $1.1 billion, to $300,000 in 1991. That is an incredible drop.

Field initiated studies, OERI does 2 percent, NIH does 56 percent, NSF does 94 percent. If you believe people out there are smart, and you don't fund their ideas, you don't advance education. In terms of basic research, and I am a strong believer in basic research. Nothing is more practical than a good idea. NSF does 94 percent, NIH does 60 percent of basic research, OERI, 5.5 percent.

Federal educational research for research and development, only half of 1 percent of educational spending is committed to research for education. It's three times that for agriculture, 21 times that for space research, 30 times that for health.

The congressman who stimulated this report said that there is a problem with educational research, it is broken. The problem with educational research is not that it is broken. It is that educational research is broke.

Thank you.

Agenda Item: Discussant - David Grissmer, RAND

DR. GRISSMER: Certainly, some of my comments will echo what Rich and Pat have said. First, I would like to congratulate the Department of Education for funding a study like this. It's always risky to have an expert panel look at what you are doing. I have found the Department of Education to be very open to the kinds of advice they get.

Secondly, I would also like to congratulate the National Academy for taking on a study like this. They get handed some of the toughest issues in scientific issues in our society. But this one I think had an absolute peculiar twist. That is, most National Academy studies tend to assess a fairly narrow area of research, go through and assess the evidence, come out with some balanced conclusions.

It's quite a different animal to try to assess whether a whole body of research is scientific or not. And that's much more difficult terrain, especially with the time period they had. I heard the word epistomological earlier. I haven't heard that since my Jesuit educational days. But since we don't have an NAP, a National Academy of Philosophy, I'm glad the National Academy of Sciences took on this task. It was a difficult one.

I think it's important to reiterate what Lisa said what the charge was. They did not take on the task of evaluating the quality of research, of existing researchers, or the existing federal agencies. What they did, what the charge was, review and synthesize the literature on the science and practice of education research and education, and consider how to support high quality science in a federal agency.

So they have described what scientific inquiry is, and the principles for doing scientific research. They have identified distinguishing features of educational research that might make it difficult or different to carry out than in other areas. They suggested research designs to make research scientific, and the design principles for federal agencies.

As such, I think this should be seen as more of a how-to manual, rather than a comparative analysis of what is actually being done with respect to what is being done, which is a different, and I think probably a future study.

I found much to like in the report. Their description of how to do scientific research of always making it sound like a recipe, but captures much of the complexity and uncertainty involved in scientific discovery. Science is perhaps the best way we have found to develop widespread consensus about theories which explain observational data. But it's often circuitous. It's indirect. It's nonlinear. And it is also a very human activity.

The goal of the construct of theories that explain ever expanding and more complex sets of observations is only accomplished in the community of scholars that involves open discussions, strong disagreements, politics, unpleasant personalities. In short, everything that occurs in any community with people. And I think more and more we are beginning to capture that part of science. And it's not a process, it is much more complex, and they have really captured that well.

In describing the practice and environment for educational research, I think the report makes several good points, although I will later disagree with some of these. The first point I think is very important. Educational research is different, because if affects people's lives, and challenges people's beliefs.

In other words, educational researchers could be doing perfect and outstanding scientific research that produces reliable results, but a cloud and fog of controversy from inside and outside the education field will always be present that makes it appear like the education research is inconclusive.

One of the more difficult things, and when I started education research I had to address is what I call this fog of advocacy which surrounds education. It will always make it more difficult not only to achieve consensus, but to make the consensus knowing within the community and outside. And I think one of the major points that we need to discuss later is the consensus panels. And how we know when we achieve consensus. We can have consensus, and not know it very easily in education, whereas, that's not true in a lot of the physical sciences.

Second, the authors state the physical and social sciences have a similar set of principles that guide the inquiry. But they sort of expect the same degree and pace of success. I think that is well said. There are many reasons for this, but primarily physical science continually developed technologies that allows them to observe and manipulate every possible object of study at ever increasing depth and reliability. And the objects do not change over time and space.

What technology allows is not only observation, but extremely high signal to noise ratios. No two children or teachers are the same. They change over time. The source of their behavior and learning takes place within the least explored, and most complex of human organs, the brain. We are still looking through a glass darkly.

So in the short run we are going to make less progress, achieve less consensus, and have more uncertainty than is found in the physical sciences. An argument can be made the brain research will eventually provide the kinds of reliable and producible data that will change all of this, but that still remains to be seen.

I also like the conclusion that the distinction between qualitative and quantitative research is overdone. I agree to much less extent about the distinction between basic and applied research may be unnecessary. One reason the qualitative research can be important is that it basically can improve quantitative research.

Quantitative educational research I think is often characterized by relatively simplistic models of development and behavior. Teachers, social workers, therapists have exceedingly more complex models of what goes on in classrooms, families, and students than do researchers. And qualitative observation should bring new hypotheses and variables to bear in helping quantitative research capture more of this complexity.

I do think a distinction between basic and applied research is absolutely necessary to maintain. While it is true that pursuing applied research can certainly make progress in basic research, basic research is needed because funding is hard to justify because of the basic.

The history of science tells us that one cannot predict where breakthroughs are going to occur, but it also provides confidence that supporting a wide array of basic research is the surest way to insure progress. So I think they make some comment that basic and applied may have outlived its usefulness. I don't think that is the case.

The section on the design of educational research distinguishes between experimental and quasi-experimental. The pitfalls of establishing causal relationships, as well as establishing reliable, generalizable, and reproducible relationships. It points out that selection effects and contextual effects are particularly difficult. I would add the difficulty of separating family, school, and community effects, and substitution effects between families and schools. The design principles for a scientific agency are hard to argue with.

Let me talk now about some of what I would like to challenge in the report, but also make some observations that primarily lie outside the scope of the report. The question we are ultimately trying to answer is why is practice not making more progress? While the report did not directly address this, my reading between the lines is that one answer would be slightly apologetic -- perhaps too harsh a term -- but it would go something like this.

Well, it's harder to make progress for a number of reasons that we're different from the physical sciences. So we shouldn't have the same expectation. Progress can be slow in any science. There are long periods of nonproductive research, often punctuated by great breakthroughs, a lot of dry holes drilled. So lower your expectations.

Emphasize we have made progress in some areas. I think there are examples of assessment, and phonetics and reading are particularly apt. I'll argue later with the example that we made progress in making the effects of resources on education.

And then the final thing is funding has been a problem. So there is sort of an underlying term in this that says we have sort of got excuses for why we haven't made progress. I'm sure that's too harsh a judgment. But let me read a couple of quotes from the report that I think tend to support this.

One is that "Studies that do not start with clear conceptual frameworks and hypotheses may still be scientific, although they are obviously at a more rudimentary level, and would generally require follow-on studies to contribute significantly to scientific knowledge."

Another point is, "In sum, it is clear that there are no bright lines that distinguish science and nonscience, or high quality from low quality research." And I think even the authors felt a little just tainted in the report, because they say at one point, "The broad view taken in this report should not be interpreted to suggest anything goes."

So in writing the report, there was sort of this tension between justifying what we are doing, and yet criticizing it. I think that for my taste, they came down too much on the sort of anything goes kind of realm. Two other examples I think illustrate this. One is that there are two kinds of research, experimental and quasi-experiment. And there is a third, which is called non-experimental.

And the differences in my mind, the quasi-experimental has some prethought as to choosing the control group, but 90 percent of research in education is non-experimental, which is nothing has been done beforehand, and it uses a set of data, and try to figure it out. And I think that distinction is important in terms of understanding what is done.

Secondly, they give an example of an example where progress is being made in the area of the effects of resources on education. Let me paraphrase very simplistically, but I think it goes something like this. We used to think resources mattered. We went through a period when we thought resources didn't matter. And we now we think resources matter, but only if they are applied in certain ways.

I think over 25 years, that's not a whole lot of progress. And the sad thing about this is that we have four literature reviews, three of which use the same set of steps, two come in one direction, one in another. The fourth literature review rejects two-thirds of the articles in the other three, because they don't meet the quality criteria.

The heart of the issue here is not whether the meaning is plus or minus on effects. That's why we have such large variants, and why we can't do a leap with the simple results. A research effort needs to go into answering that question. Why can't we get reproducible results? That's a different kind of research than just picking up a database. And I think we are going to have an opportunity to do that with new state data, where we have similar specifications on lots of different data sets to determine if we have comparable results. So I think that was a particularly bad example to use for progress in education.

Let me wander into some territory not covered by the report, and I think both Pat and Richard have done, and will make some opinions about why we aren't making more progress. Richard mentioned reproducible results. And I think if there is one basis of science that we need, it is that we obtain reproducible results. We cannot have good science without understanding why we are not getting reproducible results.

I think the second area that I think was also mentioned is there is really not a tradition in educational research or social research that theorizes. In hard science you have theorists and experimentalists. And there is a good reason for that. Without good theories, you do not have a scaffolding on which to accumulate knowledge, and it doesn't matter what the next question you ask is.

So that we need a lot more effort to discover and frame theories before we go into research in education. There is an example of this in the class size example that I provide. But I think it's the area that we really suffer, in that our research is not guided by theories, and is not enough tension to develop theories of why things happen.

I think probably it's important that there is no sure tool of knowledge researchers have. Education is as affected as much by what goes on before school, outside of school, as it is in school.

The federal government spends about $3 billion a year on research and development directed toward areas of children or youth. Education research is roughly less than 25 percent of that. I don't think we are going to be able to make great progress in education research unless we are able to solve similar problems which exist in the entire body of research.

And in order to understand education, you have to understand a much wider body of research and human learning than currently is thought. And one approach to this is a development science curriculum that would basically focus on providing the basic knowledge that people doing research on children need -- genetics, statistics, parts of economics, parts of sociology. But there is a base of knowledge developing which researchers in the field should all have. And I think the training issues in this area are fairly severe, and they need some reworking.

The absence of consensus panels in education really makes it tough to even determine whether we have consensus or not. We really need consensus panels. Very simple things, like what is the best measure of educational expenditures over time? Have scores gone up or down? There is the group of experts who have said this is what it shows, and this is where we disagree.

I'm so tired of seeing this NAEP chart that is 0-500 points and it looks flat. Well, I can make anything look flat if I can sort of address the scale. We need consensus panels so as there beings to develop a body of agreement about what we know and what we don't know.

Certainly, experimentation, but experimentation needs to be strategically framed they think. In the long run, we are going to need to depend on non-experimental analysis, because we don't have enough time and resources to experiment with everything. So experimentation should be geared toward understanding what assumptions we need to make in non-experimental analysis.

I think the Tennessee class size experiment begins down that road. It really tells a lot about how to specify non-experimental models. So that we need a body of research that is geared toward testing the assumptions in non-experimental models so eventually experiments agree with non-experimental models. That's the sort of holy grail we are looking for here. But that takes some real strategic planning to carry out.

We certainly need a serious examination of the major data collections, a look at the research centers. All these things should be on the agenda. One of the interesting things that I learned at RAND when I was doing military research is that generals need to be at the table when the research agenda is devised. You get a different research agenda with the generals at the table.

It focuses your research on the most important questions. And the education equivalent I think is having research centers connected with the chief state school officers, the Council of Great City Schools, registering with groups. But it really needs to be more of a defining of the