MS. FALKENBERG: Russ, have you thought about dividing the peer review process, so that you have one group of peers for the rigor and one group for the relevance?

What I am really trying to get at is, are practitioners going to be involved in any way?

MR. WHITEHURST: Practitioners are involved in the National Education Research Board, and it is the function of those practitioners to decide what the priorities should be, or to approve the priorities.

I assume they will be quite aggressive in focusing high priorities on questions that are relevant to education.

I think once you are dealing with the research proposals, from scientists, with all the technical content that is in there, the design aspects, the statistical aspects, the relevance to previous literature, I think at that point it is a mistake to mix practitioners into the process.

I think it is the questions themselves where it is very important to know what the field thinks, what practitioners in the field think are important.

MS. FALKENBERG: You could also push the people writing the proposals to write a proposal, but then write an abstract that could be read by a practitioner. It might help develop the capacity of the field.

MR. WHITEHURST: Abstracts of what we are doing that are understandable are certainly quite important. The funding announcements themselves, in most cases, take care of the relevance because they are focused on problems of, what preschool curriculum works best.

It is a very practical problem. Once you decide to compete for that money, you are already involving yourself in a question that a practitioner would relate to.

MR. BERKMAN: I am John Berkman(?) with the Consortium of Social Science Associations. I sort of want to extend on that question.

In Dr. Hackett's presentation this morning there was sort of this talk of the social impact of research projects.

I think there should be further considerations given to practitioners and representatives of advocacy groups on these review panels.

I think it would sort of illuminate the social benefit of potential research, that maybe some researchers would not look at.

MS. JONES: Vinetta Jones, Howard University. I have a question about the diversity of the peer review team, particularly considering that so many of the challenges that we are addressing in education are about diverse populations.

What is the mechanism for assuring that the people who would be on the peer review would reflect some of that diversity, since we know that research can be absolutely of high quality in a statistical sort of way, but not attending to relevant variables that people would attend to from various backgrounds.

MR. WHITEHURST: Of the 2,000 or so CVs that I looked at, the ethnicity of researchers was seldom evident. What we strive for is that there be deep knowledge of the problems that are being considered by the research competition, both in terms of content and in terms of methodology.

Our peer review panels last year, I think, were reasonably diverse, but diversity and ethnicity, or skin color, is not the goal or the byproduct of getting people to compete with deep knowledge.

MS. JONES: I ask this question because, on many panels, on which I sat myself, and others have expressed the same sort of thing, many of the questions that have been raised would not have been brought to the forefront, had there not be diversity there.

I think, in order to get to excellence, that if they are thinking of people as generic when it comes to deep knowledge in various areas, could be considered a mistake in searching for excellence.

MR. WHITEHURST: I certainly agree that we need people with deep knowledge of the populations that are participating in research at the table, because without that knowledge, it would be a mistake to consider it a balanced panel.

MR. REDISH: I want to make a comment and ask a question, also on diversity, but on a different axis. Seeing the Stokes quadrant that you put up, and just let me try to give you a little background, I am a theoretical physicist who, for the 12 past years, have been in education research, and whose current research is comfortably in Pasteur's quadrant with occasional forays into the Bohr's(?) and Edison's. So, this is not a personal comment, but a view from an overview from my experience in the hard sciences.

That is, if we focus too strongly on Pasteur's quadrant, we can really miss the major and most important learning how things work, that takes place in Bohr's quadrant.

In 1880, if some philanthropist had decided that Babbage had the greatest idea and was going to make the revolution of the next century, and supported practical applications for improving accounting, we would not have had the same computer revolution that we had. The current industrial revolution arises out of the work of Bohr and Kaiser and Heisenman(?) and so on.

So, my experience in education research, and even in the research funding is that it is very strongly driven by transition to practice.

Where does the diversity come from to support the fundamental learning research that needs to be done in order for us to really understand what is happening, without any concern for immediate pay off, but rather, for deep understanding.

MR. WHITEHURST: Your question, which is a good one, suggests that I wasn't as clear as I should have been in explaining what we are doing at the institute.

It is certainly not my belief that education, research that is relevant to education, should only be applied research.

The fact is that there are at least three major funding agencies, federal funding agencies -- NIH, NSF, that are junior partners to the Institute of Education.

Their goal, NSF's goal, is very different from the Institute's goal. NSF's goal, in general, is field initiated research that gets at the joints of nature, including human life.

NIH's role is more health related, but it is funding within NICHD, for example, that is generally focused on more basic research.

We all don't have to do the same thing. We all don't have to be spread across all three quadrants. In the institute, we have some funding programs that are explicitly in Pasteur's quadrant. We have a program in cognition and student learning that has immediate practical goals. We try to force people who are doing that research to get into the classroom in the course of the four-year award.

We like to see the appropriate role of the research agency in the U.S. Department of Education to do as much as it can in terms of translational research, to do research that is relevant to practice.

We won't be able to sustain that research going down the road unless there is not also a commensurate and considerable investment in research in Edison's or Bohr's quadrants.

MR. FLETCHER: Russ, if I can just jump on that horn, the department also funds research through the Office of Special Education and Rehabilitative Services.

The fact is, I believe your budget is probably the smallest of any of these four programs for research.

MR. WHITEHURST: Well, the budget -- that is nice. I remember, Jack made a presentation at AERA last year in which he said it is not that education research is broken, it is that it is broke. Maybe a little of both.

The Institute of Educational Sciences has requested roughly $140 million for research programs, a budget of $85 million for statistical studies, an account budget -- because it comes from other programs in the department -- of roughly $50 million for evaluation studies.

Clearly, we need more money. The budget request for 2004 has a $45 million increase in it from the 2003 budget. We hope to get that.

My judgement is that the capacity in the field has to be brought along. If Santa Claus showed up and gave the institute a $500 million check for last year, I am not actually sure we could spend it well.

I do think a $45 million increase could be spent well. So, if any of you apply and think that is a good idea, that is an opportunity.

MS. CHIPMAN: I am Susan Chipman. I am with the Office of Naval Research. Within this context, I was once the assistant director of the National Institute of Education for learning and development.

I was just curious, when you talked about the survey of superintendents, whether they were actually asked to identify any new research that they had actually heard of.

Back in the old days, for instance, somebody had done a study to find out where we should publish things that math teachers would read it, and the answer was, they don't read anything.

I always thought if you wanted to reach people, maybe you could plant things in Time Magazine or something.

MR. WHITEHURST: Actually, the survey went into some considerable detail about the sources of research-related information for each of the different audiences, and sought suggestions from the superintendents over the several audiences that were surveyed, as to what the institute should be doing to make information more accessible to them, and in what forms they would like information.

The one theme that occurred over and over again is that people in general would like relatively brief one or two pagers, that abstract the results of research, that they would like somebody doing that abstracting that they can trust, because there are many places you can go for two pagers and they often say different things on the same topic.

A frequent source of information is through their professional association and newsletters and update letters that are published by the professional association.

If the institute wants to do a better job at disseminating research, that it needs to form partnerships with the organizations that produce the paper that is actually read by teachers.

I think that was a very important message. They also would like to have more information available on the web, again, edited and brief, and useful, relevant to them.

MR. DE HAAN: Could you talk just very briefly about the idea of supporting research on teacher preparation, that is, university teacher preparation?

MR. WHITEHURST: We have two ongoing efforts that focus on teacher quality and preparation. One is our research funding program of teacher quality. The announcement is out. We haven't funded anything yet.

The announcement is largely focused on professional development rather than pre-professional development.

On the other hand, we have, within the evaluation center, a project that will be announced some time this year, that looks at teacher preparation issues.

We are interested in alternative certification. That is a title that covers many different things. Our challenge is to come up with particular routes for the certification of teachers that have some internal consistency in terms of models, and where we can test the outcomes and effectiveness.

There are certainly great challenges in teacher preparation in traditional colleges of education. There are ongoing discussions about what can be done in that arena, to get a better sense of what is going on and how it can be changed and what the flexibilities, that involve the developmental education community as well as the research community. So, this is a very important topic, and I think we have got our eye on the ball.

MS. LAGEMANN: In the panels you are now putting together, what percent of people are you looking at from which disciplines and which types of institutions? What kind of balances are you looking for between schools of education and disciplinary departments?

MR. WHITEHURST: Both I and ASI and the panel chairs are paying considerable attention to whether the individuals on the panel are really at the top of the field in the topics that are under consideration.

Last year -- I actually haven't done the statistical studies yet, but approximately a third of our panel members were from schools of education. That is just a rough guess. It is frankly not something that we coded.

MS. LAGEMANN: What about the fields?

MR. WHITEHURST: The fields, I think, depend on the competition. If you look at cognition and student learning, the reviewers are largely cognitive psychologists and neuropsychologists, because that is the nature of the competitions.

If we are forming a panel for teacher quality issues, I think there will be people who are much more diverse and interdisciplinary. So, it is related to the topics.

MR. WISE: Join me in thanking Russ for an informative presentation.

[Applause.]