MR. DODGE: Thank you. Certainly, I want to applaud the education scientists who highlight the goal of improving the quality of research on education by improving the quality of peer review.

I am reminded of sitting on a panel on education a number of years ago here, in which the topic of peer review came up.

One of the members of the panel was an esteemed United States senator, who will go nameless. He said, after applause, well, since we all know that all you need in order to educate a child is grandma's wisdom, then to discern a good idea in education, all you need is grandma.

I suppose, if you believe the first, you believe the second. On the other hand, if you believe that there is a science of education, then I certainly hope you believe that it is equally as important that those scientists be at the core of the review of that research.

I apply for grants and I review grants. I guess that is why I am here. I regularly review grant applications at NSF and, more so, at NIH.

I bring to the table today the goal of trying to draw in, and to reach out to, the disciplines in this arena.

Much is said about how much research related to education is irrelevant. I suppose that is the research being done by the disciplinary-based research.

My goal is to bring in the part that is sociology and economics and psychology, human factors, engineering, neuropsychiatry and genetics, into the mainstream of education research.

We are already doing research that is related. Why not draw them in. That will improve the quality of peer review. It will also improve the quality of the research they are doing and make it more relevant to education. They go hand in hand.

One can't get those folks as peer reviewers without attending to their research, as researchers. So, I want to address briefly the question of how to accomplish that goal.

First of all, it is not about pay. It is not about pay. Paying reviewers $2,000 a day will bring them to the table, but it is not about pay.

It is about quality and importance and having an impact. So, giving ratings long distance electronically and not knowing which grants get funded, not knowing whether you are having an impact, won't bring me to the table of grant reviewers. Something else would.

The NIH model is a terrific model. We don't need to reinvent the wheel. I want to say four positive points about the NIH peer review model and then one problem, at least from my perspective.

The first positive point is the essential dual system of programs, program officers, program guidance on one side, and peer review being independent from that, largely separate. Even within an institute, it is largely independent.

That is absolutely essential for me, as a reviewer for me to want to be part of the process, to resist political pressure on some of the project pressure. It brings integrity. It keeps it pure.

I have to admit, though, contracting it out gives me a lot of pause. I was one of the potential grant reviewers that received a packet in the mail from the private firm this spring, reviewing new grant applications to the Institute on Education Sciences.

It turned me off, in that I was asked to give numerical ratings, not to be a part of any process, give it to some private firm.

I didn't know who they were. I didn't know what they would do with them, how they would act, whether they were, in a sense, knowledgeable about the field at all.

Maybe that is the way it has to be done initially, but I would hope, in the long run, it could be grown from within.

The second point, investigator initiated grants are essential in order for peer review to happen, in order for the field to move forward. For me, that is imperative.

It is as if the investigators are not trusted to know what to research, if there is not an investigator-initiated grant program.

Now, of course, I think that a valid portfolio is necessary. I think it is still possible by providing broad guidelines rather than narrow guidelines on what to study.

Gosh, I don't have to give the positive examples that NIH could give a lot better than I could, about the serendipitous as well as planned positive outcomes of investigator initiated programs of research, whether it be in cancer or in my own area of violence.

That is a way to resist pressure from Congress, from the public, resist baths, quick fixes. It is the strongest program of research that we have in this country.

Most intramural programs, even with NIH, have their role, but NIH research really requires that investigator-initiated approach.

Third point, how to get the highest quality peer reviewers in order to get the highest quality reviews, reviews at all.

Again, same as NIH, which has standing study sections involving leading scientists, led by leading scientists.

I personally believe that we need to resist the pressure to include the lay public and professional teachers on those scientific peer reviews.

Therefore, it is extremely important that it belongs at the level of prioritization about broad program areas, not evaluation of the scientific quality of a specific grant application.

Blending the two in the same room, for the purpose of scoring the scientific merit of an application, I think, discredits the process.

Nobody should be in the room scoring it unless everybody is comfortable that there is a tie score and that person decides the outcome. That is why that person is in the room, is to simply decide the outcome in case of tie scores.

If the purpose is to say, this is the scientific merit, I don't think the lay public and professional teachers are qualified to do that.

Their input is maybe more important than those of researchers in the entire scheme, but their input is important at the level of prioritization, of funding and program areas.

Fourth point, process of peer review itself and how the groups work. Gosh, it requires interaction, not independent work.

Low reliability of scoring is based on totally independent numbers given. That is the system in point three, we heard today.

What happens in a peer review study section is an interaction and education of each other. I presume that is why we have juries in courts that interact with each other, rather than independently getting 12 votes.

There is nothing more gratifying for a guy like me than to hear a guy like Jack Fletcher give a tutorial on learning disabilities at the study section. I learn, as a peer reviewer.

I can make a better decision. I may not agree with his decision, but that interaction is just crucial to the process, and I would hate to see it given up. It is why I come to the table. I learn something. I think the result is more credible scores that are the outcome.

In the study sections that I have sat on, I would wager that the ultimate scores that are given have a much higher reliability than .3. It is not truly independent scoring, but it is a result of this process.

Finally, one major revision of the whole system, I think, is in order. If you look at the Department of Health and Human Services, we now have separate streams for science and practice/policy.

Even the Department of Health and Human Services, most of the money they give out is for practice and programs, and then there is money for science, but completely different systems of evaluating those are in place. I would propose that peer review needs to be brought into both parts.

We now have block grants for juvenile justice, in health and certainly in the Department of Education we have block grants. It is as if science stops, practice begins.

The policy science and practice science, dissemination of science is absolutely crucial and has an impact, and peer review of that science is also crucial.

One quick example and then I will close. Research on class size and education. There is classic research done in the state of Tennessee showing the effect of class size, good research, I wouldn't call it basic, I wouldn't call it dissemination.

Then the outcomes of that, though, are supposed to be implemented without any further science. That needs to change.

Science involves evaluating the costs and benefits, or evaluating the system level impacts. If we were to reduce class size immediately in this country, what would that do to the quality of the teachers in the classrooms, and what would that do to the buildings, et cetera. Those are research issues, not policy only issues, or practice issues. Those are research issues of policy science and practice science. Likewise, then, we need peer reviewers to get involved in that as well.

Within education, I know some about the safe schools and healthy students program, which gives out $500 million a year.

There is no peer review, there is no research, there is no even rigorous evaluation. A peer review system could be brought into that component, if the science and the practice are blended somehow. I think the quality would improve.

Let me close by just suggesting one final point, that the optimal portfolio of education research include not only investigator-initiated grants, certainly some that are responses to requests for applications and program areas that are promoted by program officers.

Second, a limited number of peer reviewed contracts and grants to answer specific questions.

Third, increased emphasis on research training related to education, if we really want to fund the system of peer review in the long run.

By my way of thinking, that involves -- at least one component of it -- would involve discipline-based education, research education in sociology and economics and psychology or whatever, but with a heavy dose of how to make it relevant, how to cross talk across the disciplines.

For a psychologist to understand the economic impact of a particular program might be essential, or for an economist to understand what is in side the black box might be essential.

That would also help researchers of the next generation provide a better bridge from the basic science to the policy.