MS. PETERSON: I notice that we are billed as perspectives from investigators. I can say that I have been an investigator, and am an investigator, and have been a reviewer for different agencies, most recently the National Science Foundation and the IERI. I have also been a journal editor for six years, and was president of AERA.

Today, however, I am actually going to speak on behalf of a group of deans. I was asked to speak on behalf of an alliance of deans of the top education schools.

This group includes Harvard, Stanford, Teachers College, Columbia, UCLA, University of California, Berkeley, University of Michigan, Michigan State, Emory University and my own school of education at Northwestern, the School of Education and Social Policy at Northwestern.

This group of deans has been meeting for the last three years for the purposes of sharing information, our own learning, and also working to improve our doctoral programs and our own training of educational researchers.

At our most recent meeting, in January, we decided that we wanted to start speaking out as a group on issues that concern us.

One issue that concerns us most is the current national discussion about the quality of educational research and how to improve it.

Consequently, we are delighted to speak here today, because we think that peer review is a real key to improvement of the quality of education research.

So, what I am going to talk about today is actually some things that we agreed upon in a conference call, and I will let Ellen off the hook because she was actually not in the conference call. So, Harvard was not represented. She may want to comment about my remarks at the end.

Teachers College, Columbia, was also not represented, because the president, Art Levine, was in Poland and could not get a connection from Poland. The other deans that I mentioned are represented.

We were asked to comment on the purposes of peer review. Our group sees three major purposes. First and foremost, we think that peer review should improve and maintain the quality of education research by recommending funding for only those proposals that meet the standards of scientific research, and we agree with the standards that you have laid out in the report. It is an excellent report. So, we won't try to go into trying to define scientific research and standards of evidence. We will use yours.

Second, we think that peer review should envision and shape the field, where the field should go. It should not only maintain high quality standards in support of what you might call normal science, but we think it should also create opportunities for risk taking and innovative educational research.

Here, I think Professor Hackett raised an interesting point, that often these innovations are intentioned with the high standards of rigor and evidence that most often represent the methodologies and the standards that we currently have.

Third and finally, and actually it might be most importantly, as Hilda pointed out, we think peer review should serve an educative function, and particularly for junior researchers in the field.

This means we think there should be a clear mechanisms for peer review to provide substantive feedback to the proposer, typically in the form of written comments.

This lack of detail and quality in reviewers' comments actually was mentioned by Russ and was identified in a study that Diane August did in 1998, which she and I are going to talk about tomorrow.

We think this probably still may be a weakness in some of the peer review processes, that is, the lack of detail that proposers do or do not give back.

You often get back ratings, that you get back, but a rating might be a four on a seven-point scale, with no written comments as to why the four and what the weaknesses were.

Well, to achieve these three purposes, our deans group actually likes the model used by NIH, the study section, study group model.

This was actually a recommendation of the 1998 report that was never acted upon. The recommendation was for OERI to develop such a model.

One of us, Amy Dorr(?), who is the dean at UCLA, actually talked about being on the NIMH study section for five years, and she was chair for two years.

She said it worked really well, because there is, for one thing, a well-articulated process that included training of the reviewers, talking about the initiatives, and norm setting by the panel.

In addition to reviewing research, Amy's group also talked about what the field needed and helped shape the direction of future research.

She, herself, commented on the importance of having good staff support as well as a good chair. She modestly didn't point out that herself was probably appointed chair because she was a great chair and they knew she would be.

She said that she felt a good chair of the study panel needed to be open but also be firm and realize when folks on the panel needed to give up on arguments that weren't really going anywhere, which is a good thing to do, but it is sometimes difficult to move the discussion along.

Some of our deans might think that is important at faculty meetings, too, to give up on arguments that aren't going anywhere.

I just want to mention my own experience. I think it comes closest to this. I was reminded of it when Professor Hackett was talking.

When I was on the faculty at the University of Wisconsin, Madison, I served on the graduate school research committee.

Many of you may know that Wisconsin has a huge amount of money that is called the Warf fund, which is generated because somebody who is on the faculty invented warfarin, which is rat poison.

They generated a lot of royalties. The principal investigator gets to keep royalties, but the university also got a lot of royalties from this. They also invented the vitamin D additive for milk.

At the time I was there, which was 1976 to 1987, they had a huge amount of money that they were giving away. The way they did it is, they would hand select what they saw as outstanding researchers at the university and then they would create these panels.

I was on the social science and humanities panel which, at the time, really had some stellar people on it. It actually was the best educational experience of my whole career.

It functioned like a study panel. You were on it for three years, and you met every Wednesday afternoon all afternoon and every Saturday morning. So, it was like giving blood.

It was just outstanding. What we did is, we would review all the proposals that were made, and any investigator could ask for money.

Typically, each department would have about 20 proposals, and you reviewed a department that wasn't yours. I usually got psychology because I actually was not in the psych department. I was in the educational psychology department.

It was amazing that you could talk across these. We reviewed art and we had to talk about what was the scholarly contribution in art and psychology and actually the school of agriculture. Marta Tanda(?) was on the panel at the time, and she is a demographer.

I think that one of the important things is that, being together in a long-term kind of way and developing an alliance and being able to discuss and really ask each other hard questions, and coming to understand the expertise on the panel, and the expertise was quite diverse.

Anyway, I think that that kind of process can be really excellent, and it certainly was an educative process for me, because I learned a lot not only about other disciplinary perspectives, but also about how to make an argument and how to explain to somebody else why a particular piece of research is important and why it is well done.

You are trying to explain to somebody in English why a particular psychological study in the area of cognition should be done. It really sharpens your own ability to be able to articulate rules of evidence and say why research is important.

Just in conclusion, we think that, in talking about how these panels might be put together, we see peer review as done by our peers who are researchers.

We would see that a review panel of the kind that Amy served on or that I served on for the Warf research initiative, should collectively be knowledgeable in a wide range of methodologies, and also have a wide range of specialized expertise.

No one panelist, I think, as has been pointed out, can have all the expertise. So, it really relies upon, you have this distributed expertise, and being able to develop a discourse and a learning from one another, I think, to make the decisions at the panel.

We think the panel should at least have some researchers who understand the importance of application. I don't think necessarily all of them have to, if you are thinking of the panel as having distributive expertise.

We think there need to be mechanisms for the panel to be held accountable for providing substantive reviews and supporting the ratings that they are given.

We also believe there need to be clear guidelines about the importance of the panels in funding decisions, and the decisions should be consistent with the priorities of the agency and also with the recommendations of the panel.