MS. BORKO: So, I took seriously the idea of five minutes and I want to just highlight probably a couple of points and then the assumption, I guess, is that we will come back and discuss those that are particularly intriguing or that we had some thoughts about the process.

I want to focus on the two major purposes of peer review that we have been hearing about today, one being the quality control issue, to ensure high quality, scientifically appropriate research being funded, and the second, the educational commitment, or the educational purpose, which I think has been mentioned a little bit less.

I guess, as a educator, as a researcher and as chair elect, president elect, or AERA, the educational purpose is probably particularly important to me.

I guess I see the educational importance for a couple of reasons. One is to promote the quality of future research. The next is to mentor the next generation of researchers, and the third is to shape the direction in which the field develops.

I think that, by mentoring and by providing good feedback as part of the review process, we can do all three of those things.

I am going to focus primarily on the issue of qualification of participants on the review panel, and I am going to frame this around the idea that we are talking about review panels. I know that is not the only way to think about it. Then, some of the processes involved in those review panels.

With respect to qualifications, I think there are three dimensions that we can talk about with qualifications.

My argument would be that people ought to be qualified in all three of these dimensions, the first being substantive expertise, the second being methodological expertise, and the third being knowledge of policy and practice.

Again, I think the ideal is really to constitute panels where each person has expertise in all of these areas.

I would argue that, to the extent we have diverse perspectives, we have diverse perspectives within each of those areas. Let me say a little bit more about that.

For example, in the area of substantive expertise, in education, that would, from my perspective, mean at least expertise in the subject matter, whether it be literacy, whether it be mathematics, whether it be science, and also expertise in teaching and learning.

I think it is really important for people to have both. In 1986, Lee Schulman first talked about the term, pedagogical content knowledge.

I think the notion of pedagogical content knowledge, the knowledge that we need about mathematics to conduct educational research in mathematics is different from the knowledge that the theoretical mathematician would have. They are certainly overlapping.

It is also different from the knowledge that a person who works, say, in areas of classroom management would have, and an area strictly of pedagogy.

So, to review proposals about mathematics teaching and learning, you really do need to know about mathematics, and I think you need to know about teaching and learning.

Pedagogical content knowledge is kind of the nexus of those two aspects of knowledge.

If you think about theoretical and methodological perspectives, I think everybody on a panel needs to have theoretical and methodological expertise.

Again, where the diversity would come in would be the diverse perspectives of the kinds of expertise. So, in educational research, it would be important to have people who come from a psychological perspective, people who come from a sociological perspective, people who come from an anthropological perspective.

It would be important to have people who bring research expertise as qualitative researchers, research expertise as quantitative researchers.

So, everybody needs some sort of methodological expertise and then, across the panel, I think we need to have diverse perspectives represented. That is probably enough to stimulate some conversation about qualifications.

In terms of the kinds of support and guidance for reviewers, I think this is a really important aspect of how the panels run.

I think Russ did a nice job talking about some of the ways in which we can do some screening in advance, so that the proposals that are brought to the table are proposals that meet a certain minimal criterion of appropriateness to the call and completeness.

I think I have also many years of being an editor of a journal, reviewing for journals and being chair of the publication committee for AERA.

So, I know there are always issues of how much screening do you want to do before you get to the panel of experts. That is one of those fine lines, but I think some pre-screening needs to be done, so that the review panel can really focus on the higher quality proposals and then decide among them.

I also like the way that NSF does its review panels, where you have a primary reader, secondary reader, and then other people on the panel who are responsible for knowing something about the -- some groups at NSF, I should say -- so, some other people on the panel who have the proposals at their disposal, have looked at them to various degrees, but are not necessarily providing written comments in advance.

I think what that does, it makes sure that you have some people who have devoted a lot of time and thought to a smaller number of proposals, but a group of people that is also prepared to talk about all of the proposals.

The numbers that have been suggested, I don't have any data, but to me, my experience is that 10 to 12 reviewers, maybe 20 proposals, is a number that you can get a handle on and a number you can deal with carefully in the amount of time of a couple of days. So, that is a way of thinking about the work load.

I guess along with the work load, the use of technology, I find what NSF is doing with fast lane, as a reviewer, I found that it is a very nice model for ensuring some collaboration and some communication among the reviewers.

The issue of multi-tasking, I thought, was an interesting one when Ed was talking about it today. My take on that is, I like the multi-tasking, and I think that the multi-tasking that gets done in the review panel, where people can listen to a conversation, one person is in charge of constructing a summary of the group conversation, and then that is passed around the table electronically, and then people can read it, revise it, and sign off on it, is a nice way of capturing the flavor of the substance of the conversation by the experts on the panel who are engaged in that conversation.

I do think that there is also a need for some kind of focused time on putting together those summaries, and maybe a way of doing that where you have both the real time collaboration, but also some quiet time to do some reflection, and maybe put some of the detail in those, would be a nice model to work with.

The other thing that I think is really helpful for reviewers is really adequate information, maybe what you might call professional development for reviewers.

So, really, information provided in advance to the reviewers, so that the reviewers understand the criteria for judging proposals.

Those are really different in different review panels and different for different sets of proposals, and we need to know that at the time that we are doing the reviews, as we are starting to think about what criteria should we be using.

I also think that expectations for the kind of depth and detail for reviews is helpful. One of the things that I found very useful as a journal editor was distributing reviews across the reviewers and, especially, distributing exemplary reviews, so that people have a sense of what the standards for review are.

I think the people then kind of aspire to higher standards, if you have some exemplary ones in front of you as models.

I guess the other thing is a set of very specific questions to guide the reviewers. Again, this is the situation where different panels have different sets of criteria.

The more reviewers can know in advance the criteria that the funding agency has, and the specific questions and rubrics that are appropriate, again, the more detailed reviews that you are going to get. I think that is probably it.