MS. LAGEMANN: A lot of stuff was put on the table this morning. Four things stood out to me. One point was that the peer review is not the answer to everything.

A second is that the goals of peer review. One goal, we can use the phrase, that this is not really about pay. It is really about other things. I think pretty much everybody agrees on that.

Clearly, it is about the field capacity, creating the discipline, but I think the goals is more the discussion.

There seems to be considerable disagreement among the panelists about the role of educational expertise as opposed to discipline expertise as opposed to the perspectives of practitioners. I don't think people wind up on the same page about those issues.

The fourth theme that seemed to run through the presentations was this issue of having study panels that are sustained and supported, as Hilda put it, where there are clear guidelines, people get to know each other, they have conversations and that.

So, those were the themes that struck me. There may be other ones that struck all of you, and the floor is open to the committee, or to anybody else who hasn't fallen asleep yet, because it is 1:00 o'clock and we have just eaten.

What about this issue of the balance on panels or several panels? How do we get all the different kinds of expertise we need represented?

Ken Dodge was talking about disciplines. Hilda was talking about pedagogical content knowledge. These are not the same thing.

MR. FLETCHER: What is confusing me is this idea that I heard from Hilda, which was that a single reviewer needs to embody all these things. That was what you said. Let me just elucidate that.

The idea that all reviewers had to have pedagogical expertise, versus an idea that you could compose a panel where you would have diverse perspectives embodied.

To give a really extreme example, what do you do with a grant where somebody proposes to do brain imaging before and after an intervention. Where are you going to find a brain imaging researcher with pedagogical expertise, for example, but how are you going to review the grant if you don't have that perspective or that discipline on the panel?

MS. BORKO: I think those are good questions. So, let me back up. Let me kind of reinforce some things I said again, and back off a little bit from one.

I think it is really important that people have both substantive expertise and methodological expertise, and I really don't want to fudge on that one.

That is going to speak to this issue of whether practitioners should be on panels, at least indirectly.

I think you have a good point, Jack, in terms of everybody having expertise in the subject matter discipline, and expertise in education.

Depending on the nature of the review panel, and the nature of the kind of proposals that are being reviewed, it may be that that is too idealistic and we can't really find people who have that combination of expertise and, in that case, it would be important to make sure that diverse perspectives are represented on the panel, and that, among the community of reviewers on the panel, you would have to make sure that all of the perspectives are represented.

I think that some panels and some charges, some calls for proposals, actually the range of proposals is narrow enough that we could expect everybody to have both the educational and the substantive expertise and, in some, that is really not going to be possible.

MR. DODGE: To pick up on that point of this variety of expertise that I think is necessary to evaluate fully a number of proposals, you don't get to the optimal outcome simply by saying, okay, we will have a methodologist rate it and we will have somebody with classroom experience rate it, and somebody who has got this brain imaging experience rate it, and then take the mean of those ratings, as if those people were not talking to each other at all.

Instead, the context of the peer review, then, becomes crucial, so this diverse group gets together and talks.

This brain imaging person says, well, they are competent at this or not at that, or they are not worrying about X,Y or Z. The person with intervention experience can come in and say something else.

The outcome is not the average of the two. The outcome might be an unpredicted outcome from anybody.

You can also get additional expertise from outside reviews, which often happens in NIH reviews. Particularly if you want deep knowledge in an area, you go get a review or a consultation from somebody who is not sitting on the study section, but can provide that expertise.

All of that tells me is that it is not only -- the beauty of a peer review system is not only constituting a bunch of good individuals. It is worrying about the dynamic and worrying about how they interact.

I think the NIH falls to the scientific review administrator. I have been on good panels, interesting panels, as well as problematic panels, and I think that is one of the reasons.

MR. HAKEL: I would like to add to that, if I might, just one point. I think it depends a great deal on when you go out to that specific external reviewer, whether that reviewer understands the context, understands who the audience is so that, in fact, you get a knowledgeable review.

One thing I find problematic sometimes is in asking people to review only one proposal with the general kind of criteria, the given impact on society, the contribution to knowledge.

Those are so general, that it could be looked at and zeroed out, and then you would have lost something that should be been funded, had the reviewer understood more about the context in which that was being requested.

MR. HACKETT: Just sort of a comment in response to the question you just threw to the floor. You said two things I want to talk about.

One is, I agree with Ken that the idea I put out on the table for this is that, to establish the review process in a way that embodies what Don Campbell called, 35 years ago, I think, the fish scale model of collective omniscience, so that you have overlapping fish scales available on a panel.

You are not going to get it all on one person and you don't want much space between the scales. So, you have a fish scale model, and then a dual staged review with ad hoc mail reviewers and a panel as a way to get to it.

If you decide you absolutely have got to have a person who has done ethnography in an Asian community, you find the one best Asian ethnographer and make sure that person gives you an ad hoc review.

Coupled to this is the idea that you try to establish it so that panels don't make decisions. This is the best piece of advice I got from my predecessor with 24 years on the job.

He nearly lost his job because he had forgotten the panel is an advisory panel. So, ultimately, someone else makes the decision. They want the panel's best judgement, which is why the scores are less important than the words.

You want their expertise and advice. Leave aside someone later to place it on the pay line above or below.

The second point, I will try to make more quickly, this issue of citizen involvement is really a difficult one.

I think there might be something to learn from the National Institute of Health's struggling with this for a long time.

They talk a lot there about translational research, bridging bench and bedside. They have had a lot of engagement with activist groups, they have a lot of disease lobbies and such, and tried in a systematic way to find a point of entry for that influence. I think education might be in a similar spot.

I think it is not going to be a choice whether you do or don't want citizens in. I think with GPRA, they are here. The challenge is finding a way to use that.

MS. BORKO: One thing that I want to build on, I agree that citizen participation, at some point in the process, is important.

I think one thing to keep in mind -- this goes back to what I was saying earlier about the NSF model and how you do the summary statement -- one of the things I have noticed on review panels is that, when practitioners or citizens are asked to give a summary statement, it is a very difficult task for them.

If we are going to continue to have shared responsibility for putting together the summaries that capture all the issues, I think we have to be careful how we allocate those responsibilities within a panel situation. That is just another part of the complexity of that.

MS. LAGEMANN: I think there are some people on this panel who don't think practitioners or citizens should be on the panels at all.

MR. DODGE: Let me clarify that. I think that expertise is needed in scoring a proposal. Representation from constituencies is needed in prioritizing funding. Those are two different, important functions.

I am not qualified to evaluate a proposal on brain imaging. Just because I have a PhD, and it might come to my study section, that could be a problem.

I am not sure I ought to be voting on that kind of an application. So, too, if I come into the room as a representative of a citizen group. That doesn't make me expert in the scientific characteristics of that application. That is what I mean. I think that is important.

It does not mean at all that my role as a citizen or my role as a classroom teacher or a school principal is not important in the entire enterprise.

It is crucially important in deciding what kind of research, broadly defined, ought to be done.

MR. REDISH: The issue seems to me to conflate two separate things that have to happen in peer review, that I mentioned in my remarks, namely, the field defining itself and the field negotiating with the society.

I have never been on a panel that had either of these type of people on it. I would be very interested to know what people's experiences are. Does this work? Are these people useful?

It seems to me it is useful to keep these two issues separate, as Ed just mentioned. These certainly are important issues, but these are issues that, in my experience, have always been brought to the panel by the program officer, who was the intermediary. I don't know how it works when they are actually there on the spot. I would be very interested in hearing someone comment on it.

MS. PETERSON: I have been on panels with practitioners, OERI panels primarily. I think it only works well if the person has a deep understanding of research and research methods.

Also, I guess I think it is important that people actually be actively involved in doing research. I do think that the peers should be researchers.

That doesn't mean that they couldn't be a practitioner who is also actively involved in doing research.

I think I shifted my position over time. I mean, 10 or 15 minutes, I probably thought, oh, we should have practitioners on the panel, but there is too big a gap. We really are reviewing research proposals, and we really do need to have a substantive specialized knowledge and understanding of research.

MR. HAKEL: I think this conversation illustrates the tension between rigor and relevance in a way that really has a sharp edge to it.

I think it is incumbent on us as scientists to be able to reach out as much as possible and, given that part of the agenda for this meeting was supposed to be educative function for peer review, to think beyond the immediate peer group of scientists to also peer practitioners, if that is not a total oxymoron, and to think even more broadly to the public as a whole.

One of my fears has been, all along, that rigor can really drive out the interesting and useful and appropriate kinds of things. If you push rigor too far, it becomes rigor mortis.

You can have dead ends in research, and part of the concern, I am sure, particularly from the science policy people is, how do you know, if you are not a scientist particularly, if you are not in this community, whether this should be continued, and by what criteria and by what evidence.

If you want evidence based practice, where are you going to find it? There are huge holes in all of these fields.

We have not defined the constructs that we are trying to impact in terms of how do you know it when you see it, or how would a politician know it when he or she sees it, and yet, we need to get there.

So, the whole process and the kind of series of steps in terms of having procedural justice as a document is reviewed, and how that fits into an overall system, is what really makes this so complicated.

MR. DODGE: I think rigor is extremely important and relevance is extremely important. It doesn't fit for me to caricaturize the scientist as embodying rigor and the practitioner or the citizen as embodying relevance.

What we really want to do in this whole game here is figure out how to take something from basic science or our basic knowledge about the way human beings are, and translate it into something relevant to have an impact.

I hope neither of those two persons, if they are both sitting on a panel, would represent only one. As a scientist, I have got to evaluate applications for their promise of translating the rigor or the basic knowledge into relevance.

Likewise, if I am representing a consumer group, if that is what it is, I have to be able to see what rigor, what base ideas and what methods could be used that could be relevant.

I am not sure -- to me, that over-simplifies it, to say that all relevance is lost if you only have scientists on the panel.

MS. BORKO: I want to come back to Ed's point, I think, on the role of the panel and being advisory and giving feedback, and not making decisions.

I think different panels have different roles and different charges, and the question of who should be sitting around the table on a panel is really tied to the role of the panel.

If part of the role of the panel is advisory on both relevance and vigor, and if part of the role of the panel is to give as complete a perspective as you can, then I think a diversity of perspectives does make sense on the panel, again, with the caveat of how the conversation is captured and who is doing the capturing of the conversation.

Another thing I think is important is how much time is allotted to discussions of each of the proposals on the table.

If you have 15 minutes or 10 minutes to discuss a proposal, then it is very hard to get a conversation that is going to deal with all the different aspects.

If there is enough time to really view all of the different aspects, if there is enough time to really think about what is the relevance, what is the rigor, how do they go together, what are the trade offs of different kinds of decisions that this particular team has made, when they put together the proposal, then I think the mix of perspectives at the same table enhances the conversation.

MR. REDISH: I want to press Ken a little bit on something that he just said that I am not sure I heard correctly.

When I think of myself on a panel, when I think of myself in my role as a scientist, I like to think that I am focusing on the construction of new knowledge, and that the issue of impact or practicality comes out of my thinking of myself as an educator.

You seem to be mixing those two together and I would like to separate them, because I am very much concerned, as a scientist, about the creation of new knowledge, some of which may have long-term impact, but not obvious impact.

MR. DODGE: What if what you are trying to discover is knowledge about how to have an impact.

MR. REDISH: Sure, but in order to discover that knowledge, you have to understand how the components of the system function.

That is very different from knowing how to manipulate or control those elements. So, I think it is really important to have those.

MR. DODGE: I am with you that there is a crucial role for the basic research in areas related to education, not only research that would have an immediate impact. I am with you 100 percent there.

What I am also saying is that, to me, it is not the case that there is science and then implementation. There is a science of implementation. There is a science of practice, there is a science of policy that doesn't answer all the questions, but it most certainly is underrepresented in our field today and is, I think, an area of high priority.

MS. LAGEMANN: Partly what is called translational research, where you take the findings of research and you translate them into the kinds of things that practitioners really need.

One of the things that worries me about the conversation these days is that there seems to be a sense that we need science, and then it will improve practice. It is missing a whole layer of steps in the middle that we need.

MR. HACKETT: I want this understood in the context of trying to explore the issue, not start an argument, because I am not really sure where I come out on this, but I think if we raise the issues, we can try to figure out what to do with them, and I agree with what you just said.

In engineering lately, there has been interest in what is called participative design. The notion is that you don't design a technology and then figure out how to make it suit the users later after it is done. You would rather get them involved right at the headwaters of the design process.

If you think of it that way, then involving practitioners and lay persons at the headwaters of research that is intended to influence practice, it might be a good idea, if you can figure out the right strategy.

The second element I have got in mind here is the NSF criteria, the one that looks at societal benefit. In precisely the same way that practitioners may not be the best judges of science, scientists may not be the best judges of practical impacts.

You may need practice experts in the room to make the arguments about what benefits your stuff is going to have down the road.

If I can have just a second to tell a story, I was on an NIH study section that was one of my more frustrating experiences, because everyone loved this one proposal, to look at how people whose children had genetic diseases understood the information given them by physicians. It was over a $1 million proposal.

I was the only person in the room troubled by the idea that there was no group that didn't have genetic diseases that were having medical things explain to them, to compare whether genetic information and other medical information differed in understanding.

This one treatment, no control design, troubled me. No lawyer, no doctor, no researcher in the room but me. I wrote a minority report, because everyone was in love with it. I share your frustration.

MS. LAGEMANN: other comments, questions? What about, Hilda, in your comments you talked about -- I am going to change the topic unless someone has something. In your comments, you talked about the peer review process being somewhat a mentoring process.

If you see it that way, how would you think about that coming about. Do you have any ideas?

MS. BORKO: I talked about two aspects of mentoring. So, clarify for me which one. One is the mentoring of the proposal writers and the other is the mentoring of the reviewers.

MS. LAGEMANN: First, mentoring of reviewers.

MS. BORKO: Okay, I guess a couple of things in mind. One was the notion of providing for reviewers exemplary reviews, so that people have a sense of what the standards are for reviews, the kinds of issues that should be dealt with in reviews, how extensive you want the comments to be.

I guess that would be one thing. Another is having clear criteria for what the call is, what the proposals -- what the call is that the proposals are addressing.

There is probably -- I haven't thought this through. There is probably a mentoring that would go on in the actual panels.

So, having panels composed of senior scholars who have been on a lot of panels, and a number of more junior scholars who are up and coming, who need to be kind of introduced into this whole proposal review process, to learn what it is all about, and to learn by doing.

So, if you have a panel of too many junior people, then you don't have the kind of mentoring, but bring a couple in for each panel. That is just off the top of my head.

MR. REDISH: Maybe it is because I am interdisciplinary and coming to these panels from the outside but, for me, reading the proposals has sometimes been an immensely educational experience, to see what people were thinking about in the field, to have new ideas, additional areas that I haven't looked at.

I found just reading the proposals and then, of course, the discussions afterwards, to be immensely educational and a very large part of my professional development as an education research. It is an extremely valuable experience.

MS. BORKO: I am going to have to remind the audience that sometimes you are going to put on a panel people who have won awards.

So, one of the criteria for being on a panel and being a reviewer is that you have to have, in the past, gone through the process. Do you think that is the kind of thing that is important to do, given that people should be on peer review panels -- do they have to have won awards?

MS. PETERSON: I think sometimes they do that because they know that they can get people to be on it, because people feel like they need to give something back.

Being in the life sciences, I know several people who have said that, that they don't really want to do this, but on the other hand, they feel that it is an obligation.

MS. SCHNEIDER: Can we get beyond that, though? I really want to talk about it as an ethical issue and politics within this whole mentoring concept.

Do you think that is a worthwhile criteria? Do people have to be funded to sit on panels? What would be your take on that?

MR. DODGE: I think it is a good indicator that is one indicator -- I can't think of one indicator that is absolutely essential, but that is a definite plus.

It doesn't have to be by that agency, but they have gone through the writing of an application and the review of it, maybe successfully, at another review panel. I see that as a plus for sure.

MS. SCHNEIDER: I am asking it in the context of, if we are mentoring people, who are we mentoring and who are we asking to serve on these panels?

If we are asking junior people to serve on these panels, how are we going to distinguish which junior people they should be, and what are the criteria that we should use if we are going to put junior people on the panel.

MR. DODGE: The panels that I have sat on are a lot of work, and I would not advise an untenured professor to sit on one regularly, maybe once, one time only as a special occasion to see, like everybody has to go Las Vegas once. I would not recommend it as a standing commitment, not until they are tenured.

MS. BORKO: Let me clarify what I mean by junior, because I am not thinking about the person who is two or three years into their academic career. I am thinking of junior in terms of junior on a panel.

As people receive tenure, they are still pretty junior. I don't think it is real useful to the field to have somebody sit on the panel once and then never again, then mentoring is something you are not going to able to --

MR. REDISH: What are you talking about mentoring, mentoring them to be better researchers or mentoring them to be better reviewers?

MS. BORKO: In this context, I would say mentoring them to be better reviewers, so we could create a pool of --

MR. REDISH: So, they could be very senior and still need mentoring to be a better reviewer.

MS. BORKO: Yes, and that is why I say junior as really being junior in the process of becoming a reviewer.

MR. REDISH: I think it works the other way, too, in that I certainly know junior colleagues who are five years into their academic appointments, who have not been able to get funding, and then have sat on a panel and now understood what were the criteria, and have become better proposal writers as a result. That can be very valuable, too.

MR. FLETCHER: I think it is always a matter of balance. I don't know why you wouldn't use study section experience to mentor and train up some really promising people in the field.

I mean, in my own case, I was on a standing panel four years out of graduate school, and it was the defining experience in my career as a researcher. I left that committee knowing how to write grants.

MR. HAKEL: I think it would be fun to do a little experiment in which silver backs nominated our best recent graduate for panel memberships, and they replaced us entirely.

I would bet the rank order of the decisions wouldn't be any different in terms of the panel decisions. This is my bet, and the rationale to support them would be just as articulated, if not perhaps more so.

MR. FLETCHER: I also think that people who are closer to their tenure and their training have more association with new ideas, and I think it actually may -- it is an empirical question -- help with the innovation component.

MR. EISENHART: Ken, you said very forcefully that it is not about money. I am wondering, if it is not about money, what is it? what does it take to motivate some researchers to participate in this process, especially when you said it is very labor intensive.

It is, in some ways, thankless with regard to a university promotion and tenure kinds of awards. So, what is it that we could build in that would make this activity more motivating?

MR. DODGE: Based on my own experience, the most valuable, positive, rewarding aspects of reviewing have been what I have learned in my interactions with my other reviewers.

That is why, for me, having even only telephone call panels or electronic panels would be much less exciting to me. I am serious when I say I learned an awful lot listening to Fletcher talk about learning disabilities or Bob Siegler talk about cognitive development or Benett Burton talk about infancy.

I learned an awful lot. That is enough for me to want to go again. So, I would look at who is on the panel and whether the panel is constructed in a way that will afford some of that.

MR. REDISH: I think that is another -- I certainly agree with what was just said, 100 percent. There is another factor and that is, researchers do research because they want to influence the community, they want to help people understand things.

When you are on a panel, you have an opportunity to exert some influence in the choice of where your field is going.

I think many of us take that very seriously. That is part of what we want to do in our research, and the panel is just one more step in working with the community and trying to help it move in directions you think are appropriate.

MS. LAGEMANN: Any last comments or questions? Thank you.

[Applause.]