MR. FLETCHER: Before we begin the discussion, I am just going to try to tell you some of the things that I learned that are really important.

The first thing that I learned is that there actually are models other than peer review to invest in research.

I heard about four different models. One of them is called ear marking which Congress does, another is the strong manager model, which we heard of from the Office of Naval Research.

The third, which didn't get much discussion, was formula funding, the idea is that there is so little money available, we may just want to divide it up between the different universities, and then the fourth model is peer review.

I am actually curious as to whether there are any models that weren't discussed. If anybody knows one, I hope they will chime in.

I found really helpful the discussion by Dr. Hackett about competing values. What it makes me really aware of is that peer review exists in a system of balances, where people are trying to balance issues around the allocation of resources according to multiple criteria, competing values like openness versus secrecy, effectiveness versus efficiency, sensitivity versus selection, responsive versus inertial, meritocratic versus fair, and reliable versus balance.

I heard those themes sort of repeated throughout our discussion in the different presentations. I heard it from the agencies.

For example, I felt like we heard that the different agencies had different ways of trying to balance these six sorts of continuums, and different ways of thinking about how to do that.

What I also heard, Dr. Hackett also talked about the importance of pluralist possibilities, and we heard different ways that the agencies have of addressing this idea of pluralism.

How do you balance, for example, decisions and information that comes from the peer review process itself, with other issues, like the institute's priorities and the ways the institute, for example, brings up the funding for grants that might not have the highest rating, but are really relevant to the institute's priorities, and sort of how that gets done varies for different areas of educational research.

We had a nice discussion, I thought, from our panel of investigators -- deans and people like that -- on characteristics of reviewers and the purposes of reviews, the idea that reviewers have a blend of substantive expertise, methodological expertise, some idea of what is important because of policy and practice.

There was a lot of discussion about how much of that should be inside the reviewer or distributed on the panel itself.

I thought the discussion of purposes was really helpful and echoed things that we heard throughout the discussion, the idea that the purposes are to improve and maintain the quality of research, to envision, the shape and the feel, to provide feedback, to reach out and integrate disciplinary dollars -- the point that Ken Dodge made, that I thought was really valuable -- as a product of peer review, and that has been, for example, very much my experience in peer review.

I have been on a lot of interdisciplinary committees, and what really happens is that you have to develop a sense of trust and respect for people who have very different perspectives, for what is going to happen where what is going to happen is nothing gets funded, nothing gets out of the study group.

Today, we heard really quite a bit about some of the logistical difficulties that influence peer review, the impact of changes in institutes' priorities. We had this whole issue of reliability. So, it smacked us in the face.

I think the take home message was really the importance of well organized peer review, the importance of being able to actually run peer review without being bogged down by legislative priorities and knowing when your money is going to come, those sorts of logistical issues, as well as the importance of professional development and training for reviewers, which I think, from our agency presentations, we really did not hear a great deal about.

Terry Levitin, for example, showed us the materials that she has developed for training. That is one institute at NIH, and we heard very little about that from the other speakers.

The one thing that was agreed consistently from all the discussions is data. There is really very little data, very little research.

A lot of people are making assumptions about what makes for effective peer review or effective funding decisions, but it really seemed like the process itself is hardly ever set, that there is relatively little empirical data or research on characteristics of peer review in any of the agencies, for example.

So, those are some of the things that I sort of thought about and heard sort of echoed. Of course, I would be interested in other comments and questions.

MS. FALKENBERG: I had one other comment that I would like to add to that. I thought that the responses, when there were questions about accommodating diversity, were not quite to what I wished they would be, particularly in light of the fact that a majority of the NSF funded programs for education really are targeted for students in school districts that are primarily minority, or disadvantaged in other ways, just being in a rural district that has very little resources.

I know myself, when I have sat on panels, the composition of the panels, the composition of the people in the room at the plenary session ahead of time, did not particularly reflect that population.

I do think that, when I heard people say, in my opinion, they didn't acknowledge the fact that I think there is a real need for improvement in that area.

MR. HENLEY: We didn't hear anyone suggest a different process than the peer review process. So, the real question, it seems to me, is how to strengthen the peer review process and make it more reliable.

MR. FLODEN: I got a lot of the same things out of it that you did. One of the things that really struck me in the conversation today was in the competing models, there is a disagreement about how much people feel that the process is one that ought to be educating the proposers, what we talked about yesterday, providing tutorials, and how much effort and expense should be put into that, ranging from the goal is just to pick the best proposal, to what is really important is to provide detailed feedback to people so that they can do better work.

That seems an important choice point in constructing that peer review system, and you can really invest a lot of resources into giving people more time, to provide more extensive reviews, if that is the purpose.

MS. LAGEMANN: It strikes me that one issue that bubbles up periodically is the whole question of how do we involve the public and how do we involve practitioners.

The drift seems to be toward involving them in setting priorities, but I think it is something worth thinking about, how we do it.

MS. SCHNEIDER: I think the other is with scheduling and systematic grant competitions and funding to be able to put peer review panels together.

MS. LEVINE: I am Felice Levine and I am currently executive director at the American Educational Research Association.

I want to also mention that I was a program officer at the National Science Foundation for 12 years. I was one of the program officers much more heavily involved in developing expertise in peer review, in addition to running a program that I ran, which was an interdisciplinary program.

One of my experiences with that hat, because I think it is germane to today's discussion, is that I was asked in the mid-1980s, by the Director of NSF, to work with the Office of Juvenile Justice and Delinquency Prevention, a more mission oriented agency, in the development of their procedures, and to think through how to develop a culture of science within the context of a mission agency that also has been quite politicized.

In that context, we worked with both the NIH model and the NSF model, in trying to examine what would be the framework, and the nature and the strengths of the various strategies.

While, from a formal organizational point of view, we have heard a lot of the different structural aspects, and the differences between how each of these science agencies unfold, in many respects, they are quite compatible and really complementary, and they are variations around a theme.

One of the themes, in the context of a smaller agency, like the National Science Foundation, which perhaps fits better to the OERI or the IES model, is that it is a smaller staff with potentially more flexibility in developing a strategy for the funding of science.

I want to say that one of the things that represents a really terrific syntheses is this current situation of out-sourcing the peer review process.

I think that really needs a tremendous amount of thinking, about whether that is the best or the most appropriate way for Dr. Whitehurst, at this point, to be thinking about it.

I appreciate the way he presented it. He presented it in the way of an integration of a process that would have a fire wall, which is sort of a variation of the NIH model.

In this larger context of creating a culture of science within OERI and IES, we talked about the importance of the stability of the process, we talked about the possibility of standing panels, and what standing panels can really do in the context of especially interdisciplinary problems.

It is not only in the funding of research. One of the real strengths of the possibility of re-thinking how to do this is that you are not only making decisions about a piece of research.

You are also making macro level decisions about developing a philosophy of science that guides programs at institutions about how to operate. So, you practice the notion of the separation of the out-sourcing in this context.

In the context of the NIH, while they create a fire wall and a separation, it is not as distinct as it may appear.

The program officers are there. They get the benefit of the discussion. They are sometimes asked for advisory opinions and give guidance.

The RTG leaders are also scientists in these areas who can also step back and talk to program officers after a review is complete.

So, there is an interaction between a micro level, if you want to put it, the teaching of the job, the reviewing of the individual proposal, and a more macro level kind of guidance that permits better decision making and also permits better plans.

So, I have, within this list of items, in the moving target of what IES can potentially become, I think a real challenge for it in the development as a fine institute within a large agency is how to really make that translation, and whether this particular component, which is the most recent innovation, one might say, is really, in the long haul, going to work.

I think one of its most strategically problematic implications is the way to attract and retain quality scientists on staff, not just because you are not making those decisions but because you are not part of the review process, where you can learn and plan and operate more effectively.

MR. FLETCHER: I think that is very helpful. In fairness to Dr. Whitehurst, though, I think he would say pretty quickly that one reason that he has outsourced is because he doesn't have a board yet, and he doesn't have the functional staff to actually conduct reviews of the sort that he wants to conduct.

He described this, I think, pretty clearly as an interim arrangement, pending the appointment of his board. Ultimately, it is going to be up to the board as to what sort of review is done at IES.

MS. LEVINE: In giving that illustration, it was, in a way, an illustration. The larger question is how one works within the contours of an IES or an OERI to develop the kind of stable culture of science that is much more characteristic of NSF or NIH.

MR. FLETCHER: A really provocative question -- at least I think it is provocative -- is, given that the Department of Education does a lot of peer review -- they do peer review at IES, which has some influence on the department now, and they have a big review process at OSEP, for example, for the resource practice division -- they are also doing peer review for the elementary and secondary education acts, for Title I applications, for Me First.

Peer review is becoming increasingly pervasive, and you wonder why they don't develop a separate peer review section and run all of these competitions out of some sort of central organization, you know, CSR in the Department of Education.

I mean, I am not raising that. I am just sort of raising that as a question.

MS. LEVINE: I think one of the most useful roles for this panel is in giving guidance on a range of those issues.

Also, in so doing, I think one of the concerns that was expressed by a number of agencies was about the absence of the motivated review.

For me as a program officer, I had people serve on panels for three years on rotations, and they read 60 proposals, read reviews in advance, submitted reviews in advance, so that we could discuss and deliberate.

One of my concerns about your data presentation is that, in part -- and it is not just the wise program officer, but it is also that wise group, struggling, struggling with the issues and the ideas, and then reaching not a false consensus, but reaching a scientific judgement in that middle zone, that is so tough.

How do you develop a willingness on the part of the community to see themselves as part of the process. At NIH and NSF, the resident scientists, whether they are visiting scholars or the career scientist, feel they are a part of the research community, but they recognize their role and hat.

The extramural community also feels responsible, not just because they got the grant, but because they want to make that engine go. I think the mission agencies face a challenge in that.

There was one report in the late 1970s by the National Academy of Science that gave, I thought, very good guidance for the National Institute of Justice, that really struggled similarly with these issues.

The National Institute of Justice really developed, in the context of the Justice Department. There are still struggles over issues of crime and violence, and policing, and the same kinds of translational controversies that education faces, because it is a highly contentious area.

I think they really built a fine peer review process over the years, although I will say, recently, there are similar concerns about disruptions in the articulation of programs, outsourcing and other kinds of things that have been sort of moved into the Department of Education. I think some guidance in that generic area would be extremely valuable.

For education research generically, and as it can unfold and hopefully grow in the Department of Education.

MS. KLINE: Hi, I am Sue Kline from the Department of Education speaking as a private individual. I would like to commend Felice's comments. I agree with them 100 percent.

I would also like to point out that in our new age, now, I think, of technology, we have lots of possibilities of doing much better as far as all kinds of review activities.

We do need the conceptual and agency leadership in the various areas, but also with technology, I think we have a lot to learn from the NSF fast track and how they are managing the submission and review of proposals.

I think we got a good presentation from NIDA on how technology can be helpful in the training. I think we have some preliminary research that shows that, for research activities, we have good combinations of technology and reviewers doing this, distance-wise, but perhaps in many cases it is still important to have some of the interpersonal interaction and the long-term ability to work with a group and learn and trust them and act with them, related to standing panels of various sorts, with some of the good variations that we heard from, from the other review processes and the other agencies. I think hopefully your recommendations will be well attended to.

MR. FLETCHER: NIH actually is close to adopting electronic submission and electronic review, I think within a year or so.

Also, I believe IES, the group they outsourced to, arranged for electronic submission of grants. I think that is coming. I look forward to it.

MR. LEVINE: One other point that I thought Steve make yesterday, and that is in answer to a question that Lisa had raised, and that is about how do you know your processes are working.

He made reference in his presentation to the routinized process within NSF of a committee of visitors that intensively scrutinizes the peer review process.

Now, we all heard of committees of visitors, when they come into academic departments, they sort of say nice things and they say also some very helpful things.

Let me tell you, as someone who went through it three times, this is an extraordinarily thorough process. There is a series of procedures. It looks at issues of the quality of review. They are literally looking at the reviews, the proposals.

They will interrogate the program officer with a series of questions in terms of how you have gone about your decision making, and they also look at demographic issues in terms of the patterns of funding, the nature of funding, potential biases and methodology.

So, I think one of the important issues is not necessarily to have that process in place, but what kind of internal review process can give you periodic and routine and informative information with respect to the efficacy of your peer review process.

MR. FLETCHER: I wonder, the response that some would make to the idea of a committee of visitors is that it is just more pigs at the same trough.

In essence, you know, anybody who evaluates peer review is invested in the system, that we are all just trying to support a system from which we all benefit in different sorts of ways.

How do you get beyond those sorts of criticisms? The people who serve on the committee of visitors are our peers; right?

MS. LEVINE: It is a peer review panel. It is something like the difference between a trial court and appellate court, if you will.

There is something about -- I mean, with all of this discussion about the divergence in norms across and between fields of science, I think Robert Merton very profoundly talked about some fundamental elements of science that really guide all fields in a really very rich and dispositive way.

When you are in a particular role as a scientist, whether it is as a program director or reviewer, one of the tenets of the independence of how you are performing your role -- and let me say that it is really an oversight process that is extraordinarily expensive and intensive, and in no sense, do we all bring some of our norms to the table? Well, yes.

I think the creation of those oversight review panels, not by the program director, but by the assistant director of the directorate, and that scientist -- whether it is a Judith Ramalie(?) or a Normal Bradberg -- needs to know they are getting good guidance on the functioning and effectiveness of their operation and that they can both be accountable to the directors of their institute and also to the Congress, with respect to the efficacy of what they are doing.

So, I think those norms are really not to be denied in the context of the National Academy of Science, but those norms are really to be understood as terribly well internalized.

In a way, the question is how to translate that in a stable and routine way within a mission agency, mission agencies that have tended to be more labile on these issues.

MR. HACKETT: In that context, I wait to raise again a couple of ideas that I began with yesterday morning.

Two that the committee might like to consider are, first, the idea that peer review should be thought of as a boundary process.

Don't think of it as IES' peer review process, but I would think about a peer review process for educational research that is going to sit across the boundary of federal agencies and the academic research community. You want to have shared ownership of this.

The second idea I would bring up again is the legitimacy, that the procedures that you institute have to be recognized as legitimate by all the parties that participate in the review process.

They have to work for a Russ Whitehurst, and they have to work for that researcher out at the state university. They have to be brought in equally.

When you think about it that way, then there might be a role for the societies to play, perhaps catalyzed by this committee.

So, societies that intersect with the interests of educational research could be in cooperation with the funding agencies in the federal government that support educational research -- that is agencies plural -- so it doesn't become a Department of Education problem, but education at NSF and NIH, where it intersects with the societies. You raised this as an issue.

This way you can develop a reviewer culture, and this then reaches into the reward structures of departments, the training of graduate students, what deans encourage their chairs and encourage their faculties to do, to do responsible reviews and participate.

It should be treated as a sort of wholesale climate and community, not a narrow fix by IES. I would encourage you to do that.

MS. SCHNEIDER: Back to the point that Margaret made, which is sort of the negative to positive, how will we know when it is working well.

I think Felice said, well, this is one way to get some indication about how it is working, the visiting community, but I think that is a real question for us as a committee. How will we know when it is working well.

MR. FLETCHER: I am not going to let people leave without reminding our group what we learned in the last presentation, which is that there are serious questions about the reliability of peer review for certain judgements.

I personally found the data Dom presented fairly devastating, although I believe it, based on several different experiences.

I mean, one tale that I will tell on myself is that one of the things that I have done in the past, when I had trouble with a grant that I couldn't get through a particular study section is, I basically just changed the PI, rewrite the abstract, and submit it to a different agency.

On more than one occasion, it has been funded by doing that. I mean, the reliability issue is really quite serious.

MR. WISE: I agree that that was a fairly depressing presentation, until the last few slides, which did offer some hope that, with appropriate attention to training, things could be turned around.

MR. FLETCHER: I think Brent is still here. What did you think about the training issue? Oh, he and Barry left.

MS. PETERSON: One thing I learned is that we are not as bad off as behavioral and social sciences.

MR. FLETCHER: One of the things we learned pretty quickly when we did the science and education report is that we are really not different on multiple dimensions. This is just another example of that.

MS. PETERSON: I think it would be great for your committee to say that.

MR. FLETCHER: I am sure we will say it again.

MS. BERTENTHAL: One of the questions, I thought, that has come up in some of the papers we have read, as well as in Dom's presentation, and that is, you might expect more reliance if we had more similarity across agencies.

So, the processes, let's say, at NIH are different than the processes at NSF. You might expect that you might have a grant that is suitable for funding at one, but maybe doesn't fit into the portfolio in the other.

There has been some expression in some of these papers about how similar peer review, or how similar for purposes of cross agencies it should be, versus keeping them different, so that different kinds of research could be funded.

MR. FLETCHER: Anybody else have comments? I think we are done. Thank you very much.