MR. BRECKLER: I am Steve Breckler. I am the program director for the social psychology division at NSF.

To situate that, most of the psychology programs that deal with human aspects of psychology are in the Social, Behavioral and Economic Sciences Directorate at NSF, and that is where I sit.

NSF Is divided into these three directorates, and Barry represents the education and resources directorate, which is like a whole different land.

I have been at NSF for seven-and-a-half years as a program director. That is actually a very important thing to understand about NSF.

Before I came to NSF, I was, like most of you, a professor at a university doing research. I had been a member of the previous panel on the investigator perspective of peer review, because that is all I knew about peer review.

I showed up at NSF as what we call a rotating program director. About half of us are rotating program directors.

I was thrown into this mess of managing money and managing reviewers and managing panels and all this kind of stuff, and I had to either swim or sink.

What I am going to talk about is what I have learned about the NSF culture of peer review, and my own interpretation of it, and why I think it works, and why it works in particular at NSF.

I am not representing all of NSF. What you will find is that NSF actually includes many cultures for peer review, and I think Barry will talk about a different sort of culture or sub-culture of peer review.

I am going to talk about peer review as the social and behavioral scientists approach it. I think it is probably fair to say, in the NSF context, that the biologists approach this the same way, because historically, the social and behavioral science groups will run with the biologists, in the history of NSF. So, the culture of peer review sort of remains.

At NSF, peer review actually refers to a collection of different approaches to review, and they all mesh in interesting and important ways. It is the whole fabric that emerges that makes the NSF peer review process work.

I don't think we would get any argument that the NSF peer review process works, and produces good outcomes in a subcontext of things, and it works efficiently and works well and it is considered, I think, in most circles, as a good model for peer review, in the context of basic science. That is what NSF is in the business of funding.

So, what are the elements of this fabric? NSF actually relies on three different players in the peer review process.

One is the panels. We have been talking a lot about panels, peer review panels, how they operate, how they work and the process and so on, and NSF surely relies a lot on panels.

NSF, more than NIH, relies also on what we sometimes call mail reviewers or outside reviewers, ad hoc reviewers. These are people who don't sit on the standing panels. We just go to them the way a journal editor would go to somebody and ask for a one shot, one time review of a proposal. Those people are ad hoc reviewers.

A third person in this process is the program officer, who is also a peer in most senses. That is why I said it was so important to understand who the program officers are at NSF.

They come out of the research missions and disciplines of the same people who are now submitting proposals and being evaluated. So, they also are peers.

The NSF process is set up, if there is a continuum in the philosophy of managing funding programs, NSF is sort of a hybrid between two extremes.

We heard about -- Ed did a good job of characterizing them. At one extreme, the NIH model, which is the strong separation of powers model, which is a good model for the NIH context, and a lot of people have commented on the value of that kind of approach.

At the other extreme is the powerful manager model, which doesn't rely really on peer review at all, but relies on somebody who controls the purse strings, makes the decisions and that is it.

NSF is somewhere in the middle. We don't have a separation of power. We have program officers, and they are powerful program officers because they simultaneously control the purse strings and run the review process.

What is surprising about that whole mechanism is that, although you would think, in principle, that that kind of peer review system is open to abuse, in fact, it is rarely abused because it is a very open process, and it is one that is scrutinized a lot by other people. So, it tends not to be abused.

The philosophy at NSF is to solicit peer review from panels and from ad hoc reviewers, so that they form advice to a program director.

Ultimately, it is the program director that has to make a decision and make recommendations. So, it adds that kind of third perspective to the peer review.

What is important to understand is that the panels and the ad hoc reviewers are used for very different purposes, and both of them, together, complement one another in really important ways.

This, I think, does pervade a lot of NSF. We are torn between the tension of having to get enough people looking at a proposal that we can get good advice, and we know where a proposal should land, and getting deep expertise looking at those proposals.

This is a struggle, because if you have a reasonably sized panel, of less than 15 people, more like eight or nine people, you can't possibly get the deep expertise that you need to cover every single proposal.

If you only to ad hoc, outside, one time reviewers, you lose the consensus and you lose the judgement that you would get from a panel kind of situation, but you can use both simultaneously, very effectively.

So, what we tend to do in the social and behavioral panels is that we use panels, or standing panels -- which is people who stay on a panel, usually, for a period of three years or so, and they come back again.

Everybody on the panel sees all the proposals that are there in that particular funding cycle, and they are there not necessarily to provide the deep expertise judgement. They are there to be the judges in what effectively is a competition.

It is a horse race. What we need to know as program officers, in managing that kind of process, is the relative judging. Which of these two outstanding proposals is the one we should fund, if we are down to our last dollar? Which one?

That doesn't come down to the intellectual merit or the scientific merit of what is in the proposal. It is a relative judgement. Maybe we should go with this one because it is a junior investigator. Maybe we should go with that one because it is from somebody who is in a geographically underserved part of the country. Those kinds of judgements become relevant.

You can't get that kind of judgement unless you have people who have seen the whole array of proposals in front of you. That is the beauty, that is the power of a panel, and that is why we tend to use panels at NSF.

We rely on the outside or ad hoc reviewers -- as somebody said, I can't remember now who said it -- to give us the specific expertise in a particular problem area that we don't have represented on the panel.

If we need the Asian ethnographer to comment on that particular aspect of ethnography, then we will go to that person or people, where they need substantive expertise.

They mesh together. They give you different kinds of information, and both are really important in the NSF scheme of things.

What also makes the process work at NSF -- I have heard people comment on this a little bit -- is that NSF, as a rule, does not produce formal ratings or rankings for priority scores, the way you would at NIH. You won't get those.

You can ask yourself, why is that? Wouldn't that help you make funding decisions? Ed Hackett commented on one of the problems with that, and that is that the metric isn't fine enough to use that the way you want to use it, beyond integers of one, two, three, four and five. They really don't help you make the fine grained logistics decisions that you need to make.

The other is that that takes away the ability of the third peer, the program officer, to make judgements and decisions.

Ultimately, at NSF, what it comes down to is investment in human judgement, not formulaic judgement, not heuristic judgement, but human judgement. Some person has to sit there and look at the results of evaluation and make some kind of decision.

One of the reasons I think it works so well at NSF is that there is a person who is personally responsible for all of this.

I am a social psychologist. So, I can dump all kinds of social psychology on top of all this stuff, but one of the things that social psychologists will tell you is that, one of the single best ways to motivate a person to act responsibly and to take care and pay attention is to make that person identifiable and individually responsible for what they are doing.

It is a basic principle. That is the way the NSF peer review system works. A program officer is personally responsible and liable for the judgements that they make.

So, there are signed documents in which you are saying, I have looked at these reviewers and I have selected reviewers in a fair way and so on.

Every three or four years -- somebody mentioned this before -- we have what we call a committee of visitors that comes to NSF.

We open the books and the committee will look at the books, meaning, they look at all the proposals we have received, the ones we have funded, the ones we didn't fund. They look at all the reviewers that we solicited, they look at the membership of all the panels. They can read the reviews that the panel members wrote, that the outside reviewers wrote.

They can look at which reviewers were asked to review but refused to comment, and they look at how we cared about conflicts of interest and so on. All the books are open.

This is what keeps the system honest, and also what makes program officers personally accountable for the results of what they do. It is all these things together that makes the NSF peer review process work. It is sort of complicated.

The other thing about the NSF landscape -- and it may be peculiar to NSF -- is that this isn't the way we have to do it at NSF. There is ultimate flexibility.

The only rule that is carved in stone at NSF about peer review is that every proposal, with a couple of exceptions, every proposal has to be seen and written about by three different people in three written reviews on every proposal. They can modify the review process, depending on what is important to them.

With all of that said, and with the greatness of the results that the NSF peer review system produces, there are challenges and there are problems with it.

I will mention them briefly because I know Barry has thought a lot more about some of these things. One of the problems is getting reviewers.

Any agency that is facing the prospects of constructing or relying on peer review knows that it is difficult to get people who are going to dedicate themselves to do peer review.

The consensus, I think, around NSF is that it is getting increasingly difficult. We are not sure exactly why that is, but it is harder and harder to get people to do reviews, and spend the time to do it.

The other problem -- and it is one that we wrestle with a lot -- is that the review process is inherently conservative, we think, meaning that, if you are too innovative and too creative and off the wall, most review processes are going to ding you for that, and it is hard to get funding.

It is one of the reasons that the NSF uses the personal program manager approach, because you can override that conservative review process.

The program managers are capable of saying, the panel didn't like it because it was too creative, it was too innovative, it was too different, it was too funky. The program officers have the capability of saying, that is the one that I want to fund, the one that they didn't like. That is critical to the peer review process at NSF.

The other thing I will say in closing is, NSF is, I think, at the leading edge of using technology to manage the peer review process. It has a fast lane system, gets all kinds of rave reviews.

What I would like to point out, there is one facet of it -- there is also some psychology to this -- that makes it especially potentially powerful in managing the peer review process.

It has to do with this idea of, how do you get peer reviewers to focus on the dimensions of judgement that you want them to focus on.

People like Hal Arkus(?) suggest that you disagregate all of this and get people to focus on just very narrow chunks of the proposal. Technology is actually very useful for doing that. NSF does this in a very modest way, but you could see how it could be expanded.

At NSF, we want our reviewers to comment on two separate things. We want our reviewers to comment on the intellectual merit of a proposal, and we want our reviewers to talk about the broader impacts of a proposal.

The way we try to get them to do that now, in the fast lane system, when you submit a review, there are separate boxes where you can put text.

There is a lot of resistance to that. People don't want to do that, but you use one box and then the other box will say, see the other box.

In principle, what the technology helps you to do is channel people's reviews into the boxes that you want to channel them to.

That is actually turning out to work. It makes it easier for us to pick up on reviewers' comments separately. So, I will stop there.