MR. STANFIELD: I am Brent Stanfield, the deputy director of the Center for Scientific Review at NIH. Jack, you are more than welcome to come to any of our meetings, as long as you are a reviewer.

MR. FLETCHER: I can't actually, because I am on council.

MR. STANFIELD: You are right, you may not come in. A lot has been said about the NIH review today. My role here is just to -- I know a lot of you know a lot about NIH review. Others of you may not be so familiar. So, I am just going to give a quick overview of how peer review is conducted at NIH.

The Center for Scientific Review is the focal point of peer review at NIH. NIH consists of 24 separate funding institutes and centers, that are focused on either particular diseases or body parts or something, such as the National Cancer Institute, National Eye Institute, National Heart, Lung and Blood Institute, et cetera. CSR is a service center that provides the peer review function for all of these.

We receive all the applications. We are the federal receipt point for public health service applications. We do referrals to institutes and to the initial review groups and study sections.

What I mean by that is, applications come in, and we decide if this is going to be of interest to Heart, Lung and Blood, or if it is going to be of interest to the Cancer Institute or the Mental Health Institute.

We also decide where, if it is going to be reviewed within CSR, where it is going to be reviewed.

We also convene the review of most of these research and research training applications for scientific merit.

This is a summary of the NIH process. The school or medical school or research center submits an application on behalf of investigators through NIH.

It comes to us. We do the assignments, assign it to a study section and to an institute or center to consider for funding.

It goes to a study section. The study section evaluates it for scientific merit. Then it goes also to the institute to which we referred it.

The institute, the program staff, evaluates it for programmatic relevance. Then it goes under a second level of review at the National Advisory Council boards.

I should say, and this has been discussed a little bit, the advisory council consists of not only scientists of extremely high stature, but also folks who are interested in the disease or the orientation or focus of the research in the scientific area of the institute.

So, this is where, at NIH, we get a lot of advice in terms of priority, in terms of suggesting that, although an application may have been scored a certain way, it is of such great relevance to the mission of the institute that it ought to be considered. That comes in at this level, not at the evaluation of scientific merit at NIH.

The advisory council, then, makes recommendations to the institute director, who finally allocates funds. It actually goes back to the research school.

As I said, we get all the applications. There are now over 52,000 of these that come in every year. About 20 to 30 percent are, depending on the IC you are talking about, end up being funded.

We go through three review cycles every year. So, three times a year, we convene a huge number of study sections.

Now, if I could just take a moment for an aside, which was discussed before, why do peer reviewers do it if it is not the money.

I think I have a slightly different take on that. Two years ago, in September, one of our review cycles -- our review meetings are all held in October. So, in October, we put together about 250 meetings where people come to Washington, D.C. and review.

Two years ago, September 11 happened, and this was just a few weeks before all our meetings were going to happen.

It turns out that all of our meetings happened, and there were only a few people who didn't fly to D.C. to take part in the study sections.

When I visited study sections and talked to people, many of them would say that this was the only travel that they were allowing themselves to do, during that period of time, because they were so concerned about travel, with the recent terrorism.

For the most part, people said, they felt this was that important. I believe that the reason people do this is that they really believe in the peer review system, and they really believe in the importance of what they do in study sections.

I think that is what brought all those people, a month after September 11, to the Washington area, in spite of having to negotiate with their spouse and sometimes even with their children.

So, now, applications, like I said, are referred to peer review groups on specific scientific guidelines that we have for each of our review groups.

I guess I would say that so far we do have approximately 165 standing review panels. We also have guidelines to put applications through institutes, based on the mission of the institutes, and also on specific programmatic mandates.

Now I will talk a little bit more specifically about peer review and CSR review, which is sort of a model generally.

The CSR study sections are convened by an SRA, a scientific review administrator. They used to be called executive secretary.

They are the designated federal official, the DFO. Similar to other agencies, we are required by law to have a DFO run the meetings.

The SRAs perform administrative and technical review of the applications to ensure the completeness and accuracy.

This is really very helpful to applicants, because sometimes the SRAs are able to identify -- I mean not only not everyone sits up nights reading the instructions to the 3IA form. So, we are able to help applicants who have not quite gotten everything together when they submit their applications.

The DSA also selects the reviews, manages the study section meeting itself, and prepares a summary statement.

In addition, the SRAs attend the national advisory councils to provide any information they require about the peer review that occurred.

This is a study section meeting. Actually, this is a little bit old now. Almost all the folks would have a lap top in front of them if this was recent.

Each study section has 12 to 24, sometimes more, members. We convene face to face meetings, which we consider to be of considerable advantage over some other ways of doing it. As many as 60 to 100, sometimes more, applications are reviewed in each study section.

So, who does what in a study section? I told you a little bit about what the SRA is there for. The DFO is there to make sure things are done by the rules.

The chair convenes and runs the meeting. It is a collaboration with the SRA. The chair, however, is the person who cuts off discussion if it is really not going to get them anywhere, is the person who moves things along, helps summarize things, helps focus the discussion.

The reviewers write critiques before they come to the study section. They inform the review group; that is to say, they give them their point of view. They discuss the applications and then they score the applications.

Program administrators, as I alluded to before -- there is a separation at NIH between review and program -- program administrators do attend the study section meetings, but they are wall paper. They sit there and they listen. They are there to gather information, not to participate in the review.

We have five review criteria that we use to review grants. There is no separate score given to each of these criteria. How you come up with a score is how, in your own mind, these are related together.

It is not really -- Harold Varmus came up with these five. He came up with innovation, because there was a lot of concern because there were some areas of science where stuff needed to be done and by its very nature it is not innovative.

So, the thought is that you don't have to score high on any of these criteria, it the appropriateness of the weight of the criteria for the application in hand.

These are the actions that study sections can take. They can score an application. They can unscore. What we do now is, we try to, before the meeting or at the very first part of the meeting, identify those applications that are basically in the bottom half in terms of quality. So, then only the ones in the top half are discussed, and that allows them to focus their discussion on those applications that are most likely to get funded.

So, they can unscore, they can score. They can refer, in case there is some missing information that needs to be discovered, or there is a conflict that was undiscovered until the meeting happened, or they can not recommend it for approval or consideration.

This is usually used if there is some apparently egregious human subject or animal concern. It is just put in a category where it really shouldn't be considered for funding at all.

Our scoring system ranges from one to five, one is the best, five is the worst. However, since we unscore about 50 percent of the applications, three is the score that is given to those that are discussed, that is the lowest score.

The final priority score is the mean of the individual scores, and all the reviewers around the table score, multiply by 100. So, what you will get as a result is a score between 100 and 500.

Then the applications receive a percentile. This is a priority score that has been normalized based on scores from this meeting plus the prior two meetings of that committee. So, it is percentile score, the history, the recent history, if you will, of that study section.

Applications, like I said, are unscored. This is when the whole committee agrees that the application falls in the lower half. There is no priority scores assigned and the application is not discussed at the meeting.

This I just sort of summarized the procedures that actually go on at a meeting. The chair will announce the applications. Conflicts will leave the room. The chair calls for the initial level of enthusiasm by three, four or five reviewers who have actually read, or learned something about the application.

The first reviewer will present an overview and cite the strengths and weaknesses, using the five criteria I showed earlier.

Other reviewers or the readers, can concur or present other views. We try to encourage all these folks, to read their reviews, and that sometimes happens. We encourage the other reviewers not to repeat what has already been said, but to bring up new things that they feel have not already been addressed at the meeting.

Then the chair brings the discussion to a close. The chair should summarize the discussion. Then there is a reassessment of scores, and often you will find that people have been persuaded and their scores come to more of a consensus, and then all the reviewers vote their scores.

I wanted to mention briefly just a few things we have done recently. We now have all of our applications for a study section put on a CD and sent out to the reviewers.

We used to give them what we called books, which are huge boxes of applications in their offices a while. Then they mailed it back to their hotel and never looked at it.

Now we have a CD they can look at. The CD is actually very nice because it has -- it is searchable. So, if you are interested in basal ganglia, you can go in and look and see if there is an application on your CD that has basal ganglia in it.

Reviewers love this. So, all they have to do is just put it in their pocket, come to a meeting. They can actually look at it on the airplane. They can look at the CD and go through applications they weren't assigned but that are going to be discussed at the study section.

We also, all of our study sections use what we call IAPR, innovative assisted peer review. So, a week or so before the study section meeting, we post your critiques of the applications on a secured internet site.

Then, several days before the meeting, the curtain is lifted, so that all the reviewers can look at all the other comments.

So, the chair of the SRA can go in and say, okay, these 20 applications, it looks like there is pretty good agreement about these, and we don't even discuss these applications. However, there are these five applications that there is clearly some disagreement. We are going to have to have allotted time to discuss those.

It really really enhances the peer review process, we have found and, again, the reviewers seem to love it.

Then, of course, after the meeting, it is the responsibility of the SRAs to prepare a summary statement which is sent to the PI and to the institute for the funding decision.

It contains an overall resume that the SRA writes, a summary of review discussions. Again, this is not done for unscored applications. Also, we essentially have unedited critiques from the reviewers.

It also shows the priority score, percentile ranking, any budget recommendations or administrative notes that were brought up by the review.

I got a different list than Barry did. I was asked to discuss briefly some of the advantages and the disadvantages of the NIH system.

The next slide, I just randomly picked out advantages. Separation of review from program management. So, if you are approaching NIH for a grant, you can talk to your program manager, who can turn into your advocate and can give you very frank advice on what your next step should be.

If the grant wasn't funded, they can give you advice on whether you should come in or not, or whether you should go back to the drawing board. That is totally separate from the federal employee notes who is running the review. We think it works very well for NIH.

Now, we have, again, face to face review, which facilitates full discussion. Face to face review also informs the program management, because the program officers, like I said, attend the meeting, listen to the meeting.

The way things are said, you can hear them and get more information than what is actually written down in summary statements, and that program officer can talk to the PI.

Another advantage is that the percentile balances the study section score information. So, it makes more of a level field.

Now, in the next slide, I think this has been said earlier. Decisions by committees can naturally be conservative. That can be a disadvantage, if you want to seek out really innovative applications. It is probably not going to be done terribly well at the study section level.

Convening face to face meetings presents logistical hurdles. It would be a lot easier to do everything by mail. It is very difficult to get very busy scientists to come to Washington three times a year for four years.

Face to face review advantages rhetorical skills. You see this at study section all the time. There are some people who just love to argue and are very good at it. There are other people who are just as bright and can write very good critiques but don't have the facility for rhetoric that their colleagues might. I think this is something that you would have to consider, when you consider peer review.

Percentiling, although it normalizes, it assumes that the quality of science among study sections is similar. So, it really depends how you parse up the science.

If you have your phrenology study section, well, when the scores are percentiled, there are going to be some really good percentiles on the applications in that phrenology study section. So, it really depends on how you parse the science into the various study sections. So, that is my last slide.