MS. LEVITIN: I think the only thing worse than being the last speaker on the first day is being the last one on the last day. If my presentation is choppy, I am going along deleting things that you have already heard.

So, let me talk a little bit, first, just to explain to you, most applications that come to NIH, as you may know, go to the Center for Scientific Review for review. That is where the largest number of study sections reside.

It is also the case that each NIH has its own internal review system. I am at the National Institute on Drug Abuse, NIDA, and we also have in house several review committees.

Each institute typically keeps its training grant applications and other kinds of mechanisms like its center grant applications, but it varies institute by institute.

What doesn't vary are the rules and regulations and procedures that are standard across NIH. So, really, what Brent said about selecting reviewers, about training reviewers, is as true for NIDA as it is for CSR. It is just that CSR is on a much larger scale, and because we have different kinds of mechanisms, like center reviews and training grant applications, we need to train people in addition to the more generic kinds of things that they need to know.

I want to talk a little bit about some of the squishier things that need to be taken into consideration. Brent, of course, presented, and I absolutely agree with, as I am sure you do, the importance of only selecting people who have the appropriate scientific expertise.

I mean, that is what we are about and that is certainly what we look for. We need to have people who are unbiased in their assessment of the quality of the science in front of them.

There are other qualities that we need to search for. We need to have people who will listen intelligently and respectfully to their colleagues, and who will be open intellectually to genres of research other than their own.

We need people of absolute integrity. We talked earlier about conflict of interest and confidentiality. You need to trust people's integrity to help you identify some of those kinds of things that aren't obvious in the written materials, and we obviously need to tell if someone is a co-investigator on an application. His or her name will be there.

What you don't know is whether or not someone is also someone's best friend or best enemy. So, we need people to report to us when they can't give a fair and unbiased review, because the integrity of the system really depends on the integrity of the individual reviewers that we have in.

We can have any number of checks and balances, and we do, but still, the bottom line is they have to be people of integrity.

They also have to be willing to adhere to the NIH policies and procedures. They may think it is psychometrically nuts to have a scale that goes from one to five, but we have a scale that goes from one to five and, if they don't want to play, we need to find that out.

Again, we need to have agreement that they will, indeed, adhere to the NIH policies and procedures and they have got to be willing to leave their egos at the door, which is not always easy.

The very people we want are people who are successful in their areas of science, committed to their areas of science, and sometimes that brings with it a sense that their areas of science are the best and only, and they may not work cooperatively.

So, we want people who neither dominate nor hide under the table when there are discussions, but are able to engage intellectually in a way that really will assess the quality of each and every application.

So, where do we find such people? A great scientist, as Brent said, is not necessarily a good reviewer, although I do think that a good reviewer is always a strong scientist.

It isn't easy and, again, to follow on Brent's comment, it is getting harder. It may be most difficult because of managed care to get clinician researchers, but boy, it is not easy to get other people from academia either, even though they are not in the position of losing money.

As Ken said earlier today, money is not the incentive to get them to come anyhow. So, it is difficult to find people who are generous of spirit but rigorous of mind.

That is what I try to train for. That is what I try to select for, because you don't need people who think the reward system is to criticize others in a harsh ad hominem or ad feminem way.

You don't need people who think that, by showing off to their colleagues, they really are gaining points, and there are, in fact, study sections -- none that Brent knows of and none that I know of -- where the reward system or the culture has been one of dumping on the applications as they come in.

People need to be reminded that applicants have taken very seriously the writing of an application. It can mean the difference between tenure or not, staying at a particular institution or not, and reviewers have to take every bit as seriously, no matter how lousy it is, every application needs to get its serious moment of total concentration and serious review by the people sitting around the table.

Well, how do we find such saints? How can we persuade them to come? It really is not easy. The training process really begins immediately, the first time you call someone, the first time you talk to someone at a professional meeting.

You are sizing up, as the scientific review administrator at NIH, whether or not that person is likely to be able to be somebody whom I have listed a moment ago.

So, the training goes on from that very first meeting. It is both formal and informal. We try to train for general principles and policies, because there is no way we can anticipate every contingency.

If people do understand a few fundamental rules, and they are paying attention to those, then they really will, in most instances, be able to know what is necessary to do.

Brent mentioned that sometimes there will be orientation sessions the night before. Sometimes there are orientation sessions at breakfast that morning, again, depending on how the SRA believes is best.

By the time the people come to the meeting, as I think you said or somebody else said, it is kind of too late. They have done their reviews at home.

So, a whole lot of training has to go into the front end, and not just sending materials, not just talking with people, but really being a good diagnostician, is what an SRA has to be at that point.

If you pick up -- I hate to use the term, but I am from California so I can -- bad vibes at any point along the process, from an initial meeting with someone to actually, perhaps, seeing drafts of their reviews, which I have asked for, and SRAs sometimes ask for -- it is not to tell them how to do the science, but to ensure that they are doing the reviews appropriately, if at any time it doesn't feel right, the SRA needs, or the NIH needs, to be able to say, this is not going to work. That is not easy either and, of course, mistakes are made.

I think on the whole, because the process is so rigorous from that first step, that first meeting, that generally reviewers do a good job.

Of course, there are also remedies when they don't. I think one of the things that hasn't been mentioned is that if the scientific review administrator is biased or inappropriate, has not met the standards, that application can be deferred or re-reviewed.

Again, the scientific review administrator would want to discuss that with his or her supervisor, but there are remedies, and they have been used when one recognizes that the review has not been fair.

What we do afterwards, for example, is we look at the scores to see, are there any outliers. If those folks have spoken up and said why their position is, that is fine.

What we don't want is people sitting in the corner, around the table -- you know what I mean -- sitting there and thinking, everyone else says this is great, but I Am going to give it the worst possible score because I don't like it, or vice versa.

We really do, again, depend on people's integrity to speak up, but also we check to see how they are actually scored.

So, there are these checks and balances. I brought along some training material that we actually use. We are going to zip through this really quickly.

This is a course that was just developed in our office, and the credit really goes to Doug Mace(?), who is my deputy, for putting this together after hours and hours and hours of discussion as to how best to train reviewers.

What this is, is the absolute minimum of what we believe reviewers need to know to do the job. In fact, they get tons of other materials, but this is what we want them to do before they start to do their reviews.

This is what we want people to look at. Now, we had a discussion as to whether we should tell reviewers, if you do a good job, you will be invited back.

I should tell you, that really this is not going to be seen as a positive kind of thing, but we put it in anyhow.

So, this is what we would like people to do before the meeting. I am not going to go through each one, because it is really self explanatory.

The course can't do everything, but it will give them, again, the floor, but by the time they get there, of course, we want them up to the ceiling. Again, before the meeting -- we can go through these faster.

Some of these things are obvious. Do you know what you have to tell people? You have to tell people we expect them to stay for the entire meeting. It is kind of simple, but if you forget to do that, you discover half of them have reservations to leave the morning of the first day and you have got trouble.

So, it varies from very basic good manners -- you don't leave a meeting until it is over -- to much more sophisticated and complex kinds of training that are needed.

We used to say open the box, now we say, put in the disk. Now, again, we talked about unique review criteria. That is because this is institute review. There are criteria for reviewing scholarships, there are criteria for fellowships.

If they are reviewing training grant applications, there are different criteria there, and we really need to make sure that he appropriate criteria are applied to the appropriate mechanisms.

Again, these are some suggestions. We have talked earlier about the tutorial role of reviewers. It used to be the case that summary statements would say things like, you should do this, you shouldn't do that. That was thought to be inappropriate. That was not the role of the study section. Again, echoing what Brent said, it is not the role of the study section to provide a tutorial.

Now, under the new thinking about what a summary statement is supposed to do, it reminds you that you really are not supposed to say, you should, to the applicant.

These are other recommendations. Here is a pop quiz. Again, one of the things on there, it said, decide on a preliminary score before you come into the meeting.

We debated about this. I should tell you about some of the things we debated about. When Meryl first asked me about participating in this, I thought she was going to change her mind.

I kept saying to her, well, it depends. So, every time she would ask a question I would say, well, it kind of depends on what you want, because of the trade offs.

If you ask reviewers, at home, to come up with a score, but they have -- particularly in the case of new reviewers -- never reviewed before, it is sometimes real hard for them to know what the anchor points are.

If they show up with a score that they then publicly announce, they may be locked into it prematurely, because they are not going to want to look dumb or incompetent or whatever, if they subsequently change their score, even though you say, it is perfectly okay to change your score.

So, one thing to think about is, in advance of the meeting, do you want people to score an application, do you want them to come with a range? Do you want them just to come with a thumbs up, thumbs down, because there are different consequences for each of those kinds of instructions.

So, our instructions to reviewers now are, come with a tentative score. Come with a possible score. Come thinking about a score. We don't want them to come and say, this is a score, I know it is true, this is the score I am going to argue for, and this is what I am wedded to and, no matter what, I will stick to this score, even if it turns out that, when I get to the meeting, I discovered that a two for me was a one for everybody else, or what I assess as a two, other people are assessing as a one.

So, again, Meryl, it is a trade off of what do you want and how are you going to get it. I must have said that to you how many times? She lost count. I am still here, but maybe not for long.

I don't know how much of this you want me to go through versus questions. I am really here at your disposal. So, do you want to run through this?

MS. BERTENTHAL: Could you mail it to us?

MS. LEVITIN: Yes, it is all there. In fact, I put only one condition. When I send things out to people, I expect feedback.

So, if you look through this, learn from it, please get back to me or to Bill, and let us know what suggestions or recommendations you would make to improve it.