MR. WISE: are there any other sort of key parts that you want to summarize now, or we can take questions.

MS. LEVITIN: Again, I think you know what the system is by now, certainly. This is again just going through. I think not.

Oh, just one thing. I just want to show you there are a lot of different ways of scoring. We have come up with a couple of ways to help people anchor, again, prior to the meeting.

So, in doing this, we are saying, hey, there isn't just one way to think about it. There isn't just one kind of scoring system to use. You might want to think about it this way, or you might want to think about it this way.

So, the idea is, there are different ways to think about scoring, but be flexible, particularly if you haven't done it before, because at the meetings you will be able to recalibrate.

MR. FLETCHER: I still don't understand the rationale behind not attempting to tutor applicants? I always thought that one purpose of peer review was to shape or bring along a field.

It seems to me, for example, that if you are reviewing a grant and someone uses a correlation coefficient, or they are doing some sort of search and they commit several faux pas in terms of identifying the population, you would tell them there are different ways -- you need to do this and this and this and this to identify the population of interest.

MS. LEVITIN: It is the difference between, you should have done that versus, one of the weaknesses in the application was here.

MR. FLETCHER: If you are going to say a weakness, I think it is incumbent on the reviewer to go another step forward, if it is the technical issues, to give the applicant advice about what to do about it.

MS. LEVITIN: I think that when Harold Varmus, the previous director of NIH, declared that summary statements should not be tutorials, he was doing that on the basis of a number of complaints that he heard from people in the field, where people were resentful that they were being told how to do their own research.

My own view is that, for every 10 people who resented being told how to do their own research, there were 10 people who appreciated the different suggestions of reviewers.

As I said a moment ago, if you want to be part of the NIH, you need to adhere to the NIH policies and procedures, which is what we have to ask our reviewers to do, which is not to provide tutorials, whether we think it makes sense to do so, or whether we think it is useful or not.

I mean, you are in a position, I think, to think through, to what extent do you want tutorial reviews, with the understanding that sometimes they are really off the wall.

MR. FLETCHER: At times, yes, but junior people apply for grants and, if you identify a weakness, I think it is incumbent on a reviewer to talk about how to remedy it, and not just say, this is wrong and this is wrong and this is wrong and this is wrong.

MR. STANFIELD: I think the other part of what Harold was concerned about is that sometimes the pink sheets would come out looking like, here are the five things you need to do to get funded. Then the application would come back in, having done those things, and it wouldn't get a fundable score. It was sending a bad message to the community. I think that was part of it.

MR. FLETCHER: There is a big difference between telling someone how to get funded and making irresponsible critiques of a grant.

What you open yourself up for is somebody just goes through and says, this is wrong, this is wrong, this is wrong and this is wrong, without demonstrating that they know what they are talking about.

MS. LEVITIN: Right, but there is a difference between saying, this measure is weak, this measure is weak and you ought to use this one instead, this measure is weak for the following reasons and you ought to use this one instead and, furthermore, if you do use this one, there is the likelihood that your score will be better.

We are talking about at what level of information to the applicant do we want to, under the NIH system, tell the reviewers they need to stop.

MR. CICHETTI: I think one of the basic issues here is not whether we are providing a tutorial or not, but a matter of, whatever we recommend as a reviewer, that we can document that with specific references.

Then the applicant has something to go by. He can go back or she can go back to the literature and read and learn.

Maybe the reviewer has cited the wrong references, but it is not just a matter of telling them what to do, no matter how polite they are. It is a matter of documenting everything that you do as a reviewer.

MS. LEVITIN: I don't think we really know, or at least I don't know if it has been studied in any rigorous way, the extent to which the prior more tutorial summary statements were more applicable to the field or subfields or individuals than not.

I wish we had studies of before and after. It is too late now, because we just have the after and anecdotes. We have a lot of anecdotes, but anecdotes do not research make.

MR. FLETCHER: Neither is lack of evidence the reason to think that the policies are to the benefit of the NIH process. Is this documented somewhere?

MS. LEVITIN: We have instructions to reviewers.

MR. FLETCHER: I mean, these were written in the instructions to reviewers.

MS. LEVITIN: The instructions to reviewers say, don't tell the applicant how to do his or her research.

MR. WISE: I suspect we will come back to this. Are there other or different questions that people want to ask now?

MS. SCHNEIDER: [Off microphone.] -- either because the reviewer was new or because the field is small, or maybe there aren't people who haven't properly demonstrated they are scientifically appropriate. I am just curious to know how you go about developing a peer pool.

MS. LEVITIN: It is very difficult. If there isn't a pool out there already, sometimes what you have to do is bring in more junior people than you would like.

Sometimes what you have to do is bring in the people who are not directly in the area, but are in a related area.

Sometimes what you can do is you think an application is a jigsaw puzzle, and you bring in four people instead of two, perhaps, to make sure that all the areas are covered.

It is extremely difficult to do that, and to ensure that you have gotten peer review. Also, the definition of a peer often gets stretched, and has to be stretched, to ensure that you have enough people to review.

By the way, having a pool available doesn't mean that you have a pool that will agree. That is the other thing.

MR. STANFIELD: For I guess over a year now NIH has been allowed to fund research involving human embryonic stem cells.

Right now, there are 45 PIs that are funded in that. So, it is very difficult, or at least it was a couple of rounds ago, to find someone on a review panel who had experience with cell lines and, in that sense, was right on to review.

It wasn't so difficult to find folks who knew a lot about stem cells, folks who knew the human literature such as it was, and were competent to perform the review. That is the sort of thing that you have to do when you are looking at emerging areas.

The other problem that happens, NIGMS has a mechanism they call a glue grant, which funds both a specific area of research throughout the country. So, basically, everyone in this area is in conflict. This is a real challenge.

MR. HACKETT: A quick comment about the training. We have been focusing a lot about training reviewers as a problem for the agencies.

I would suggest you think about peer review as a boundary process, since it costs government, and it is a responsibility to train reviewers, and it should be part of graduate education.

One thing I do as a professor, with NSF's permission, and permitted to do is for the faculty members recruited as reviewers to consult their students in the review. I had a student, I named him and said, I would like to have him look at this proposal with me.

I wouldn't think that it is a process that agencies own, that they have to provide as a service to their customers, but a boundary process, with the communities co-producing with the agencies. Then you can take some of the training process and shift it into education.

MS. LEVITIN: I think that is a very good way of looking at it. I think one of the real benefits of the web now, is that there are so many -- it used to be that staff were gate keepers of information and if you knew -- if you had the key to get in, you could find the information you needed.

That is not the case any more. All kinds of things are up on the web. So, training materials go up, recommendations on how to write applications go up.

I mean, there really is more than you can absorb. That is the down side. The good side is, you don't have to depend on a program officer or anybody else, really, to have access to that information. You have access to it directly yourself, including what Brent and I put on today, and everyone else.

MR. STANFIELD: I think that is an excellent idea. I think it could easily be rolled into having a couple of people at the university who serve as peer reviewers, be rolled into a course on reviewing who might be more interested in spending their resources on how to write a grant application.

MS. LEVITIN: We do that kind of thing. We hold mock reviews. We hold all kinds of training things at professional associations, and sometimes we are invited to do the same thing at universities, but it is haphazard. It hasn't been institutionalized, as you are recommending.

MR. WISE: One question, I was interested, you were talking about and emphasizing that, in addition to sort of technical expertise, it also called for good citizenship, listening and flexibility.

Is there some way that you have found that you can, either through training or monitoring, encourage this behavior, teach it?

MS. LEVITIN: I think it is really hard to teach good judgement. You certainly can teach policies and procedures, but I don't think you can train really effectively on some of these citizenship kinds of things that I mentioned.

So, then it really becomes a screening process, and you do need, as I said, SRAs to be good diagnosticians, to be able to ferret out -- you know, it is amazing, though, sometimes people will tell you that they don't like a particular kind of research.

I remember, years ago, someone telling me that anything using the Bailey scales, she thought the Bailey scales were really awful and she wouldn't support it. I thought, okay, thank you, because almost everything was using those scales.

I would like to go back to the point, though, about the extent to which, if at all, reviewers have a responsibility to provide tutorial kinds of things, because I think it is terribly important.

If you are going to be in a position to make some decisions about what kind of review process you want, I think that is one of the key issues that needs to be discussed.

I will tell you personally, I didn't like going from the tutorial to the non-tutorial. I think there was much value in doing it the other way, but I don't have hard data for you to demonstrate that.

MR. STANFIELD: I would like to follow up on your point. I think it is interesting, when you look at it from my perspective, that the citizenship and a whole bunch of what you might call personality profiling -- I am way beyond my own field now -- seems to differ among scientific fields.

So, there are scientific fields where they commonly eat their young, and it is very difficult to get new investigators started.

It is amazing, but you go from, especially the science that we review, from the social and behavior science, up to physical chemistry, and every area of science has a different ethic and a different personality.

I think that is going to be important to the panel when they consider different aspects of the peer review system, because you know the ethos of your community.

MS. LEVITIN: I don't think they eat their young. He is talking about the social and behavioral sciences.

MR. STANFIELD: I plead the fifth.

MR. FLETCHER: I do think we eat our young.

MS. LEVITIN: I think it is a little different. I think what it is, is that in some areas of science, there is much more agreement about method, much more agreement about approach.

When you have that, much more agreement about a sub-area of science, it is much easier to know what is really good and what is not so good and what is really lousy.

I believe, in the social and behavioral sciences, it is much more complicated, I think, to get up a large longitudinal study with seven different data points, at 25 different places across the country. There is not as much consensus as there might be about what buffer to use.

It may be less that social scientists want to eat their young, and probably leave their old out to die, than there are differences in the maturity or in the heterogeneity of approach or in the agreement among members of those sub-areas as to what constitutes good science.

MR. STANFIELD: what I would like to say, I disagree with Terry on this. There are certain segments, certain areas of science where folks just take our request to hold things confidential a lot differently than other areas.

In all sorts of aspects of the peer review system, it is just amazing how there are different arenas of science that have their own personalities.

MS. LEVITIN: I will agree with you on that.

MR. FLETCHER: I just want to respond to your comment. I think that the reason the behavioral scientists and social scientists eat their young is that they haven't gotten very good at weighting the significance versus the technical components of a grant proposal.

They consistently overweight technical components and don't take risks.

MS. LEVITIN: And your evidence for that is?

MR. FLETCHER: Craft knowledge, and experience as a reviewer. If you want to go to evidence, you don't have much evidence that the peer review system in NIH is very effective.

I mean, we don't know, for example, what the reliability of judgement is, for example. I don't think it is fair to hold me to a different evidentiary standard than you are holding for yourself.

MS. LEVITIN: I agree.

MR. HACKETT: Just a word about the tutorials. I think that I can see the point that, if you are having trouble finding reviewers, it is difficult to say, okay, and so, you should rewrite the methods section, we don't like it. I agree with Dom that you have to provide something.

I think there is a rhetorical fix, which is someplace in between, where you could say something like, you don't have to take the following advice, but I would consider a larger N using these other three measures, or look at this literature.

So, that is something less than a tutorial and more than it is wrong, it is wrong, it is wrong.

MS. LEVITIN: I think at one point -- I am trying to remember where I heard Varmus speak, when you asked that question about, is it in writing. I do remember hearing him speak at one point and talking about, well, who should be doing the tutoring.

His point was, well, that should be done at the home site, at your university, at your research institute, by your colleagues there, and it was really their responsibility to ensure that an application had the best possible form, not the responsibility of the reviewers.

Again, there is a lot of disagreement on that, and I end up thinking that it really was good to provide the additional information, particularly with new investigators.

MS. CHIPMAN: Yes, on that point, what if a person comes from a weak institution, where they don't have those resources, and your review panel may provide them.

It might depend on the context of the research field, where it is extremely competitive, there are a lot more good researchers out there than there are resources, versus you are having a hard time finding anybody to do research in this area, and therefore -- I mean, I may do this myself, dealing with an individual.

Their proposal may have serious flaws, but I want research done on this and this person shows some promise. So, I may work with them to develop a satisfactory approach.

MS. LEVITIN: Of course, program officers at NIH do provide that kind of assistance, and should be providing that kind of assistance to applicants.

MS. CHIPMAN: I don't have that much experience with the NIH review system, but I have seen a sort of concept of the fatal flaw.

The fatal flaw, actually, usually seems to be something that is easily fixed by someone telling somebody a better methodological approach, as opposed to what I think is a truly fatal flaw, which is, the research question is of no interest in the first place, or something like that, which is very difficult.

MS. LEVITIN: Again, with the NIH dual system, where program functions and review functions, although interlocking, are still separate, we depend on program staff to work with applicants, either for an initial application or for a revision.

MR. WISE: I think we have hit the end of a very long day. First, I want to thank the presenters for their time.

[Applause.]