MS. PETERSON: First of all, I wanted to say that our panel was incredibly impressed with the research that Diane did. I think this is just an incredible study.

As somebody who admits this is not what she is most interested in, I think she really spent a lot of time, and we were very, very impressed with the work that she did.

I think there were a couple of things. It is hard to remember. I am glad that Diane said she had a hard time remembering this eight years ago.

I can't remember a whole lot about it, but I do remember that the panel was rather appalled, I guess. We were particularly concerned about the ratings of the quality of the reviews.

The fact that the minority of the reviews were good, just in terms of the substance and the depth of the reviews, and the majority were either indifferent or poor struck us as really appalling.

I think the second thing that we were really struck by was the match or lack of match between what was required to do the review and the expertise of the review panel.

So, the fact that on some of the FIS panels of three, that there was either no, or one, person who had research expertise out of three people, was of real concern to us.

Then, the fact that she found -- I don't know if Diane said this -- but she found that that was correlated with the quality of the review.

So, if you had a panel that did not have a researcher on it, or fewer researchers on it, their reviews actually tended to be more poor to indifferent.

So, the first recommendation we came up with was to enhance the match between the applications that needed to be reviewed and the reviewer expertise.

I guess it seems pretty straightforward that the reviewers should meet the requirements set forth by the standards but, in fact, they didn't, in what Diane found.

Now, we wanted to say that, you know, if you didn't have researchers on the panel, or you had one person, the people who didn't know anything about research perhaps shouldn't rate the research design, and the person who does know about research should rate the research design.

Another recommendation was just to increase the size of the panel because, as was pointed out, three is pretty small.

Because Diane has done some analyses of what was going on in the other agencies, we did suggest that we thought it would be a good idea to have a standing peer review panel for each institute, of the time that I think at that time it was NIH.

We also suggested to improve the data base of reviewers. It wasn't clear how people got selected as reviewers. A lot of times I guess it seemed like they were scraping the bottom of the barrel to get enough people to come to Washington.

I was actually in the competition that really all -- the one where we physically came together and we were in panels of three.

I remember that it was just kind of a zoo. They were all in this hotel and each of the three panels were in different rooms, and there were hundreds and hundreds of people there, and we had to all do it simultaneously.

In fact, there were a lot of people there that I didn't know, that I didn't recognize, a lot of real heavy duty researchers.

I guess there were two things, just to increase the amount of time available to the reviewers, and have the submission dates staggered by institute.

At this particular FIS competition, the proposals all came in at once, and therefore, they had to review them all at once on the very same day, and that is why they were all there in this hotel, working madly.

The second recommendation is to reduce the reviewer workload, decrease the number of applications to be reviewed, provide detailed evaluations for competitive applications only, increase logistics and other support, and disqualify applications that are not research based.

That wasn't done in this case -- is that right, Diane? They actually reviewed all the applications. There wasn't any kind of screening that occurred.

MS. AUGUST: There is some screening. We found, like over the two years at FIS, there were six studies that were included that weren't reviewed.

MS. PETERSON: So, there were some other recommendations about ways to reduce the reviewer work load, provide multiple submission dates, and assign primary, secondary and tertiary reviewers to each of the applications.

A third recommendation -- and I think we saw a great example of this yesterday, how NIH did professional development, I was very, very impressed -- is just to bolster the professional development that is provided to the reviewers.

There wasn't much in the FIS, as I recall. That is what we found, and they gave us a few little descriptions about procedures of what we were supposed to do, but it was not certainly as extensive as we heard yesterday that there was from NIH.

Also, we suggested providing professional development for the OERI staff, to ensure that they understand the requirements of the standards.

MR. FLETCHER (Committee on Research in Education): I don't have a question. I just want to point out that what Terry Levitin was showing yesterday in terms of professional development is something that she was proposing. They don't actually do that.

MS. PETERSON: Oh, they don't?

MR. FLETCHER: No.

MS. PETERSON: There is some kind of professional development at NIH.

PARTICIPANT: It is at that one institute.

MR. FLETCHER: It is not widespread or systematic or anything of that sort.

MS. PETERSON: Do you think it should be?

MR. FLETCHER: Well, I mean, we can talk about that later, obviously. I just didn't want the committee to think that that was NIH-wide.

MS. PETERSON: Okay, thanks. The fourth recommendation was to clarify the standards. There was some lack of understanding of what the standards meant, for example, that FIS has multiple perspectives being represented. Did that mean multiple perspectives in terms of ethnicity, gender, region. How does that get played out in terms of the panel.

Define the conflict of interest and the relationship of people, their professional relationships.

I think just clarifying and getting a shared understanding not only on the part of the OERI staff, but also on the part of reviewers, what the standards meant and required.

Then, just simply modifying the review criteria, and clarifying the meaning of national significance, I think that some of the recommendations came out of the difference between the reviews for the centers and the FIS competition. Is that right, Diane? There was a difference rating for management for center applications, which is obviously much more important that you have a good management in a center, but yet, that same standard was used for FIS competitions, where you might have just one researcher proposing the project.

MS. AUGUST: The other problem -- which actually created a lot of problems for the center reviews -- was not just management issues. People didn't know what to say and it wasn't given enough weight for a center, but the project design criteria was framed in the context of one site.

So, people were using the same criteria to rate the FIS and grant applications. So, the reviewers really didn't know what to do with it. They were rating a whole series of studies under the center competition as proposed studies, but what they were being asked to rate was like really one study, because it was sort of FIS type standards that were being applied for rating the center applications and this was really problematic.

MS. PETERSON: And the meaning of national significance was different, too.

MS. AUGUST: Yes, exactly, but I think the major issue was the rating for management under center proposals and the fact that you really needed a different criteria for rating project designs for a center than you do for like one research site.

So, what happened was, the center reviewers, is they tended to write really globally about the research designs of the center projects because they really didn't know what to do.

MS. PETERSON: So, one of the other recommendations was to not employ sort of a one size fits all for both the FIS and the center competitions. That is, don't have a standardized rating form, but to think about how the national center competition should be set up, what you are going to be rating that is different from the FIS.

MS. TOWNE (Committee on Research in Education) : Could I ask a question about that? Do you think that could have been done within the legislative framework? Weren't the standards legislatively mandated?

MS. PETERSON: Yes, they were, but you could interpret them differently. I think about what happens in the state of Illinois, when something gets put into the standards, you can still interpret the wording slightly differently. Do you have a comment on that, Diane?

MS. AUGUST: No, I think that is right, though.

MS. PETERSON: So, the idea is not having one size fit all across the two kinds of competition, which were very different.

Provide feedback on successful applicants. I think yesterday we talked a lot about the need to, for purposes of providing professional development, to think about why they need to get feedback, but also good feedback rather than poor or indifferent feedback, one would think.

Finally, explore the use of technology. When this particular competition occurred, as I recall, they had a whole bunch of computers. It was all done in a hotel and they had a whole bunch of computers. So, we did all this work on the computer.

It was nothing like the fast lane set up that they have at NSF where everybody is sitting there, you have a screen in front of you and they are putting everything in in real time, live, and seeing what each other had written.

As I recall -- maybe Diane can comment on this -- there really wasn't much opportunity at all like that. I didn't see what anybody else had written on my panel. There was discussion, but you never saw any text that anybody else had written. So, questions?

MR. WISE: Could you back up two slides? Recommendation four. What was the last bullet about?

MS. AUGUST: Listing priorities for FIS competitions. I think this was under the FIS competitions. The department said, we are interested in applications in this area.

MR. WISE: The current practice seems to be being even more specific.

MS. PETERSON: It is interesting. When Russ said yesterday that he had no clear priorities, I think what was said here is actually different from that, that you shouldn't list those areas.

MS. AUGUST: I am trying to remember exactly why it came up. Do you remember, Penny?

MS. PETERSON: I actually don't.

MS. AUGUST: Panelists interviewed supported the use of priorities for directed research competition, but not for FIS competition.

Other agencies with more funds available for research fund several types of competition. Research priorities are not specified for FIS grant competitions, whereas they are specified for direct research grant competitions.

MS. PETERSON: Maybe that is the view the field. The field doesn't want competition --

MS. AUGUST: I think this was the issue. The issue is there is so little money for these FIS grants that people felt like it should be more open to the field, to send in things they were really interested in.

MR. WISE: So, it is not really about peer review.

MS. AUGUST: I think it was both. I think there was some concern about that, too, that there is hardly any money, and OERI is setting the agenda here, and people in the field didn't like that. They really wanted, given the amount of money available, to be able to propose things they thought were important.

MS. PETERSON: And there is a lot of money tied up in the centers, I think. That makes people feel like there needs to be some leeway.

MS. AUGUST: That was the other thing, that there was so much of this money for research tied up in the centers, as you all know, that people felt that, for what was left, if it was discretionary, it needed to be more open. I have the numbers here.

MS. SCHNEIDER (Committee on Research in Education): I think part of the problem, as I recall, is that this is not systematic research, but just personal observations, that FIS is really the place that people went to get basic research dollars.

You went to centers or you went to FIS, and there were very limited opportunities for people. What they didn't want to have happen is have that dry up as a place to be able to study theory and other -- it really was pretty open, and it was open -- another thing that I think people were unhappy about is when FIS became something that looked very much at specific kinds of practices in evaluations.

People who were doing more basic kinds of research questions felt that they had nowhere to go.

MS. AUGUST: I think that is absolutely right. I think that is why people in the field did not want the FIS money to be tied up with the OERI diagnostics.

MS. PETERSON: Before you say that, one of the things I was just going to say, they have these three person panels. So, then, to go across the panels, they had to use standard high scores based on an N of three on each panel. That is how they came up with the number of the ones, because they had to get them all on the same scale. I just wanted to mention that methodological point.

MS. AUGUST: Do you want some numbers? This is really unbelievable. It is even worse than what I remembered. Fiscal year 1996, the achievement center, 13 funded FIS studies out of 190 and, in 1997, five out of 106, for at risk, 15 funded out of 201, and in 1997, six funded out of 198.

For early childhood, seven funded out of 111 in 1996, six funded out of 104 in 1997. In the policy institute, five funded out of 49 in 1996 and six funded out of 45 in 1997. Post-secondary, six funded out of 92 in 1996, and six funded out of 79 in 1997.

Can I also just clarify, because I also had in my notes here how we sampled. For the FIS panels, for the five successful panels -- that means, for the five panels that the department had considered successful, we interviewed all panelists and the successful applicants.

They nominated, for each of these agencies, the panel that they considered very successful. For the remaining 15 panels, we interviewed two randomly selected panel members and two applicants for each.

Successful applicants associated with a panel were automatically included. If there were not successful applicants, two randomly selected unsuccessful applicants were interviewed.

For the center competitions, we interviewed three applicants, including the successful applicant in each of these competitions. These are the seven that we didn't do in depth.

As with the FIS competitions, for each of the other seven center competitions, we constructed a normative picture.

We interviewed two randomly selected reviewers and one randomly selected unsuccessful applicant, as well as a successful applicant.