MR. WISE: We have a few minutes left if there are other questions for either Diane or Penny or comments relative to this study.

MS. LAGEMANN: Can you say a little bit more about the instances where there were panels without people who had appropriate research credentials? Was that because they couldn't find them? Was it because they thought somehow teachers should be on these panels, or what was that about?

MS. AUGUST: I think it was a combination. I think partly people in the department didn't really understand the standards, which is one of the reasons that we advocating for instructions to this effect.

They thought you could take this mix and match approach, one sort of policy practice person, one with substantive expertise in the area, one researcher with content knowledge and one person with research expertise.

I think that is one reason. The other reason is, it is really hard for the department people to find these reviewers.

I mean, there is no status associated with this. People didn't get paid very well. It is a lot of work and, in many instances -- in some cases -- the department had like three weeks to find reviewers for all these applications.

So, it was a combination of lack of clarity with regard to the standards, as well as just the impossibility of actually creating workable panels.

That is why one of our major recommendations was to create a standing panel, where you actually get a group of people together who are committed to do this, where you endow some prestige in this endeavor.

You know, this is an important thing to do. You are really going to be working with your colleagues on this. You are going to do this for two or three years, with some accommodations for people who want to submit applications to these competitions.

Really, that is why we ended up with this idea of a standing panel as one possibility to deal with this. Given the way this was set up, nobody could have found the right reviewers.

It is not that people are not competent in the department. The thing was arranged in such a way that you just were not going to be successful.

MS. PETERSON: In the case of the FIS competitions, they didn't know what topics the applications would come in on. That is why they were three weeks before the end.

They did try to organize them around topics. I know the one I was on was school organization, or disorganization.

Then they had to sort of madly get these groups together and they had to make sure that that group would have 10 proposals. So, matching the topics with the number of proposals with reviewers that actually had expertise in that area.

MR. FLETCHER: Can I just jump on this point real quick? We heard yesterday from Lou Danielson that he still struggles with the same issues within the department because of uncertainties about when money is going to be available and how to schedule competitions and things of that sort.

In the President's Commission on special education, very similar recommendations were made, particularly noting the need to cultivate what the Commission report described as a review culture, where reviewing was seen as a contribution to the field, something that would enhance professional development and so on, which was felt to be missing.

MS. PETERSON: Exactly, absolutely.

MS. SCHNEIDER: I have one question, because this is really bothering me. The system that you are describing is so different than the system that I participated in for over probably seven or eight years in OERI and NIE.

I was kind of hoping that there might be somebody in the audience or somewhere here who could clarify why they went to this three person system, with these hundreds of reviewers that Penny is describing.

That certainly is very different than anything that I recall from nearly eight years serving as a reviewer for OERI and NIE. Does anybody here know anything about why they did this in 1996 or 1997?

MR. SLOANE: I don't know. However, in the inter-agency scenario, initiative, the department did post the competitions, but the department had the constraint with respect to how much money they could spend on the review process.

MS. AUGUST: That was a problem, too. It was one percent.

MR. SLOANE: That is very different with the NSF. We take money out of -- we consider the review process as part of the human resource component. As a consequence, we actually take money out of the competition to fund the reviews.

I am not making a statement about the department. I don't know what is going on there, but I do know that this year, at least, it was suggested that they have a financial problem. When you fund six out of 111, if the money was cut back --

MR. FLETCHER: One percent of nothing isn't very much.

MS. SCHNEIDER: The question I was asking and the point that I was trying to raise is that yesterday when Russ spoke, he felt that he needed to do something to strengthen the review process that was currently in place at IES.

The kinds of things that he was doing was, he wanted to improve the field and do a better review process to strengthen the field and, by having a stronger review process, the implication is that you get a stronger field.

In this particular situation, where you have 100 people at a hotel with little groups of three reading 110 proposals, and just trying to understand what was the rationale behind that.

Money wouldn't -- I mean, if you have 100 people coming to the hotel, that is a lot of money, as well as having a standing panel and asking people in the field to do these reviews.

It used to be a two-staged process. Proposals would go out to reviewers. People would bring them in and they would score them. The highest scores would come to panel and then panel would review them. I mean, that is what happened.

MR. WISE: Why don't we put this on our list of things we would like to find out a bit more about, and I think the staff can get a more definitive answer.

MR. FLETCHER: OSEP has a statutory limit of five percent that it can spend on review. My understanding is that NIH spends between one and two percent of their total budget on review, depending on the institute.

MS. ZEEDICK: A couple of times during your presentation you mentioned that applicant, assessment of reviews and their comments and how they could write better comments. One of your other expectations was that reviews would provide feedback.

Yesterday we were talking about how we don't want to tutor applicants. Doesn't giving more explicit feedback to an applicant make for better peer reviewers, teaching them how to make better comments, and then applications that come in, in the future, will be better. Therefore, we will save time and money and help to streamline the peer review process?

MS. AUGUST: Well, one of the things we recommended was that people who were doing the reviews be given as models examples of exemplary reviews, so that they could see what a really good review looked like, number one.

We also recommended that for each review criteria, for like national significance and overall design of the project, reviewers be given questions to ask of themselves, so that they would more thoroughly review those components of the applications.

So, sort of a rubric for reviewers to guide them through the review process. We thought this would be important, because then in looking at management and budget, you would be asking more specific questions that you would hope the application would answer and, if not, you could comment on that. That would strengthen the review.

MS. ZEEDICK: And with the number of reviews, I think yesterday there was the example of 60 applications, if we could streamline the process and standardize it more, I guess there would be less of the human error in the process by giving them a rubric. It might save us some time.

MS. AUGUST: We also recommended really reducing the number of applications that went through the full review process.

We were saying -- these were our examples, preliminary reviews to reduce the number of full applications receiving detailed evaluations.

There are quite a number that you know are not going to go anywhere. You don't provide a really detailed, or you provide a less detailed evaluation of those applications than the ones that you think have a chance of being funded.

That is sort of counter to the providing detailed reviews for people, so that next time around they have a better idea of what to do.

Another possibility was pre-applications with only a subset of applicants preparing applications, which we thought, at least, with center competitions, might not be such a bad idea, because these things took a really long time to review.

Also, people writing the center applications were telling us, it takes forever to write these things. So, why don't you let us do something smaller, outlining what we hope to do, see if we make the first cut. It will save us a lot of trouble, you will save yourself a lot of trouble.

In the end, you are just reviewing maybe the top three applications for the center competition.

Also, with regard to center competitions, there was no limit, at least, on the number of information appendices. So, these appendices went on for 322 pages, and nobody was reading them.

Absolutely, all those things you are talking about, we actually put in the report and made recommendations.

MS. ZEEDICK: I have another question about the recommendations, where you said the use of technology. What did you mean by that? Do you mean computer technology?

MS. AUGUST: One is just having the technology available for review, so you could actually do these reviews electronically.

The other thing we were thinking about is just having people sort of network. So, they write the reviews at home, and other people can see the reviews that have been written. So, when you arrive -- in fact, you wouldn't have to go anywhere. That is what we were talking about, exactly what NSF is doing now.

MS. PETERSON: I think it also helps -- they give you a template that helps you frame your review, whether it is giving you questions or it gives you a blank, and you have to write something there.

It does sort of guide you through how you should be doing your review and thinking about your review.

MS. AUGUST: That is what we were talking about in the report, too, that kind of a sort of prompt for the reviewers, questions that you have to respond to, so we get them to think more fully about all components of the design or budget or whatever.

MS. ZEEDICK: I think any preparation you can do before you come to the full peer review panel is certainly time saving. If NSF has fast lane, that is worth looking at.

MS. PETERSON: There is also a lot of peer pressure. If you don't have your review in and everybody else does, you start looking at other people's reviews and thinking, gosh, that was really detailed, and you look back at your own.

MS. DEMEREST: I am Betty Demerest(?), and I retired last summer from OERI after 31 years with the government, about half of it at OERI.

I agree that Diane's report is a wonderful description and analysis of what went on in that time period.

I can't speak to the last six months of OERI from first-hand knowledge, but I think, based on a sense prior to when I left, I think there may be some new issues that have surfaced since that time.

One of the last things I did before leaving OERI was to provide staff support for the cognition and learning grant competition, which was the first of that type they had.

So, my responsibilities included pulling reviewers from the lists we were given, sitting in on the sessions, reading the reports for accuracy, that were done by the panel, et cetera.

The lists I was given to call for that particularly competition was different from any I had ever seen before, in that it included very few people from schools of education, a very high proportion of people from schools of medicine, and also a very high proportion of people from academic departments outside of schools of education.

When I called these potential reviewers, some of them, particularly the people from schools of medicine said, I can't imagine why they are calling me. I have never done any work in the field of education.

One person said to me, from a school of medicine, that -- he said, if you are with the Department of Education, I presume you are interested in humans. I said, yes. He said, well, he wasn't interested in humans. He was only interested in mice.

I think an issue you might want to consider is what would be an appropriate proportion of people on a panel from schools of education versus people from fields outside of education.

I think another issue you may want to consider is politicization of research. I have lived through many, many different administrations and I think I would have to say that, in my judgement, the situation when I left earlier on, I would describe it as an unprecedented level of politicization.

I don't think there is that much written documentation of that, but you might want to look at Science Magazine. It has been doing quite a good job in their editorials and letters to the editorial, as part of that documentation about politicization around the government.

I think it does vary a great deal from agency to agency, you know, where it may be, in fact, more or less.

I think along those lines you may also want to look at a letter on the AERA web site that was sent by AERA and a number of other associations to the department on the issue of web sites, and what is being retained on web sites and what is being taken off web sites, particularly in the Department of Education. Those are two more recent issues that I think you might want to look at.

MS. PETERSON: I just wanted to comment on something following from Barbara's comment. It seems to me that one of the things that really strikes me now, looking back at this and based on you saying that the review process was completely different for FIS when you reviewed, is that it is really sort of sad in a way that we have to keep re-inventing.

It seems that OERI has kept re-inventing the peer review process. The peer review process keeps changing, compared to some of these other agencies that have well established review processes that they have been using for years and years and years and everybody trusts them, and they have knowledge and infrastructure to carry them out.

If nothing else, I think that is really something that is a little bit appalling. How can we accumulate knowledge, how can we do the research, how can we accumulate knowledge if we don't even have a review process that stays established, that has credibility and continues for more than, say, one administration?

MR. HENLEY: I am trying to understand the difference in culture. I don't understand why one has such difficulty getting reviewers in the National Science Foundation or in NASA. I don't think any of us have trouble getting reviewers, because everyone feels it is a professional obligation to help out. So, what causes the difference?

MR. AUGUST: I think that is a really good question.

MR. FLODEN: I think in the FIS competitions, one of the things that was problematic is that because it was the only discretionary money, a large number of people applied for these things.

So, you had, in different areas, a lot of people who would potentially be good reviewers were ineligible because they were competing.

When you have 200 proposals, often with multiple people on the proposal, at least sometimes people considered it problematic if you were reviewing when somebody at your institution was also in the competition.

That is a problem with the center reviews, too, in relatively small areas, where basically everybody in the country with a name was on some proposal. It was hard to find reviewers because there was nobody left.

MS. AUGUST: Right, which is one of the other reasons we were advocating for a standing panel, with some exceptions made, for people that actually wanted to submit applications.

We just sort of deal with this morale problem, you get a standing panel of colleagues, people you consider as colleagues, with some prestige associated with it, and you make a commitment to do that.

With FIS panels, you are right. Given the number of applications, it disqualifies people, at least in a given institute, in applying, but it wasn't just that.

MR. FLODEN: The other thing about building a culture -- this goes back to what Jack was saying -- that funding has been so irregular that it was hard to set up a regular culture because you didn't know when the next competition as going to be.

If you look at NSF in a number of different areas, they have a regular competition cycles. You can count on it year in and year out, there will be a competition twice a year or three times a year in a particular area.

It is a lot easier to set up a system that continues than the FIS that some years there was, some years there wasn't. It would be one time during a year, it would be no times, there would be priorities, there wouldn't be priorities.

MS. AUGUST: Right, exactly, that was a recommendation we made also, was to sort of standardize that.

MS. FALKENBERG: I have a question. What was the acceptance of these recommendations and were any of them ever implemented?

MS. AUGUST: You would have to ask OERI staff that. I am not sure.

MS. PETERSON: We don't think they were implemented. In fact, not too many copies of the report still exist, for some reason.

PARTICIPANT: I can't tell you, I wasn't there during that time, but most of the time, there was a period of time when we had a number of acting assistant secretaries, there was tension in terms of the leadership and the board. That is one major reason.

For specific recommendations and why they weren't implemented, I just wasn't there for that. If anybody else could speak to that, that would be great. That is the main reason.

MR. FLETCHER: I think you get to a certain amount of nihilism about the value of review when there is so little to distribute. When your success rates are so low, I mean, what is the point. I can't imagine that a certain degree of nihilism or cynicism wouldn't develop around the review process.