MR. SLOANE: I would like to apologize before I begin. My mind just doesn't flow the same way as Steve's. So, mine will be a lot more choppy.

Actually, Ms. Bertenthal had sent us a series of questions, I think, last Friday. I got them Saturday. So, it is choppy because I am responding to the questions.

The questions included, what were the policy goals that overlooked and directed decision making at the NSF. How are proposals selected under that.

The policy ones, I think, are quite different, I think, than the other agencies. In a very general sense, what I would like to suggest to the audience is, if one was to think about the medical model for research, where there are about four different layers -- and Jack would be much better able to articulate this than I am -- the early layers are layers of feasibility, and the degree of methodological control and rigor increases as you move up these layers through a consensus model, which NIH actually runs, when there is a large national trial.

NSF's cultural slot in that continuum is much closer to the feasibility, this four-stage process. I think that is something we should keep in mind when we are thinking about the criteria that Steve and Brent will be presenting here.

One other thing that goes on in the medical model, to sort of set a framework, is that across each of the four stages, there are about 10 studies at the feasibility stage before one study at the phase two stage. This 10 to one ratio seems to be real across.

I am going to basically suggest that culturally we slot in at the earlier end. So, later on in my talk, we were asked, you know, what were we doing, what does our agency do particularly to support innovation. There was a visceral response of, well, that is sort of where we are.

We were asked, how were proposals selected for review. Basically, any proposal that is submitted goes through the review process.

There are ways to knock proposals out of the review process, but we don't execute them, generally speaking.

In another context -- Steve said that he represents SPB, and I represent education and human resources. The human resource side of the NSF is a very critical component of the agency.

So, within education and human resources, we have a division on graduate education, a division for undergraduate education. Much of the dollars that come through EHR actually go to things like scholarships for students in physics, mathematics and engineering, although they would be EHR dollars.

In reality, EHR's research portfolio in terms of dollars is significantly smaller than the actual dollar allocation in NSF as a whole.

The directorate, as a whole, got somewhere in the range of $700 to $800 million, and research dollars within that -- that would include research evaluation and information -- is probably -- I know it is much less than 10 percent. It is split across a number of programs, and it is probably closer to the range of seven to eight percent.

So, all proposals are submitted electronically. In some programs, we ask for pre-proposals. These proposals get reviewed internally and often go off to ad hoc reviewers.

When we ask for pre-proposals, we can ask for pre-proposals under two bases. We can make them compulsory and we can make them non-compulsory, which means that you can still get a proposal into the hopper later in the game.

Also, the feedback to the PIs in advance of the proposal phase, in the programs that I represent, which I am program manager for AERA, the educational research initiative is a cross-agency initiative. So, I speak only to the ROLE program, which is research on learning and education, because it is specific to NSF only.

Every piece of information that goes out to a particular PI before submission of the proposal, is provided free of charge. You can choose to ignore it, you can choose to pay attention to it. None of that information is provided to the panel. So, you go past go, you collect $200.

At the second stage, then, we have the full proposal process. We were asked, what were the methods for review. The questions were simply -- my understanding of the question, at least, was it following an electronic system, and the answer is yes. All the reviews are stored electronically, whether they be panel reviews or mail reviews.

What we consider to be the strengths of the peer review system, as it manifests itself in the agency, are that we believe it elevates the quality of the submissions following the pre-proposal stage, as it affects the full proposal stage. We also believe that feed back loops, also, over time, improve the quality of the submissions.

Another single component that we believe is important is that the review process allows public notice of the panel. So, any panel that is upcoming, you guys can come watch. Very few people come to watch. It is just not that entertaining.

The electronic system, to echo Steve's position, is now regarded as state of the art. It wasn't always state of the art. It was state of the art at points in time.

For many of us, much like Steve, I also came to the NSF as a rotator. Before that period, I served as a reviewer for a number of years, and we get some very interesting interactions, when you print off your review and share it with other people, as well as trying to figure out the electronic system that supports it. The system is actually much, much stronger. Iteratively, it is getting better.

Another strength of the system, we believe that the panelists benefit from the review process. Again, that is a critical component within EHR.

Within EHR, in the work that we are involved in, somebody asked earlier, simply put, if it isn't the money, why do the people do it.

I think there is a huge return on investment for serving on a panel. You get a sense, what is dragged out of one's own focus area and attention, to get a sense for what national questions other people are posing, and response to those questions.

You get a sense for what does it actually mean when the NSF uses large grant terms like intellectual merit. What does that mean on a very practical level.

There are a lot of other -- I mean, I believe it was Professor Dodge who said that it was the actual interactions alone.

From the program staff side, sitting in panel is probably the best part of my work, to hear other people who are current in the literature, articulate a case for or against something.

You should keep in mind that we actually read the proposals before we come to panel. As program staff, we are required. So, it is kind of fun to hear people echo things. It is self serving, but it is fun. It is also very illuminating to find out what you missed.

Another strength of the system, as we see it, is the opportunity for ad hoc review, either pre or post panel. So, to try to generate some balance, because we do have this problem of how do you generate a balanced review process, I was thinking, if you were going to literally do that, you would have maybe 300 or 400 people on a panel.

I am being facetious. I am know that you can be very serious, but to do this well, it takes a lot of work and effort. It takes a lot of knowledge. It is the willingness of panels to come and interact as equals that makes it work.

We also believe that turn around time is a critical component of the electronic system.

What do we consider are challenges to the system as they manifest themselves in the program that i work in? What is an optimal N for review?

Steve had mentioned that. Legally, we are required to have at least three reviewers. Is three enough? Is five enough? Is 10 enough?

I know that in work that we are doing for IES, we are kind of negotiating this for inter-agency sharing. It is hard to know what are the optimal Ns.

It is also very difficult to know how those people should be drawn, what areas of expertise should you have represented.

In that regard, the pre-proposal phase, or in other programs, we ask for letters of intent. While people use those for their own information, the agency also uses them to make sure that the panels are manned appropriately, so we don't get blind sided quite as easily in terms of what it is we are going to be reviewing.

What is the optimal N? How do you generate a sensible inter-rater reliability both within panels and across panels?

We, in our part of the agency, actually don't have standing panels. Part of this, you will realize, is associated with our human resource strategy. So, when folks were talking earlier about the notion of should you have younger or junior or whatever the right people are, to be involved? We actually have a strategy that fosters that.

Another challenge, that we consider a challenge is, what is the reliability when proposals are submitted a second time, and how do you try to make the process, the second time through, as reliable as possible.

One of the things that we do, given that we don't have standing panels, is to use the ad hoc review process carefully.

So, you can actually go back to people who reviewed it the first time and ask them if they think that the proposal responded to the panel's review, for example.

We also suggest to folks who are submitting a second time, that they highlight in the proposal that this is a second submission, that the first panel suggested that they pay attention to X, Y and Z, and that they highlight that for the second panel.

We also try to have at least one of the reviewers from the previous panel come and make a case for or against the proposal the second time.

While I know that turnaround time is a positive in the system, I also know that it is a negative in the system.

We were also asked, what provisions do we have in place for supporting research by young scientists. I highlight a number of programs in a certain sense, and positions that we take.

One is that we have an early faculty career program which actively encourages untenured people to apply, and programs of research enrolled are for studies that are three years in duration, and the early career program is five years in duration. It allows the young researcher to be brought out for some duration of their five-year period.

We utilize a program called SGERS, which is small grants for exploratory research, and they are often used to support young researchers.

I know Barbara is in the AERA in a joint program that is funded by the ROLE program by NCES. Within that program, there are small monetary rewards for dissertations and also for program fellows.

So, there is a way to help move young scientists along in their career in their professions.

Another important point for us is the panel review process, and how we introduce young reviewers to that process. Again, this use of younger or junior reviewers, we make an effort to have about 20 or 25 percent of our panels be people who are not tenured.

Another phenomenon within the NSF -- and Steve alluded to this -- is that there are multiple sources of money within NSF.

There is a program within the NSF at large called the Experimental Program that stimulates competitive research. Locally, we refer to it as ABSCOR(?). Basically, ABSCOR is to stimulate research in states that don't have RO1s. We have more than 20 states in that category at this point in time.

So, when you see a proposal that is on the bubble that is strong and worthy of funding but doesn't make the cut, if it is from an ABSCOR state, we can get on the phone to another division or another director and beg some dollars to support that researcher, whether they be a junior researcher or a senior researcher.

Also, I would say that, amongst the program staff there is a general ethos of supporting as many people as we think is sensible.

How do we support innovative research? Again, as I mentioned earlier, when I put these slides together, one of the things I sort of said to myself is, is this we believe that all research is innovative itself, by definition.

The federal government assumes a higher risk in innovative research than other sectors. So, it expects that we are going to flop, to some degree.

My memory of when Rita Collin(?) came aboard the NSF, there was a rather blunt e mail to every program officer which basically said, why are you here. It said, if we are only funding excellent research, if our research all receives an excellent rating across the board, we do not need programs.

The notion behind the e mail was to suggest that we are in the business of funding higher risk stuff. I have just been informed of that, although I will not articulate it in another 10 pages of overheads.

Thank you. At 3:00 o'clock, when this session is over, I will be leaving, not because I am not interested in what will go on for the rest of the day, but thank you for having me.

MR. FLETCHER (Committee on Research in Education): Could we ask Barry to clarify something before we go on? Barry, you said that -- you basically implied that panels at EHR are open to the public. Is that what you intended when you told us?

MR. SLOANE: Yes, open and independent.

MR. FLETCHER: So, if my grant is being reviewed, I can come to the panel.

MR. SLOANE: Yes. Some people who come to the review and listen are people, the public --

MR. FLETCHER: They are not my competitors, for example. If Barbara wanted to find out what is going on down in Houston --

MR. SLOANE: No, I am talking about the ordinary lay person.

MR. FLETCHER: That is what I wanted to clarify.

MR. BRECKLER: There is a little qualification of that. Technically, panels at NSF are exempt from the government in the sunshine act, whatever it is called, which means that, in fact, parts of the panel meetings where proposals are being discussed are not open to the public, because they contain confidential information like people's salaries and stuff like that. So, you can't just walk in.

What is required is that all panel meetings must be posted and the public notified that they are happening, so that people know when they are happening.

What is required is that all public meetings like that devote at least a certain portion of their time each calendar year to being literally open where people can come in.

So, we do that, but we won't be discussing proposals at the time. We will be discussing other things. So, it is actually a mixed model. The parts of it where you talk about the juicy stuff are not open.

MR. FLETCHER: Now I feel better.

MS. SCHNEIDER (Committee on Research in Education): Since you are going to leave, I have one question, too, that I wanted to ask you, and that is that a quarter of the panel is junior people. In light of the kinds of things that we were talking about before, could you say something about the criteria for the selection of those junior people?

MR. SLOANE: Generally, we are looking for people who have published, not necessarily that they have received a large NSF or any other agency grant. We are looking at the publication record.

MS. SCHNEIDER: Are these publications in reviewed journals?

MR. SLOANE: Publications in reviewed journals, that kind of thing, or refereed presentations at AERA.

MS. SCHNEIDER: These junior people, you talked about having these experiences, and are there letters that people checked with who these people are?

MR. SLOANE: Yes, and also Steve's point that we don't hand this work out to an outside agency. Program staff do the work.

It is difficult to get people. Let me echo, again, Steve's position. One of the things to keep in mind when you are doing educational research is that school starts September 1. If you want to do something in this area, you have to have interaction with that during the summer.

The government funding cycle, in terms of when decisions are made, when we actually have dollars to spend, don't always align with the research cycle in certain schools.

The other component part of that is that, at the end, they internally meet at NSF. The fiscal year closes September 30.

We have one group, one of these slides, Steve said the program officer is a different beat at the NSF than at other agencies. There are other layers of sign off.

In the division of grants and awards, the program staff actually doesn't make an award. The program staff makes a case to the agency for an award, but the program officer never makes the award. The division of grants and awards makes the award to the university.

As a consequence of having one office that supports all of those decisions, financial decisions or recommendations have to be made much earlier than September 30, so you can get your budget cleaned up by the end of the fiscal year.

That, in turn, impacts when you can have a panel in the summer. Having panels in the summer is influenced by people taking summer vacations.

MS. SCHNEIDER: You know that the junior people never take summer vacations.

MR. SLOANE: They don't. They just work like crazy. We have all been junior people.