MS. CHIPMAN: My talk is entitled, proposal review at ONR, because we don't do what is ordinarily considered peer review. We have a different approach.

The typical mode of review at ONR in program areas is really individual decision by the responsible program officers, obviously under supervision. That is, there are several people up the chain who sign off on the recommendations for funding.

Like NSF and I assume NIH, too, it really is the contracts and grants office that truly makes the awards, but we make the recommendations.

There are all kinds of other processes that are used. Maybe we will have a special program initiative cutting across several departments leading to a committee of program officers who will review the proposals received.

We have some big so-called multidisciplinary university research initiative awards from the IOSC, typically $1 million a year for five years, on topics that we think we publish.

For instance, I have two now that are on tutorial dialogue, natural language interaction, capability for intelligent query systems, and one that is called automated skills for cognitive readiness, and is largely fMRI research looking at the development of skills.

Those would have at least three reviewers, maybe a dozen reviewers. All the services have to be represented amongst the reviewers, whether it is inside government people, small business initiative things.

Again, topics that we publish for competitive votes must have three reviewers, which are always government people, although not necessarily from our own agency. They might include representatives of the customers, because these are somewhat applied, usually, or they might include people from military labs with related research interest.

Sometimes we use traditional external peer review. The biologists, who are my close organizational neighbors, do use this approach. They contract with a professional society, and we think of them as being lazy for doing this, for unloading some of the work.

I might say, ironically, the one little scandal I have noticed, at least, having to do with awards, involved biologists. So, it wasn't insurance against problems.

Why do we do this? Why was this approach selected for the cognitive science programs in ONR? It is because of tradition.

I wish I had the Fiddler on the Roof for you, but as Steve said, I came to this agency and it had a long-standing tradition, and this is how things are done. So, we do that.

There might be some background behind this. Organizational context. ONR is a mission-oriented agency. Our goal is to support scientific research that will do good things for the navy, maybe on a very long time frame.

ONR has always had a strong basic research orientation, and 30 years in terms of pay off is both considered reasonable and reflects the truth.

For instance, if you want to talk about the area of artificial intelligence tutoring systems, ONR awarded the first contracts in that area to Bolt, Muranic and Newman(?) in 1969.

Approximately 30 years later, we saw the first examples of an intelligent tutor that was purchased by the navy as a regular training system procurement. There aren't very many of them yet. This teaches the operation of the spy-1 radar.

Now, it took a $2 million computer to do this work in 1969 and much later I found NSF science education programs weren't willing to invest in something that was that far off in the future in terms of practicality, in classroom applications.

Even in the long-range basic research programs, we are not supporting anything and everything that is good science, but particular areas that we think will pay off for the navy.

So, somebody might send us a proposal and we might think it is great science in our field, but still not decide to fund it because it doesn't fit into a coherent program.

The program officers are responsible to plan a coherent program within the available resources. So, we may restrict what we are investing in quite narrowly, because we only have that much money available.

We are called upon to defend the program in some detail and, in serious presentations to both the agency leadership and usually some outside reviewers, similar to the visitors that Steve presented, and this can be a fairly stressful experience.

In fact, if some of you are familiar with some of the things that are said when people talk about the participation, for instance, of women in science and the kind of nasty cultural attack, ONR exemplifies that.

So, we endeavor -- another part of our job is to endeavor to see that the potential applications of our research are pursued. We are expected to do that.

To give you a recent example of some of my long-term basic research grantees, Ventner Gettner(?) and Ken Forbis have worked on learning and reasoning from analogy, recall within cognitive modeling of the processes by which past experiences are called up from memory and judged to be good analogies.

Well, recently, Gettner(?), in collaboration with a business school professor, demonstrated spectacularly large improvements in transfer of training from the very minor manipulation of the way cases are used in instruction.

So, I wrote a small business initiative package for an authoring tool for case-based instruction which hopefully will include, as defaults, the optimum way of using those cases to maximize training. So, that kind of thing. Then the other kind of pursuit of potential applications are expected.

We also sell new program initiatives and try to garner more resources for our programs. So, we might, for instance, invest in a few grants on a particular topic over a period of years, developing the case to ask for more money.

Now, this was characterized by, for instance, the tutorial fisbars(?) kind of thing, where I have invested really pretty small amounts of money over a period of quite a few years, at least 10 or 12 years and, in fact, probably ONR has supported most of the research done on human tutorial interaction, despite the fact that there hasn't been that much money.

That gave me the basis for getting OSD to publish this topic and make bigger grants. People do this in other areas, of course, but definitely.

Part of that is, we identify gaps and opportunities that are opening up that typically become the basis for the sales pitch.

Certainly, for instance, the advent of fMRI research opened up a lot of new possibilities for research looking at brain activity in a different way from past traditions.

For instance, I sold a program about 15 years ago called neuroconstraints on cognitive architecture, which was perhaps a little bit premature, but did recruit some of our leading cognitive experimental psychologists for doing this kind of research and has had some impact.

Program officers I see as being really something like brokers, between the interests of the researchers and the goals of the agencies.

So, we are not sitting there waiting for proposals to come over the transom and simply review them ourselves.

It is similar, in some ways, maybe, to managing a big research center, but a distributed one, where we have the potential, without having to get them to move, let's say, and employing the most outstanding people in the country or, indeed, the world, because we do have the authority to give grants outside the United States with some specifications.

Our goal is not to support science in the United States, but to support science that will achieve the goals that the navy is interested in.

The funding process becomes kinds of negotiative. We prefer to start by talking to people on the phone, at a scientific meeting. Maybe they come into the office and ask to meet with me.

I will meet with them if I possibly can. We begin with a little discussion to establish that there is some mutual interest here.

Maybe a researcher has several possibilities and directions they might take, and a conversation will help them select among them, the one that is the most interesting.

Often, I have had conversations about general research issues that a person wants to address at a basic research level.

If it is that basic and general, it doesn't really matter that much what the context is, but instead of something that is going to be seen as a toy or silly or worthy of the golden fleece, maybe I can get them to address, say, a general issue of cognition addressing general problem solving in the context of basic electricity and electronics.

The realism of the budgets and the current availability of funds can be explored. Some people had in mind getting a lot more money than we can ever give them.

If somebody calls me this week, they are going to get the message, well, you can be very leisurely about developing your research ideas, because I spent all my money before the fiscal year began, in order to make sure that it would not -- well, I am not absolutely sure, but fairly sure, that it would not be allocated to other purposes, such as buying bombs to drop on Iraq. This does happen, by the way.

After all this, if a proposal is actually encouraged -- and often, depending on the time of year and so on, there may be a 10-page concept paper first -- the probability of getting funded is very high.

It is not absolute, because sometimes people really disappoint you when it gets down to putting things on paper, but that way, often, the researcher gets a very soft let down, they don't waste their time preparing a proposal that has no prayer of ever being funded, and of course, we don't have to spend the time to read a long proposal.

I do have some pretty wide experience with review of all kinds, including the traditional peer review. In fact, when I went to work for NIE in 1976, the first thing i did was assuming responsibility for a very large grants competition that was underway.

In the program office, the guy in charge -- the guy is probably years younger than I am now -- had crumbled under the stress and literally disappeared. He did reappear later, he wasn't murdered, but he vanished, and I took this over.

Just let me say it looks like the Department of Education is moving back to the way we granted some of the competitions.

We use large numbers of field reviewers. We use those field reviews to cut down from typically 100 proposals that are going after 10 awards on a topic like reading comprehension.

We would then send 30 to a panel of around eight people. I was using things like, read the borderline cases, decide which ones would go to the panels.

If somebody found some aspect of those exciting, as opposed to the universe of, oh, yuck, and it sounds like they are going back.

The NIE NSF program, let me just mention this briefly. Somebody at NSF decided, every proposal in this thing should go to two panels.

I can tell you, the panel process is not reliable. Contrary to what somebody said, having a panel does not produce any more reliable result. I don't know that anybody has really made use of this. If it was an experiment, they did not have my informed consent, but it was not reliable, and it is because I can bring together a panel I sometimes think of as applied social psychology.

What happens, when you bring these people together, they can be very unpredictable. On one of these panels, somebody may, with one point of view, may draw all the attention.