The National Academies
Committee on Science, Engineering, and Public Policy (COSEPUP)

Experiments in International Benchmarking
of US Research Fields

Statement of

Marye Anne Fox
Chair, COSEPUP Benchmarking Guidance Group
Chancellor, North Carolina State University

David Challoner
Member, COSEPUP Benchmarking Guidance Group
Director, Institute for Science and Health Policy, and
Vice Pres. for Health Affairs Emeritus, University of Florida

at

Public Briefing

March 2, 2000

MARYE ANNE FOX:

Let me begin with a few words about what we have come to call international benchmarking. This is a way of comparing the quality and impact of research in one country or region with world standards of excellence. In our experiment, we have tested the use of benchmarking as a tool to understand the relative world standing of US research in a field and to understand the factors that are critical to US leadership in that field.

Why do we want to evaluate the US status of a research field in a global context? We, as a nation, support a large research enterprise, and naturally the public expects returns on their investment. In general, there is little doubt that science and technology are powerful tools in moving the nation toward social and economic goals. The National Research Council has already prepared studies that describe the effectiveness of research investments in addressing national concerns. However, it has proved difficult to devise a consistently fair and effective mechanism for allocating federal funds for research. Furthermore, it is seldom possible to predict the outcomes of basic research or to know which fields of research will ultimately contribute to important new ideas or technologies. So we would like to have a reliable tool that allows us to evaluate the quality of our research programs and to provide us with a basis for funding allocations.

Let me give a little more background. In 1993, when COSEPUP issued its report Science, Technology, and the Federal Government: National Goals for a New Era, it recommended that the federal government would continue vigorous funding of basic research and that it would seek to support this research across the entire spectrum of scientific and technological investigation. Specifically, the report made two recommendations: first, that

The United States should be among the world leaders in all major areas of science,

and second, that

The United States should maintain clear leadership in some major areas of science.

By doing so, the U.S. would position itself among world leaders in all major fields of research so it could be ready to apply and capitalize on research advances wherever they occur.

Two years later, in 1995, a committee (on which I was a member) chaired by Frank Press, the former president of the National Academy of Sciences, stated that to continue as a world leader, the United States should strive for clear leadership in the most promising areas of science and technology and those deemed most important to our national goals. In other major fields, the United States should perform on a par with other nations so that it is ‘poised to pounce’ if future discoveries increase the importance of one of these fields.

To go back to COSEPUP’s goals report, the committee also considered the difficult question of how to measure leadership in research. In other words, how can the federal government gauge the overall health of the research enterprise as a whole and in its parts and determine whether national funding is adequate and whether it supports national research objectives. The committee wrote that it is feasible to monitor US performance with field-by-field peer assessments, and that this might be done through and I quote:

At that time, the technique of comparative international assessments, or what we came to call international benchmarking, had been discussed in theory, but had not been reduced to practice. Therefore, COSEPUP made the decision to undertake a set of experiments to test its utility in evaluating entire research fields. And I emphasize these were experiments.

The committee acknowledged that quantitative indicators are often used to assess research programs for example, dollars spent, number of papers cited, and number of researchers supported. This is all valuable information, and it would be used by the panels. But these indicators by themselves are not sufficient indicators of leadership, both because quantitative information is often difficult to obtain or compare across national borders, and because it only illuminates a portion of the research process.

COSEPUP decided that benchmarking should rely more prominently on the judgement of experts. The premise for this decision was that only leaders of a research field are in a position to judge leadership. COSEPUP charged each panel to provide answers to three primary questions:

For these experiments, the committee deliberately chose difficult subjects three fields that have a range of scope and subject matter. Of the three, mathematics is the closest to being a traditional discipline, but even mathematics is broad in the sense that it has a language and is a tool used by other research fields. Immunology is not a disciplinary field in the traditional sense (for example, there are not generally immunology departments in universities); rather, it embraces many disciplines including biochemistry, genetics, and microbiology. Materials science and engineering have even a broader span of disciplines.

DAVID CHALLONER:

I will now describe the results of the benchmarking experiments. Three fields: mathematics, materials science and engineering, and immunology were selected for analysis.

A key step is obviously the selection of panel members and each panel was nominated by an oversight group that consisted of COSEPUP and non-COSEPUP members who were experts in the field. As this oversight group developed the panel slate, they looked for a balance of US academic researchers and non-US academic researchers and non-academic researchers in that field. In addition, all needed to have a broad understanding of their field and an international perspective.

For example, of the 12 members of the mathematics panel, three were non-US researchers including a current and a former president of the respective national academies of sciences in their countries. Two were US researchers employed by industry, and one was a US Nobel Prize winning chemist who uses mathematics in his research. The remaining six were US academic researchers in mathematics.

The panels met only one time for one and half days with a number of telephone conference calls before and after the in-person meeting. The first step was to divide the panel into subfields not an easy task. Each panel then used a variety of methods to assess its field. The methods used included

The most unique of these, as I mentioned, is the virtual congress. All the panels used this technique and found it to be useful. The way this works is that each panel asked leading experts in the field to identify the best of the best researchers for particular subfields, anywhere in the world. The panel members themselves polled respected leaders on an international basis in person, by phone, or by email; taking into account the number of US and non-US researchers polled. The pollees were asked to imagine themselves as organizers of a session on their particular subfield or sub-subfield and to furnish a list of 5-20 current desired speakers.

Each panel concluded that the US was at least among the world leaders in its field. However, each panel identified subfields in which the United States lagged the world leaders. Each panel also identified key infrastructure concerns.

The mathematics panel found that although the United States was currently the world leader in mathematics, there were stormclouds on the horizon as this leadership was dependent on foreign talent that came to the United States particularly preceding and during World War II and following the collapse of the Former Soviet Union. The difficulty of attracting US talent to mathematics today and the unknown nature of recruiting foreign talent lead the panel to be concerned about the future leadership status of the United States in this field.

In the case of materials science and engineering, the United States was found to be only be among the world leaders in this field. Other countries are actively pursuing this field which is critical economically. A particular concern in materials is that major facilities in Europe and Japan are much newer than those in the United States in some cases, by 20-30 years.

In the last experiment in immunology, the United States was the world leader. However, despite the fact that United States financial investment in this field overshadows that of other countries, the United States was not the leader in all sub fields. Of particular concern was clinical immunology where managed care systems in the United States versus other countries make it more difficult to attract the patients needed for clinical studies.

The general factors influencing US performance in almost all the fields were

COSEPUP then evaluated the quality of panel results independently and via comments from the oversight groups. In addition, the feasibility and utility of benchmarking were assessed during meetings with disciplinary societies and at a full-day workshop attended by representatives of federal agencies, universities, Congress, and the executive branch. Of particular importance were the contributions of reviewers of the reports (as part of the National Academies normal report review process), as they are chosen to represent diverse industrial and academic backgrounds.

MARYE ANNE FOX:

So we have summarized some our panels’ findings, which provide interesting information on the factors that influence leadership in research. These findings of the panels are very important for COSEPUP, because they indicate how the mechanism can work. But they are only one kind of result. As a committee, our overall goal was to find out whether benchmarking works, and whether it is worth pursuing as an assessment tool. For several reasons, we think the answer to both questions is yes.

In general, the committee agreed that the technique of benchmarking was able to provide responses to the questions which we outlined earlier

COSEPUP also found international benchmarking to be rapid and inexpensive compared with procedures that rely entirely on the assembly of a huge volume of quantitative information. In the words of one panel chair, "We were able to get 80% of the value in 20% of the time, for a far lower cost."

Panel members initially questioned whether benchmarking could be reliable, given its partially subjective nature, and whether complex fields of modern research could be assessed to any degree of accuracy without large investments of money and time. However, all three of these experiments were concluded in a year or less for relatively modest investments. Panelists found that it was possible to take a "snapshot" of a field by means of a virtual congress in a matter of weeks.

COSEPUP also found good correlation among the findings produced by various indicators. For example, the qualitative judgments elicited by the virtual congress were similar to the results of quantitative indicators, such as publications cited or papers delivered at international congresses. Also lending credence to the technique was a benchmarking experiment in mathematics conducted by the National Science Foundation that produced results that were similar to COSEPUP's math study despite differences in panel makeup and their mandates by each of the organizations.

Let me conclude with a remark about relevance of benchmarking to GPRA, the Government Performance and Results Act of 1993. GPRA emphasized the need for a method to assess the results of scientific and technological research investments by the federal government. This committee’s GPRA report in 1999 Government Performance and Results Act concluded that the most effective means for evaluating federal research is to provide expert review of the quality, relevance, and leadership of research programs. Some elements of benchmarking may indeed provide useful input to agency strategies to comply with GPRA.

In summary, COSEPUP concluded that this experiment should be regarded as an encouraging first step toward the design of an efficient and reasonably objective evaluation tool. Additional benchmarking exercises could lead to more effective assessment methods, better understanding of the factors that promote research excellence, and better decision-making by those who fund science and technology.

Report: Experiments in International Benchmarking of US Research Fields