GPRA Opening Statement

Implementing the Government Performance and Results Act
For Research: A Status Report

National Academy of Sciences, National Academy of Engineering, Institute of Medicine
Committee on Science, Engineering, and Public Policy

Public Briefing
May 8, 2001

Opening Statement
by

Enriqueta A. Bond
President, Burroughs Wellcome Fund, Research Triangle Park, N.C.

and

Alan Schriesheim
Director Emeritus, Argonne National Laboratory, Argonne, Ill.

Co-chairs
Panel on Research and the Government Performance and Results Act

-------------------------------

Dr. Bond

Good afternoon. On behalf of the entire panel, Dr. Schriesheim and I are pleased to be here today to present a summary of our report that examines the implementation of the Government Performance and Results Act (GPRA) at the five largest research agencies — the Department of Energy, the Department of Defense, the National Aeronautics and Space Administration, the National Science Foundation, and the National Institutes of Health.

We gathered information through focus groups looking at the implementation of GPRA in each of these agencies, a workshop to address cross-cutting issues, and numerous conferences between agency staff and the panels.

The picture that emerged from our activities was quite complex. We found that the agencies' challenges in responding to GPRA for their research programs were complicated by the multiplicity of goals and procedures. Each agency was trying to comply with GPRA in its own way, with its own unique blend of customs. Furthermore, the agencies had completed only one cycle of GPRA performance reports and were still developing their reporting formats. The agencies were struggling to work out their procedures while the oversight bodies were struggling to understand those same procedures.

So, this report does not attempt to characterize or prescribe a single model for all agencies because that would be inappropriate. Instead, it offers a cross section or "snapshot" of agency responses to GPRA based on the agencies' own descriptions. At this point, I'd like to ask Dr. Schriesheim to discuss the conclusions of our report.

Dr. Schriesheim

Thank you. Let's now turn to 10 specific questions that the panel dealt with, as well as our conclusions.

First of all, we wanted to know whether the agencies had made a good-faith effort to respond to GPRA. We found that all five agencies had indeed worked hard at what is a very difficult job. Some agencies said that GPRA compliance had added substantial staff time and resources to their planning and evaluation activities. Others said that with some cost and effort, they have been able to integrate GPRA into their traditional planning processes.

Second, are agencies using GPRA to improve their operations? We found that some agencies did report benefits from GPRA in strengthening program management and enhancing communication about their programs to users and the general public.

The third area of inquiry concerns the evaluation criteria that the Committee on Science, Engineering, and Public Policy (COSEPUP) proposed in its first report: quality, relevance, and leadership; and the methodologies of expert review and international benchmarking, and the degree to which agencies had used this methodology and criteria. And, if so, the extent to which they were found to be useful.

We found that the agencies were using the general expert review methodology and the criteria of quality and relevance, but their actual application varied a great deal. Taking a closer look at the criteria, we found that all of the agencies already had well-established mechanisms for measuring quality through peer review. We also found that they use the second criterion, relevance, but don't always make that clear in their performance reports. The criterion of international leadership, which rests on COSEPUP's benchmarking methodology, generally is not used by federal agencies at this time. Still, several are interested in trying it and some have used other measures to assess leadership.

The fourth big question is the degree to which Congress, the Government Accounting Office, the Office of Management and Budget, and others interested in the implementation of GPRA really understand the methodology and evaluation criteria each agency is using. We found that oversight bodies expressed a desire for better understanding of the methodology and results of expert review evaluations — particularly the validation and verification of agency evaluation results.

Another question we dealt with was how agencies organize their reporting process. Some aggregate their research programs at a very high level, evaluating their entire research effort as though it were a single activity. Oversight bodies said that, in such cases, they couldn't determine how individual programs were evaluated or how research priorities were set. This made it difficult for them to clearly see and understand the methods and programs that are the focus of the analyses.

The sixth question concerned the degree to which the agencies addressed the issue of human resources. Our panel felt very strongly that human-resource development should be an objective of every agency, and it should be cited explicitly in their plans and reports. This issue is important because it affirms the nation's tradition of training graduate students who work in research programs. When there is this direct link between training and practice, it is apparent how reductions in research funding can jeopardize the education of young scientists and engineers. On the whole, the issue of human resources did not receive sufficient emphasis or visibility in GPRA plans and reports.

The next question is whether the agencies actually know what was expected by oversight bodies in terms of format, content, and procedures to be used in GPRA compliance. Upon examining this seventh question, we found that agencies often receive conflicting messages. For example, one agency tried to tie its GPRA reports more closely to its annual budget, a step that is required by the act. But a congressional committee told the same agency to use another format. And we later learned that a different agency was given the opposite instruction.

Eighth was the question of timing. Did the agencies have enough time to make changes based on feedback from their previous reports? We found that they did not. They had to start working on performance plans before their previous reports were complete, so they didn't have an opportunity to learn from recent achievements and failures. This seemed to limit the potential benefit of GPRA.

The next issue we examined relates to several of our previous conclusions. We wanted to know whether the agencies and the oversight bodies were sufficiently communicating with each other. Given our other findings, we concluded that they were not. We found that communication was not sufficiently regular, extensive, or collaborative.

And finally, how were GPRA reports being used by oversight groups? In other words, are the results of the "results act" being used? We found that agencies have not yet seen the use of their reports in the congressional decisionmaking that determines the size and priorities of their research program budgets.

With that, let me ask Dr. Bond to conclude with our list of recommendations.

Dr. Bond

The panel made quite a few specific recommendations that fall under four general challenges.

Our first challenge was to answer the question of whether agencies can respond in good faith to GPRA. And if so, how? Our finding reiterates the conclusion of the previous GPRA report. That is, that federally supported programs of basic and applied research can and should be evaluated regularly through expert review. This should be done by using the performance indicators of quality, relevance, and, where appropriate, leadership.

Our second major challenge was to ask ourselves how the agencies could continue to improve their methods of GPRA compliance. We recommend four key points of guidance.

First, agencies should work toward the goal of greater transparency. Here, I'm referring to the ability to see how and why an agency decides to emphasize or de-emphasize particular programs or areas of research. Oversight bodies or agents of the public are not expected to review the thousands of subentities that perform research within agencies. But they should have access to documents that help them answer specific questions. This is especially important for agencies that aggregate their research at a high level.

Secondly, agencies should work toward setting more-realistic reporting schedules. Although GPRA requires annual reporting on all programs, basic research often does not produce useful results in a single year and must be monitored over several years before outcomes become apparent. Agencies should experiment with alternative reporting formats, an option that GPRA allows. Such formats provide regular reporting cycles of three or more years.

Additionally, agencies should state how they verify and validate their review criteria. Outside observers can't always clearly see how expert review works — for example, how agencies select expert reviewers and choose to aggregate research programs.

And fourthly, development of human resources should be an explicit part of performance plans and reports. It must be explicit not only because it affirms the value of educating young scientists and engineers in the context of research, but also because it demonstrates how reductions in research funding could reduce the ranks of future scholars.

Another one of our overall challenges was to answer the following question: What can be done to help agencies that still are struggling to find the best ways to comply with GPRA, especially when it comes to linking their GPRA plans and reports with their budgets? Many steps are needed to resolve this matter.

Whenever possible, agencies should use procedures already in place without creating new steps. GPRA should not add to the workload of agencies, and oversight bodies should help agencies to ensure that this does not happen. At the same time, effective linkage of GPRA plans and reports with budgets may help agencies better explain their needs to Congress and justify funding levels under tight budgets.

And finally, the fourth major challenge was to ask what can be done about misunderstandings between agencies and oversight bodies, and the occasional contradictions in what is expected? Here, I think the recommendation is straightforward, if not easy. Our report urges agencies and oversight bodies to work together as needed to facilitate agencies' integration of GPRA requirements with their internal planning, budgeting, and reporting processes. Agencies should strive for effective communication with oversight bodies on the implementation of GPRA. For their part, oversight bodies should clarify their expectations and meet more often to coordinate messages to agencies.

A principal purpose of GPRA is to improve the way that agencies communicate their results to oversight groups, users of research, and the general public. More effective communication would enhance GPRA's value to all constituents.

As indicated in COSEPUP's first report on this topic, GPRA is useful to the degree that it promotes the effective use of research resources. However, the act will not meet its goals unless the Senate and House Operations committees, working with the Office of Management and Budget, respond to agencies' concerns through open discussion. And unless the agencies' responses to GPRA help Congress set priorities and budget accordingly, the value of the act may not warrant the time and effort that it requires of the federal government.

In conclusion, much has been learned about the planning, evaluation, and management of federally sponsored research in the last several years. And agencies already have gained some benefits from their own discussions about accountability. But if federal agencies and oversight bodies would work together more, agencies might be able to implement the letter and spirit of GPRA in ways that would lead to greater efficiency, lower cost, and more effective research programs.

Thank you very much. Dr. Schriesheim and I will now take your questions.

# # #