Standards of Evidence

Strategic Planning Initiative

Purpose

The National Research Council’s Division of Behavioral and Social Sciences and Education (DBASSE), with funding from the Hewlett Foundation, the National Science Foundation and the Spencer Foundation, has established a planning committee to oversee the conceptual development of a broad, long-term initiative related to the quality of evidence. The planning committee will help map out a range of topics and activities to be pursued over a multi-year period to improve the quality of research in the behavioral and social sciences and education and strengthen the ties between behavioral and social science, public policy, and practice.

Background

This is an age of evidence-based policy. For example, the landmark No Child Left Behind Act of 2002 includes 111 references to the use of programs founded on or proven by "scientifically based research." Similar pressures weigh on criminal justice, social welfare, child development, aging, public health, national security, and workforce development policies, reigniting vigorous debates over methodology, funding, and "use-inspired" research, a phrase brought to prominence by Donald Stokes influential book, Pasteur's Quadrant (1997).

The renewed emphasis on the nature and quality of evidence marks a potentially healthy stage in the continuing evolution of research aimed at solving real-world problems. Although policy research has always been about marshaling evidence for better decisions, the current wave of interest has caused something of a tempest, for at least the following combination of reasons:

The call for evidence is often laced with a nasty rebuke of social science research for its alleged lapse in standards, disregard for the realities of implementation, confusing (if not incompatible) conclusions and recommendations, and logy pace relative to the fast-moving needs of research users.

Discussions about quality of social science research revive old debates over issues such as quantitative vs. qualitative studies, experimental vs. quasi-experimental designs, correlation and causation, internal and external validity, replicability, heuristic and optimization models, modernism and postmodernism, and the nature of rational deliberation-not just in journals of philosophy of science but also in Congressional testimony and even in major legislation.

The subtext under the call for "evidence" is "random assignment," considered by some scientists and policy makers to be the gold standard in establishing causal relationships. But even the strongest advocates for more randomized trial research do not believe it is the only legitimate research method for all types of questions and purposes of research.

Arguments over methodology, especially when they explicitly or implicitly are linked to governmental policy on allocation of research funds, can eclipse equally important considerations including relative costs, benefits, and ethical considerations.

Increased emphasis on research as the basis for policy and practice inevitably raises questions of standards: Is the research good enough to support high stakes decisions? Does pursuit of the "gold standard" of evidence create a risk that even the best available evidence will not be considered sufficient as a basis for policy decisions? How can rigorous findings be made available to decision makers when they need to act? Under what conditions will results generalize to larger or different populations in different settings at different times?

Each of these core issues spawns a list of derivative questions, relates to long-standing issues that affect all of science, and invites review of fundamental principles of inquiry, the culture and politics of science, and the role of rational deliberation in policy and practice. They are not questions to be "answered" in a single report, but require, rather, sustained attention, multidisciplinary deliberation, consensus building, and experimentation with innovations in the design, organization, conduct, and communication of use-oriented research. The planning committee will help conceptualize a broad, multi-faceted initiative to foster sustained attention by scientists, policy makers, and funders to core issues that determine the quality of research and its utility in policy and practice.

Strategic Planning

Through a series of three meetings, commissioned papers, and a pilot effort (see separate pilot description), the planning committee will identify critical issues affecting the quality and utility of research in the behavioral and social sciences and education. The planning committee also will consider possible approaches for DBASSE to further explore these issues. The planning committee’s deliberations will guide a multi-year DBASSE initiative in this area. Relevant issues include, but are not limited to:

How do various scientific disciplines define and implement evidentiary standards? How can scholarly traditions in the physical and life sciences, law, medicine, history, philosophy, business and other fields shed light on opportunities and constraints faced by the behavioral and social sciences?

How is the pursuit of high evidentiary standards enhanced or constrained by multidisciplinary and interdisciplinary research teams? By considerations of utility in decision-making and practice?

How can research methods be matched to different policy questions and adapt to decision makers' needs? What is the appropriate calculus of benefits and costs when weighing alternative research methods?

How do such institutional mechanisms as peer review, data-sharing requirements, and publication policies, as well as academic incentive structures such as tenure and promotion policies affect the objectivity, rigor, and utility of research and the accumulation of usable knowledge over time?

What mechanisms might foster improved communications between research users and research producers?

How can new technology capabilities for data acquisition, storage, and communication be most fruitfully applied toward the improvement of research and its relevance to complex policy decisions?

What can organization theory, sociology of science, economics of innovation, and other strands in the behavioral and social sciences teach us about how to improve scientific research for use in public policy?

Staff

Mary Ellen O’Connell, Senior Program Officer, 202-334-2607, moconnell@nas.edu

Lisa Towne, Senior Program Officer (Pilot), 202-334-2568, ltowne@nas.edu

Tina Winters, Research Associate, 202-334-3084, twinters@nas.edu

Liz Townsend, Senior Project Assistant, 202-334-2034, etownsend@nas.edu