Scientific Research in Education and
the Reauthorization of the Office of Educational Research and Improvement
Lisa Towne, M.P.P.
Study Director, Committee on Scientific Principles in Education Research
National Research Council/National Academy of Sciences
Subcommittee on Education Reform
Committee on Education and the Workforce
United States House of Representatives
February 28, 2002
Good morning Chairman Castle and members of the Subcommittee. My name is Lisa Towne, and I am a Senior Program Officer at the National Research Council (NRC). The NRC is the operating arm of the National Academy of Sciences, chartered by Congress in 1863 to advise the government on matters of science and technology. I am accompanied today by Michael Feuer, director of the Center for Education at the NRC.
The NRC recently released a report titled Scientific Research in Education which described the nature of scientifically based education research and offered recommendations for the future of a federal education research agency. If permissible, I would like to submit a copy of the report for the record. This study was sponsored by the National Educational Research Policy and Priorities Board, the policy arm of the Office of Educational Research and Improvement (OERI), under the leadership of its chair, Kenji Hakuta. Like all NRC studies, the report was authored by an interdisciplinary committee of prominent experts. This committee was chaired by Richard Shavelson of Stanford University and included experts in education research and practice, anthropology, economics, statistics, assessment, psychology, sociology, history, philosophy (of science), demography, chemistry and cell biology.
I’ll start with some brief findings that relate broadly to understanding the nature of scientifically based education research, and then turn to the specifics of the NRC committee’s findings regarding the future of a federal education research agency and the current bill pending before this Subcommittee.
Before I do, I should say a few words about what the report does not say, because the issues before the Subcommittee cover a broader scope than the NRC committee’s work. Most importantly, the NRC committee did not take up the critical issue of research use. Understanding how to promote research-based decisionmaking and practice in schools is critical if efforts to make education a more evidence-based field are to be successful, but these issues were well beyond the scope of the charge to the committee. Indeed, another NRC effort--the Strategic Education Research Program--is taking up some of these very issues.
The Nature of Scientifically Based Research in Education
Much of the controversy about education research relates to its perceived lack of quality. It is important to say that the NRC committee did not evaluate the vast body of existing research literature related to education. Rather, it took up a related set of comparative questions to try and describe it in the ideal: is scientific education research the same as research in social and behavioral science generally or the same as research in the physical sciences? How does it compare in its basic principles to other applied fields, like medical or agricultural research? And based on its answers to these questions, the committee then explored implications for a federal educational research agency. Here, too, however, we did not evaluate the quality of OERI, but focused on the ideal conditions necessary for continued improvement in the quality of scientifically-based education research.
A key finding of this NRC committee is that at a fundamental level, scientific inquiry in education is no different from scientific inquiry in other fields and disciplines. A set of basic principles is common to all scientific endeavors: these principles include concepts like linking empirical data to theoretical models, using appropriate methods, applying rigorous reasoning, striving toward generalization, and the overarching role of the scientific community in unfettered, constructive debate in promoting the accumulation of scientific knowledge. This accumulation is similar across fields as well: it is circuitous and indirect, its path wanders over time, and it is enabled by time, money, and public support.
Does this finding mean that research in economics is the same as research in experimental ecology? Or that research in medicine is the same as research in sociology? No. Researchers in each field develop a specialization within the broad principles that unite all scientific endeavors that takes into account the specific "objects of inquiry" in a given field: heart cells, third graders, nations, black holes, or social organizations, for example.
These differences in what is studied often account for differences in method, a topic that has been at the center of debate within the research community for decades, and within this Subcommittee and other policy milieus more recently. Research methods are simply tools of the research trade. Some research methods are better than others in addressing different kinds of research questions--just like hammers, wrenches, or drills are critical for some, but obviously not all, parts of a carpentry job. It is the experienced carpenter who knows the exact tool needed for the many parts of the job. So too is it the job of the researcher to decide which tools of their trade--this time things like randomized field trials, surveys, and case studies--are appropriate for the variety of tasks that typically arise within a given investigation, whether that study be in medicine, education, or anything else.
Recommendations Regarding the Reauthorization of OERI
What does this all mean for the reauthorization of OERI? Before I address this critical question, I must make a few caveats. As I mentioned earlier, our committee did not evaluate OERI or attempt to judge its myriad past and current programs. Rather, they took their cue from their depiction of what constitutes scientific quality and relied on data from OERI and other social and behavioral science agencies and organizations within the federal government to understand the ways in which they accomplish their research missions. Further, the committee did not distinguish which branches of government should be responsible for implementing the various recommendations. Indeed, some of what we recommend may not be accomplished through legislation of any sort (and some may already be implemented in OERI currently). Nonetheless, it is essential that the authority you are crafting strike the right balance between legislative requirements and flexibility for agency staff, in close collaboration with the field, to implement the mission articulated in the statute successfully. It is in this spirit that I summarize the committee’s findings regarding the future of the agency.
The NRC committee concluded that the precise structure of the agency is not the critical determinant of its success. Rather, healthy research agencies need to develop a scientific culture supported by structures, processes, resources, and norms. To elaborate this idea and to provide concrete advice to policymakers grappling with the reauthorization, the committee developed six "design principles" to guide the agency, and within each recommended specifics mechanisms that could be used to support it:
Staff the agency with people skilled in science, leadership, and management. The director of the agency should have demonstrated outstanding leadership capabilities and be a respected researcher in education. Research staff should hold similar qualifications, as well as be adept at writing grant announcements, engaging with the field to identify research gaps and priorities, and assembling panels of peers to review proposals. Qualified staff is so critical to a healthy agency that we believe without them, little else matters. The agency should be granted maximum flexibility to build a core of permanent staff as well as options for rotating scholars, postdoctoral fellows, interns, and other short-term appointments that can infuse new knowledge and energy into the organization. Only with such staff can the norms of scientific research in education become infused into the agency.
Create structures to guide the research agenda, inform funding decisions, and monitor work. The research agenda must be developed through a collaborative process that engages the range of stakeholders in education. An advisory board of researchers, practitioners, business people and policy makers (perhaps modeled after the National Science Board) could work in collaboration with an agenda-setting committee. To provide additional input to the agenda-setting process, as well as to vet research proposals, peer review is the single best, although certainly not perfect, model. Standing peer-review panels, preferably with rotating terms, can learn from, and communicate to, the field and in turn be especially strong instruments for promoting scientific progress over time. The choice of peers with excellent scientific credentials and an ability to think across areas is the key to making this commonly used mechanism work, and depends critically on an ample talent pool of peers.
Insulate the agency from inappropriate political interference. The research agency must be insulated from political micromanagement, the distortion of research agendas by excessive focus on immediate problems, and the use of the agency as a tool to promote particular policies or positions. Thus, it should have independent authority for publishing, hiring, and disbursal of funds and the head of the agency should serve a fixed term that spans political administrations. At the same time, the agency cannot and should not be separated from politics completely, and its portfolio should include policy research and short-term work that is responsive to current priorities and needs. Given trends in "hybrid" federal organizations that support both education research and service-oriented programs (OERI and the Education and Human Resources Directorate of the National Science Foundation), we suggest that the research function of an agency be organizationally separate from, though intellectually linked to, an educational improvement mission to ensure that the research mission is nurtured.
Develop a focused and balanced portfolio of research that addresses short-, medium-, and long-term issues of importance to policy and practice. Short- and medium-term scientific studies are most responsive to the need for answers to questions of pressing problems of practice and policy. Long-term studies address fundamental questions by focusing on the development and testing of theoretical frameworks. All should be organized in coherent programs of related work and include both new investigations and regular syntheses of the knowledge base. It is only through such sustained focus that research-based knowledge has accumulated, as is the case in early reading, testing and assessment, and educational resources.
Adequately fund the agency. Estimates of the federal investment in education research have shown it to be a few tenths of one percent of the total amount spent on public elementary and secondary education each year--far less than comparable investments for agriculture and medicine. The research budget of the OERI (and its predecessor agency, the National Institute of Education) has fallen drastically since its inception: in 1973, its budget was over $400 million; today, it is approximately $130 million (both in current dollars). As funding plummeted, there has been no commensurate change in the scope of its agenda, and thus there have been few opportunities for long-term research programs. We echo the calls of several previous studies and commissions for a significantly increased research budget if a federal education research agency’s agenda is to cover the breadth of content required of its predecessors. Stagnant funding, an inconsistent commitment, or both, means that scientific research in education is not being taken seriously.
Invest in research infrastructure. The agency should consistently invest in infrastructure-building programs, to foster a scientifically competent, highly qualified community of education researchers and to strengthen its own capacity in turn. Since an agency in many ways is a reflection of the field it supports, such programs should include investment in human capital (e.g., research training and fellowship support). Promoting ethical access to research subjects and data should be an essential task as well. An agency should also do its part to facilitate relationships between practitioners and researchers both for basic access to data as well as, in many field-based research efforts, for long-term partnerships with practitioner communities to improve the research as well as its utilization.
In view of these recommendations from an expert committee of scientists and other experts, I would say that there is much to like in the bill pending before the Subcommittee I base these judgments on HR 4875 as it passed the Subcommittee in summer of 2000. At the time this written testimony was submitted, I had not seen the revised version or the Administration’s proposal. For example, it frees the incumbent agency from restrictive requirements regarding its use of funds. The committee was very clear that decisions about research opportunities must be made based on a solid understanding of the state of knowledge in a particular field and thus must be delegated to agency staff (of the sort described above) to make flexibly in both the short and long term. The bill also provides a focus for the agency to concentrate on its primary mission as a research organization by requiring that service-oriented education programs be administered outside of OERI. The committee believes that the "I" in "OERI"--that is, the improvement function--while an essential role of the federal government, has overwhelmed the agency budget and hampered its ability to develop research capacity both within its own walls and in the field more generally.
The primary problem with this bill is its inclusion of definitions for scientifically valid quantitative and scientifically qualitative methods. To be sure, many of the concepts in those draft definitions are the very same concepts that the NRC committee emphasized in its report: empirical data, replication, and peer review, for example. And the inclusion of both quantitative and qualitative methods is very positive, since both, when properly applied and implemented, can be very powerful research tools. The problem is with their use as a federal mandate. The NRC report makes clear that the objectivity and progress of scientific understanding in any field--not just education research--derives not from a given methodology or a given person. Rather, it comes from the community of researchers. Improving education research, then, requires improvements in the field itself.
A federal education research agency should play a major role in spurring those improvements. It must promote high quality scientific work through mechanisms like peer review, investing part of its annual appropriation in training and mentoring the next generation of researchers, and developing high standards of quality in close collaboration with the field.
Mr. Chairman, thank you for the invitation to appear before the Subcommittee and discuss these very important issues, and for your leadership in promoting high quality education research. I would be happy to answer any questions you or other Members may have.