PILOT PROJECT IN BRIEF:

STRUCTURED ABSTRACTS IN EDUCATION RESEARCH ARTICLES

Purpose

The NRC’s Division of Behavioral and Social Sciences and Education (DBASSE) has embarked on a strategic planning effort to map out a multi-year initiative of activities aimed at understanding and improving the quality and utility of social and behavioral sciences for improving policy and practice (see overview document for a description of this strategic planning effort). A central component of this planning phase is a pilot project, which focuses on one aspect of the social science research infrastructure (abstracts of social science articles in peer-reviewed journals) and on one field in the social and behavioral sciences and education (education research). Specifically, the pilot project is designed to meet two purposes:

To inform the DBASSE strategic planning effort. The pilot project is designed to address some of the broad issues embedded in considering the quality and utility of social science research by focusing attention and analysis on the specific topic of structured or standardized abstracts in published social and behavioral science and education research. The premise is that focusing on a small piece of the overall picture during the planning phase will generate initial ideas, lessons, and implications that will inform the development of the long-term initiative. Since the pilot is focused on structured abstracts, the lessons learned from the project are likely to be related to publication policies and knowledge management systems. However, since publication is so central to both the quality and utility of research, broader implications may also emerge. For example, the consideration of structured abstracts is likely to touch on such core issues as: areas of consensus and contention within and across subfields of social science (e.g., with respect to quality standards, or the role of research in solving social problems); philosophical considerations that influence standardization efforts and other kinds of change in the social sciences; opportunities and barriers (both social and technological) for promoting dialogue and improvement among social scientists and the users of social science research; and the role of professional cultures and incentive systems in both research and practice contexts that are relevant to considering change.

To engage stakeholders in discussion of the use of structured abstracts in education research journals. While informing the strategic planning effort, the pilot will also serve as a catalyst for dialogue and action among leaders in education research regarding the development of policies that require structured abstracts (a recent NRC report by the Committee on Research in Education recommended that journals that publish education research adopt structured abstracts: see http://books.nap.edu/catalog/11112.html).

Parameters

Abstracts—short summaries of full-length journal articles—serve a number of purposes and are used by a number of audiences. For producers of research (individual investigators, including those conducting systematic reviews), abstracts can play an important role in facilitating the development of a knowledge base over time by enabling investigators to locate and to learn from the extant literature relevant to their area of inquiry. For users of research (policymakers, practitioners), and translators, or brokers, of research (organizations or individuals who focus on making connections between producers and users), abstracts can similarly facilitate the location and use of social science research for informing and improving decision-making and action.

The pilot project is focused on structured abstracts in peer-reviewed journals that publish education research, including journals published by the American Educational Research Association as well as discipline-specific journals that focus on education problems and issues. Specifically, it will examine the different purposes abstracts serve, their role in reporting standards and publication policies, the kinds of elements that might be included in a structured abstract given differences in the nature of inquiry and standards of evidence across disciplines and fields, as well as differences in the needs of producers, users, and translators of education research.

Plan

The primary event of the pilot project is a public workshop, scheduled for January 7, 2005 in Washington, DC. This workshop will bring together research producers, users, brokers, and knowledge management professionals to explore the broad issues of quality and utility in the social and behavioral sciences by considering the idea of a structured abstract for education research articles. Editors and others involved in the publication process that represent a range of subfields (e.g., educational anthropology, educational psychology, sociology of education, evaluation and policy analysis, and mathematics education) and journal types (including journals that publish original research and journals that publish reviews of literature) will analyze the use, purpose, and elements of structured abstracts as they pertain to their subfields and journals. A range of users and brokers will explore similar issues from their varied perspectives by describing and reflecting on the ways in which they access and interface with the education research literature. And publishers, information specialists, and database producers will discuss how structured abstracts have or could be useful in their work and their role in the broader knowledge management systems in and across the social sciences.

In light of workshop and related discussions, the pilot team (see below for membership) will develop a paper that identifies key lessons and implications from the pilot project for the development and implementation of the multi-year initiative. This paper will be presented to the steering committee at its final meeting in April 2005.

Leadership

The pilot project is led by a team of five experts. To ensure close coordination with the strategic planning effort, the team includes two members of the steering committee. To bring specialized expertise in the area of publication policies and to engage a range of disciplinary perspectives within the field of education research to the pilot project, the team also includes three consultants.

The pilot team members are:

Robert Boruch. Dr. Boruch is a member of the steering committee and an expert in quantitative evaluation research, with a focus on assessing the impact of educational interventions.

Norman Bradburn. Dr. Bradburn is chair of the steering committee and an expert in survey methodology.

Catherine Emihovich. Dr. Emihovich is an expert in qualitative methods and the role of culture in education, and past editor of Anthropology in Education Quarterly.

Adam Gamoran. Dr. Gamoran is an expert in the sociology of education who has served on the editorial boards of Educational Researcher, Sociology of Education, and Educational Evaluation and Policy Analysis.

Hannah Rothstein. Dr. Rothstein is an expert in industrial psychology who leads the Information Retrieval component of the Campbell Collaboration’s Methods Coordinating Group and serves on the editorial board of the International Journal of Selection and Assessment.

The staff leader for the pilot project is Lisa Towne, Senior Program Officer in the NRC Center for Education. Questions and comments about the pilot project should be directed to her at: ltowne@nas.edu.