Applying Multiple Social Science Research Methods to Educational Problems A National Forum Center for Education National Research Council of the National Academies 500 Fifth Street, NW Rooms 100 and 110 Washington, DC December 14, 2004 Agenda
The Center for Education of the National Research Council, with support from the American Educational Research Association, the American Psychological Association, the Decade of Behavior, and the National Science Foundation, hosted this one-day forum to explore the use of multiple social science research methods in exploring educational problems. The application, fit, and articulation of different research methodologies to tackle major issues relevant to educational policy and practice is an area that stands to benefit from sustained discussion across disciplines and contexts. The objective of this Forum was to lay a foundation for such inquiry by focusing on the applicability of multiple methodologies to educational problems. The Forum featured conceptual and pragmatic discussion on the value and limitations of different methodologies and how they might be productively combined, along with illustrations of the use of multiple methods in small- and large-scale studies of important educational issues and problems. Providing both intellectual underpinnings and “real-world” examples of multiple methods in education research, the Forum aimed to bring researchers with diverse methodological expertise toward greater mutual understanding, and further the more robust application of multiple methods in future education research.
NOTE: This is an unedited verbatim transcript of the National Forum on Applying Multiple Social Science Research Methods to Educational Problems prepared by CASET Associates and is not an official report of National Academy of Sciences, National Academy of Engineering, Institute of Medicine, or National Research Council (collectively “National Academies”). Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies. Please note that the presentations below may be subject to copyright restrictions of the individual presenters.
8:30 a.m.	Welcome and Overview of Day
	Martin Orland, National Research Council		Presentation		Transcript	Audio
8:45 a.m.	Background and Policy Context
	Rena Subotnik, American Psychological Association				Transcript	Audio
9:00 a.m.	Setting the Groundwork: A Common Vocabulary for Researchers and Decision Makers
	What do we mean by multiple methods approaches in education research? What are some bellwether examples? What types of important questions for educational policy are particularly good candidates for multiple methods investigations?
	Moderator: Rena Subotnik				Transcript	Audio
	Stephen Raudenbush, University of Michigan		Paper Presentation		Transcript	Audio
	Discussant: Susan Bodilly, RAND				Transcript	Audio
	Q&A session				Transcript	Audio
10:00 a.m.	Break
10:15 a.m.	Social Science Perspectives on Multiple Methods Approaches
	How do multiple methods approaches in the social sciences add to understanding social, behavioral, and institutional processes? When are multiple methods particularly ripe for application? How have multiple methods been effectively used in domains outside of education where the results of research also have important societal and policy significance?
	Moderator: Felice Levine, American Educational Research Association				Transcript	Audio
	Thomas Cook, Northwestern University				Transcript	Audio
	Richard Murnane, Harvard University				Transcript	Audio
	Q&A Session				Transcript	Audio
11:15 a.m.	Examples
	Three concurrent mini-panel sessions featured discussion of education research investigations that have employed multiple methods. Issues addressed include: Rationale for multiple methods approach Basic design for the approach (i.e., sequencing, specific questions addressed by the different methods, data collection and analysis plans etc.) Major challenges and barriers encountered and how they were overcome How findings from different methods were synthesized in supporting inferences and developing findings How findings have affected, or have the unique potential to address, critical policy questions
	Example 1: Evaluation of the Second Step program Note: Due to recording equipment failure, transcript and audio files are not available from this session.
	Moderator: Gerald Sroufe, American Educational Research Association
	Presenter: Karin Frey, University of Washington		Presentation
	Discussant: Cynthia Hudley, University of California, Santa Barbara		Presentation
	Example 2: Evaluation of Core-Plus Mathematics Project
	Moderator: Tina Winters, National Research Council
	Presenter: Mary Ann Huntley, University of Delaware		Presentation		Transcript	Audio
	Discussant: Jere Confrey, Washington University		Presentation		Transcript	Audio
	Q&A Session				Transcript	Audio
	Example 3: Evaluation of New Hope program
	Moderator: James Greeno, University of Pittsburgh
	Presenter: Greg Duncan, Northwestern University		Presentation		Transcript	Audio
	Discussant: George Bohrnstedt, American Institutes for Research				Transcript	Audio
	Q&A Session				Transcript	Audio
12:15 p.m.	Lunch and continued discussion
1:15 p.m.	Future Prospects: Examples of Significant Policy Questions in Education that Might Be Addressed Through Multiple Methods Approaches
	This session featured discussion of a high-priority policy problem domain and what combination of methodological approaches might be usefully employed in a research program around the issue.
	Moderator: Martin Orland				Transcript	Audio
	Pascal Forgione, Superintendent, Austin Public Schools		Presentation Austin Blueprint		Transcript	Audio
	Kathryn Borman, University of South Florida		Presentation		Transcript	Audio
	Brian Rowan, University of Michigan				Transcript	Audio
	Q&A session				Transcript	Audio
2:30 p.m.	Break
2:45 p.m.	Training Researchers to Implement Multiple Methods Approaches: Challenges and Opportunities
	What kind of training is needed to adequately prepare researchers to participate in research employing multiple methods? How can the collaborations necessary for the success of this approach be encouraged? What are the major barriers to training education researchers to understand and employ multiple methods approaches, and what are the incentives that can foster them?
	Moderated Discussion:				Transcript	Audio
	Moderator: Lisa Towne, National Research Council
	Penelope Peterson, Dean, School of Education and Social Policy, Northwestern University		Presentation
	Larry Hedges, Department of Sociology, University of Chicago
	Q&A session				Transcript	Audio
3:45 p.m.	Funders Perspectives
	Supporters of educational research (both from the government and foundation sectors) discussed their reactions to the day and the future promise and challenges involved in supporting multiple methods investigations.
	Moderated Discussion:				Transcript	Audio
	Moderator: Merry Bullock, American Psychological Association
	Daniel Berch, National Institutes of Health
	Daniel Fallon, The Carnegie Corporation
	Bruce Fuchs, National Institutes of Health
	Barbara Olds, National Science Foundation
	Q&A session				Transcript	Audio
4:45 p.m.	Summary, Conclusions, Next Steps
	Janice Earle, National Science Foundation				Transcript	Audio
5:00 p.m.	Adjourn