The National Academies Board on Testing and Assessment Workshop on Multiple Measures Workshop Agenda November 16, 2007
DISCLAIMER: This is an unedited verbatim transcript of the Workshop on Multiple Measures prepared by CASET Associates and is not an official report of National Academy of Sciences, National Academy of Engineering, Institute of Medicine, or National Research Council (collectively “National Academies”). Opinions and statements included in the transcript are solely those of the individual persons or participants at the conference, and are not necessarily adopted or endorsed or verified as accurate by The National Academies.
Please note that the presentations below may be subject to copyright restrictions of the individual presenters.
OPEN SESSION
8:30 a.m.	Breakfast
9:00 a.m.	Introductions		Transcript
	Michael Hout, Committee Chair, University of California, Berkeley Michael Feuer, Executive Director of the Division of Behavioral and Social Sciences and Education, The National Academies Stuart Elliott, Director of the Board on Testing and Assessment, The National Academies
9:15 a.m.	Department of Education and Congressional Perspectives		Transcript
	Kerri Briggs, Assistant Secretary for Elementary and Secondary Education, U.S. Department of Education Jill Morningstar, Education Policy Advisor, House Education and Labor Committee - Majority Staff Roberto Rodriguez, Education Advisor, Office of Senator Kennedy Lindsay Hunsicker, Senior Education Policy Advisor, Office of Senator Enzi
	• Is ED/Congress considering changes to NCLB that would allow the use of multiple measures for determining adequate yearly progress (AYP)? If so, what changes are being considered?
	• What can the National Academies do to help?
	Discussion leader: Susanna Loeb, Committee Member, Stanford University
10:30 a.m.	Break
10:45 a.m.	Considerations when Selecting and Combining Measures		Transcript
	Robert Linn, University of Colorado	Presentation
	• What does “multiple measures” mean in educational assessment? Measure new things? Measure the same things but measure them more than once? Use criteria like graduation rates that do not depend on tests?
	• What is the intent behind using multiple measures? Do they give schools incentives to improve learning? Do they improve assessment? Do they artificially increase the number of good outcomes?
	• What do multiple measures do? Add more diverse topics? Diversify measures? Give more feedback?
	• What are the most important considerations when selecting and combining measures for accountability purposes?
	Discussion leader: Geno Flores, Committee Member, San Diego City Schools
11:30 a.m.	State Perspectives		Transcript
	Mitchell Chester, Ohio Department of Education Jan Hoegh, Nebraska Department of Education Robert Bernstein, California Department of Education	Presentation Presentation Presentation
	• How are systems of multiple measures being used for accountability in your state?
	• What lessons have you learned about the strengths and weaknesses of your approach?
	• Does your system include incentives for educators and students? Does it also create unintended incentives?
	• Does the public in your state understand how the system of multiple measures works?
	• What would be the pros and cons, from a state perspective, of using multiple measures to determine AYP?
	• Has your state conducted any studies to evaluate how well the multiple measures system is working?
	Discussion leader: Kevin Lang, Committee Member, Boston University
12:30 p.m.	Lunch
1:30 p.m.	Research and Policy Perspectives		Transcript
	Drew Gitomer, Educational Testing Service Kati Haycock, Education Trust Daniel Fuller, Association for Supervision and Curriculum Development William Taylor, Citizens’ Commission on Civil Rights
	• Would determination of AYP be improved with a system of multiple measures used in a compensatory way?
	• Some proposed measures include graduation rates, college enrollment rates, other standardized assessments (AP, International Baccalaurate), and local assessments. What are the pros and cons of each of these and other additional measures in terms of validity, reliability, and feasibility?
	• What types of incentives would multiple measures, used in a compensatory way, create for state leaders, local administrators, teachers and students? For example, what are the implications of having a graduation rate offset low math and reading scores at a high school?
	• How would one know whether a multiple measures accountability system works better than one based primarily on math and reading test scores? What kinds of studies or evaluations should be done?
	Discussant: Brian Stecher, Committee Member, RAND Discussion leader: Chris Edley, Committee Member, University of California, Berkeley
2:45 p.m.	Wrap-up		Transcript
3:15 p.m.	Adjourn workshop