The National Academies, Board on Testing and Assessment and The National Academy of Education Workshop of Assessment of 21st Century Skills Agenda January 12-13, 2011 Beckman Center Irvine, California
Wednesday, January 12
9:30	Opening Remarks Welcome Stuart Elliott (Director, Board on Testing and Assessment)
VIDEO
9:40-12:15	Session 1: Background Information
	Moderators: Joan Herman and Pat Kyllonen (ETS and Workshop Steering Committee)
	(9:40- 10:00) Why Are 21st Century Skills Important?
VIDEO	Richard Murnane, Harvard University					Presentation
	This presentation will address the following issues: • What is unique in the 21st century that makes these skills especially valuable in the labor market and/or in other life domains (learning, family life, civic engagement)? • How does the growing use of computers and technology affect the labor market and the demand for 21st century skills? • What does more recent research suggest about the skills needed to be successful in the 21st century?
	(10:00-10:15) How Will You Know if Your Students are 21st Century Ready?
VIDEO	Deborah Boisvert, Boston Area Advanced Technical Education					Presentation
	The presenter will respond to the opening presentation, reflecting her work with employers to define, teach, and assess 21st century skills of computer technicians.					Handout 1
						Handout 2
						Handout 3
	(10:15-10:45) The Teaching and Learning of 21st Century Skills
VIDEO	Eric Anderman, Ohio State University		Paper			Presentation
	This presentation will address the following issues: • What is known about the extent to which the three skill clusters and/or the skills within them can be taught and learned? • To what extent are learning, teaching, and assessment of the three skill clusters domain-specific or domain-general?
VIDEO	(10:45-11:00) Discussion Moderators will lead a question and answer session with the presenters and audience members.
11:00-11:15	Break
	(11:15-11:35) Approaches to Developing Assessments of 21st Century Skills
VIDEO	Deirdre Knapp, HumRRO					Presentation
	This presentation will address the following issues: • What are the different approaches to assessment of these skills and what steps are involved in carrying out these approaches? • What processes are used for identifying the skills to be measured, operationalizing the skills through the test blueprint, and creating assessment tasks and scoring procedures?  • How should the intended uses of the assessment results guide the test development process? • What steps should be taken to ensure that the assessments are reliable and valid?
	(11:35-11:55) Unique Challenges and Opportunities in Assessment of 21st Century Skills
VIDEO	Speaker: Steven Wise, Northwest Evaluation Association					Presentation
	This presentation will address the following issues: • What are the unique challenges and opportunities for defining and measuring these constructs, when compared to more traditional academic skills and knowledge? • How might the results of these assessments be used? Should they be used for high stakes purposes? • What issues may arise in relation to validity, reliability, and fairness of assessments of these skills?
VIDEO	(11:55-12:15) Questions and Discussion
12:15-1:15	Lunch in Beckman Center Dining Room Continued discussion of ideas presented during the morning sessions
1:15-3:45	Session 2: Assessing Cognitive Skills
	Moderators: Greg Duncan (University of California, Irvine; Workshop Steering Committee) and Paul Sackett (University of Minnesota; Workshop Steering Committee)
	(1:15-1:45) Defining and measuring cognitive skills
VIDEO	Nathan Kuncel, University of Minnesota		Presentation					Paper
	presentation will address the following issues: What are 21st century cognitive skills? To what extent do they differ from each other and from general cognitive ability? What are the conceptual differences that are proposed to exist between these constructs? What are the existing measures of these constructs, and to what extent do these existing measures match their conceptual specifications? What are the relationships between the existing measures of these constructs?
	(1:45-2:00) Questions and Discussion Moderators will lead a question and answer session with the presenters and audience members.
	(2:00-3:15) Panel Discussion: Examples of Assessments of Cognitive Skills
	Panelists will discuss examples of assessments of cognitive skills. For each example they will address: • What skill or skills are measured? Why are these skills important? • What is the purpose of the assessment? • What strategies were used to develop the assessment and why were these selected? • What assessment methods are used and why were these selected? • How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? • What data are available on the cost and practical feasibility of the assessment? (2:00-2:20) Interactive Problem Solving for PISA 2012
VIDEO	Joachim Funke, University of Heidelberg (by video-conference)					Presentation
	(2:20-2:40) Operation ARIES!: Learning critical thinking about science with intelligent conversational agents in a game environment
VIDEO	Art Graesser (University of Memphis) and Heather Butler (Claremont McKenna College)					Presentation
	(2:40-3:00) Intrusive and Unobtrusive Assessment of Entrepreneurial and Technical Skills through Simulation and Gaming
VIDEO	John Behrens, Cisco Systems						Presentation
	(3:00-3:20) Assessment of Critical Thinking and Problem Solving on the Multistate Bar Exam
VIDEO	Susan Case, National Conference of Bar Examiners					Presentation
3:20-3:30	Break
VIDEO	(3:30-4:00) Moderated Discussion
	Moderators will explore the following issues with panelists and audience members: • What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? • Do common themes or approaches emerge from the examples? How might the non-education examples generalize to education? • How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two state consortia)?  What functions can/should the assessments serve?   How might the results be used? • What equity and accessibility challenges do these assessments raise? • What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?
4:00-5:00	Synthesis of Key Ideas Moderator: Joan Herman (4:20-4:40) Discussion: Richard Murnane, Harvard
VIDEO
	Discussants will reflect on the day’s discussions and offer their synthesis of the ideas presented. Audience members will be invited to ask questions and share their ideas as well.
5:00 5:30	Conclude Formal Agenda for Day 1 Working Group Dinner at Beckman Center (in Atrium) Plan for the second day of the workshop
Thursday, January 13
9:00- 11:45	Session 3: Assessing Interpersonal Skills
	Moderators: Deirdre Knapp and Juan Sanchez (Florida International University, Workshop Steering Committee)
	(9:00-9:30) Defining and Measuring Interpersonal Skills
VIDEO	Stephen M. Fiore, University of Central Florida			Presentation				Paper
	This presentation will address the following issues: • What are 21st century interpersonal skills and why are they important? • How are these skills typically assessed? What are the challenges in assessing them? • What are some of the assessments available to evaluate these skills?
	(9:30-9:40) Questions and Discussion Panelist will discuss examples of assessments of interpersonal skills. For each example, they will address the following issues: • What skill or skills are measured? Why are these skills important? • What is the purpose of the assessment? • What strategies were used to develop the assessment and why? • What assessment methods are used and why were these selected? • How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? • What data are available on the cost and practical feasibility of the assessment?
	(9:40-10:00) Using situational Judgment Tests for Medical School Admissions
VIDEO	Filip Lievens, Ghent University, Belgium (by video-conference)					Presentation
	(10:00-10:20) On-Line Portfolio Assessments of the 4 C’s
VIDEO	Bob Lenz, Envision Schools					Presentation
	(10:20-10:40) 21st Century Skills in STEM Workforce Training Assessments
VIDEO	Louise Yarnall, SRI						Presentation
	(10:40-11:00) Assessment Centers 2011: Fifty Years of Best Practice and Today’s Innovations
VIDEO	Lynn Gracin Collins, SH&A/Fenestra						Presentation
11:00-11:10	Break
VIDEO	(11:10-11:45) Moderated Discussion
	Moderators will explore the following issues with panelists and audience members: • What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? • Do common themes or approaches emerge from the examples? How might the non-education examples generalize to education? • How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two state consortia)?  What functions can/should the assessments serve?   How might the results be used? • What equity and accessibility challenges do these assessments raise? • What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?
11:45-12:45	Working Lunch in Beckman Center Dining Room Continued discussion of ideas presented during the morning sessions
12:45-3:30	Session 4: Assessing Intra-Personal Skills
	Moderators: Pat Kyllonen and Steve Wise
	(12:45-1:15) Assessment of Self-Regulation and Related Constructs: Prospects and Challenges
VIDEO	Rick Hoyle, Duke University	Paper			Presentation
	The presentation will address the following issues:
	• What are 21st century intra-personal skills and why are they important? • How are these skills typically assessed? What are the challenges in assessing them? • What are some of the assessments available to evaluate these skills?
	(1:15-1:30) Discussion
	(1:30-3:30) Panel Discussion: Examples of Assessments of Intra-Personal Skills Panelists will discuss examples of assessments of intra-personal skills. For each example, they will address the following issues: • What skill or skills are measured? What are these skills important? • What is the purpose of the assessment? • What strategies were used to develop the assessment and why? • What assessment methods are used and why were these selected? • How is the assessment scored? What data are available on the technical quality of the assessment, including validity, reliability, fairness, and comparability across administrations? • What data are available on the cost and practical feasibility of the assessment?
VIDEO	(1:30-1:50) Integrity Testing for Employee Selection Paul Sackett, U of Minnesota; Workshop Steering Committee					Presentation
	(1:50-2:10) Targeting context-specific self-regulated learning (SRL) processes: An overview and illustration of SRL microanalysis
VIDEO	Tim Cleary, U of Wisconsin – Milwaukee						Presentation
	(2:10-2:30) Assessing behavioral problems that predict poor educational and life outcomes
VIDEO	Candice Odgers, UC Irvine					Presentation
	(2:30-2:50) Out of the maze? In search of skills for emotional intelligence
VIDEO	Gerald Matthews, U of Cincinnati					Presentation
2:50-3:00	Break
VIDEO	(3:00-3:30) Moderated Discussion
	Moderators will explore the following issues with panelists and audience members • What are the implications of the presentations (and examples) for the design of 21st century assessments for K-12 and higher education? • Do common themes or approaches emerge from the examples? How might the non-education examples generalize to education? • How might 21st century assessments be incorporated into current research efforts, such as the development of assessment systems by the two state consortia)?  What functions can/should the assessments serve?   How might the results be used? • What equity and accessibility challenges do these assessments raise? • What barriers might slow development and/or use of assessments of 21st century skills? How might they be overcome?
3:30-4:00	Session 5: Reflection and Synthesis
	Moderated discussion led by workshop steering committee
4:00 VIDEO	Closing Remarks, Adjourn Joan Herman