Assessing Informal Learning in Data Literacy

Introduction

TERC Inc. developed Kids’ Survey Network (KSN), an apprenticeship network. The goal was to prepare the nation’s future adults to understand and use survey data that are pertinent to everyday concerns and public opinions of government, business, science, health, and other domains affecting the quality of our everyday lives. Working in teams, KSN participants, ages 11-14, developed their research, survey question writing, and data analysis capabilities through playing games, taking surveys, watching videos, providing feedback to other KSN teams, engaging in challenging online learning activities, and completing their own survey projects on topics of their choice. The project’s research study examined the apprenticeship network as a model for delivering challenging activities linked to STEM learning in youth informal learning settings.

The project was funded by the National Science Foundation (NSF DRL 0741601). SRI International evaluated the project. The evaluation team was led by Dr. Daniel R. Zalles, Senior Learning Scientist in SRI’s Center for Technology and Learning. This paper describes one component of the evaluation, the assessment of cognitive impacts through collaborative youth group performance on pre-post short answer constructed response tasks that were referred to as “brainteasers”. Following up on recommendations by the KSN advisory panel, these “brainteaser” tasks were designed to allow the KSN evaluators to identify gains that groups of youth achieved in KSN activities at pilot implementation sites. The pilot sites were out-of-school time (OST) programs in Oakland and East Palo Alto, California. The brainteaser tasks aligned to the major learning objectives that were addressed in KSN’s components (i.e., data games, youth survey projects, and videos of youth talking about doing surveys). SRI designed four sets of brainteaser questions in a progression of increased cognitive challenge.

Each of the four sets of tasks addressed a different learning construct:

• bias in survey item wording in relation to a survey purpose.
• appropriate plot structures for representing survey data
• interpretability of data in relation to the categorization scheme
• interpretability of data in relation to graph structure

Each set of brainteaser tasks contained a master prompt and four tasks, and each task required a response to a question. As the participants worked through the increasingly challenging tasks within a set, they need to use increasingly sophisticated thinking. Appendix A contains the complete protocol of brainteaser tasks, plus commentary (in italics) about the purpose of each task and how it fit into a progression. Appendix B contains a detailed group of constructed response assessment results from Group A at a KSN pilot in the OST site in Oakland CA, which was operated by Citizen Schools.