Reverse Engineering the NAEP Floating Pencil Task Using the PADI Design System (PADI Technical Report 16)

Abstract

Large-scale science assessments have been criticized for not tapping rich, authentic scientific problems. The use of evidence-centered design (ECD) principles in large-scale assessment design can potentially improve the quality of performance assessment tasks for scientific inquiry. The Principled Assessment Designs for Inquiry (PADI) design system has been used by the PADI team to analyze several well-known science assessments. This report will discuss the reverse engineering of the National Assessment of Educational Progress (NAEP) Floating Pencil task. The PADI design system was used as an analytical tool for understanding the characteristics and underlying assessment argument of this chosen task. Carrying out this work, we grappled with task complexity and came to understand how science performance assessment tasks might be constructed in the future.

In reverse engineering the Floating Pencil task into an assessment blueprint, our team created a task specification on the PADI design system. In designing the Floating Pencil task specification, we specified the Task Model, Student Model, and Evidence Model. In defining the Task Model, we considered what family of tasks Floating Pencil might be a member of—helping to define the fixed characteristics and Task Model Variables for Floating Pencil. Our team considered a variety of potential Student Models and chose one based on the NAEP content-by-process framework (Allen, Carlson, & Zelenak, 1999). The chosen Evidence Model included an evaluative submodel that mirrored NAEP’s rubric for Floating Pencil (publicly released, see https://nces.ed.gov/nationsreportcard/itmrlsx/sampleq/96sci8.pdf. We defined unidimensional Rasch Measurement Models for each Activity within Floating Pencil.

The use of the PADI design system to reverse engineer the Floating Pencil task resulted in the creation of new assessment knowledge, general and specific to Floating Pencil, for our team and the PADI project. We considered the coherence and linkages among the Task Model, Student Model, and Evidence Model and grappled with the underlying assessm ent argument for Floating Pencil. Reverse engineering the Floating Pencil task not only contributed to our knowledge of the characteristics of one particular large-scale performance assessment task, but shed light on how new science performance assessments might be forward engineered.