Booklet Design for a Longitudinal Study: Measuring Progress in New-Immigrant Children’s Mathematics Achievement

Abstract

Because of the increasing number of new-immigrant children in Taiwan, we conducted a large-scale assessment of mathematics achievement at the Grade 4 level in 2012 and performed a follow-up of a sample of panel members in 2014. The objective of the study was to develop a valid instrument in order to measure and investigate the average new-immigrant student’s progress between Grade 4 and Grade 6 in mathematical literacy. We compiled 78 selected-response items into a set of 13 booklets. All the test items were selected from the Taiwan Assessment of Student Achievement (TASA 2012), which is the largest nationally representative and continuing assessment in Taiwan. Each item is designed to measure one of the four mathematics content areas, which are number and measurement, geometry, statistics and probability, and algebra. We propose three rules for identifying and selecting appropriate items. First, the number of anchor items is 39 (50 %), providing the basis for equating scores on different grades. Second, we designated as high priorities Grade 4 items with a difficulty parameter larger than −0.5 and Grade 6 items with a parameter between −1 and 0.5. Third, the proportions of each content area should be similar to those listed in the General Guidelines of Grades 1–9 Curriculum for Mathematics. We applied a three-parameter logistic model and used PARSCALE to estimate the item parameters. After these procedures, each test block contained six items, and each booklet comprised four blocks based on the balanced incomplete block design. The mean difficulty of the items in each block ranged from 0.053 to 0.153. Consequently, the mean difficulty of the items in each booklet ranged from 0.083 to 0.128. The distributions of items across the content areas were 64.10 %, 19.23 %, 5.13 %, and 11.54 %, mostly corresponding to the curriculum framework. This study demonstrated that a detailed consideration of the percentage of anchor items, the range of item difficulties, and the distribution of content areas can be useful for constructing a measurement tool in a longitudinal study.