Understanding Test-Taking Strategies for a Reading Comprehension Test via Latent Variable Regression with Pratt’s Importance Measures

Abstract

This chapter considers how the process-based variables of test-taking strategies as reported by test-takers can help to explain the differences in the outcome of a reading comprehension test and serve to provide process level evidence of validity. With the process variables as the explanatory variables, test-takers’ performance was analyzed via a latent variable regression in a structural equation model (SEM), along with Pratt’s importance measures (Pratt, 1987) to assist in understanding the score variation in the latent outcome. We consider how understanding test-taking strategy can help inform test design and validation practices.

Appendices

Appendices

Appendix A

Statements of the test-taking strategy survey items and descriptive statistics

Type	Item #	When answering a question, …	M	SD
CM	1	I needed to understand the main ideas of the passage.	3.81	1.07
	2	I needed to understand some specific sentences in the passage.	3.73	1.00
	3	I needed to read some parts again carefully.	3.75	0.94
SM	4	I quickly summarized or took notes.	2.42	1.20
	5	I translated some words/sentences of the passage.	2.32	1.20
	6	I tried to guess from other sentences.	3.19	1.18
	7	I used clues in the other questions to guess the answer.	2.84	1.22
TW	8	I simply chose the answer that seemed the least wrong.	2.55	1.19
	9	I selected an option that had an important word.	2.49	1.16
	10	I guessed blindly.	2.01	1.19

1. Note. CM comprehending-meaning, SM score-maximizing, TW test-wiseness. The descriptive statistics were computed based on the responses to all four tasks

Appendix B

• 1. Note that texts in italic face followed by “!” are the descriptions of the Mplus syntax.

• 2. Use Mplus standardized outputs under the heading of “STD” to obtain both the $\hat{\beta}$ and $\hat{r}$ for computing the Pratt’s measures.

$$ --------------------------------------------- $$

Title: Regression of the Latent Task Performance (Task-1, Letter) on the 3 the Observed Strategy Types

Data: File is Strategy for Mplus.dat;

Format is 455F5;

Variable:

NAMES ARE id L1–L11 LCM LSM LTW;

! L1-L11 are scores for reading questions of Letter (Task-1).

! LCM, LSM and LTW are the observed scores for CM, SM and TW strategies.

USEVAR ARE L1-L11 LCM LSM LTW;

CATEGORICAL ARE L1-L11;

Missing are all (99);

Model:

P1 by L1-L11; ! Measurement model for latent performance (P1) for Letter (Task-1)

P1 on LCM LSM LTW; ! Latent variable regression (P1) on three types of Strategy

! To obtain correlations for P1 with LCM LSM and LTW, replace “on” by “with” (no “on” command in the model)

Output: Standardized; ! To obtain R ² , $\hat{\beta}$ and $\hat{r}$ for computing Pratt’s measures (Use “STD” standardized)