Other Errors to Avoid

Abstract

This chapter continues the discussion of errors that can be avoided without requiring an extraordinary amount of time or resources but that would, when not avoided, seriously weaken the study design. The errors discussed in this chapter are typically made when researchers want to speed up the evaluation process by spending less time on applied tasks such as collecting test data or installing comparison systems. These errors are made when researchers reuse test data during the evaluation or when they use a demo or default version of comparison systems. Both cases result in an unfair comparison of the new system with another: the first because the system has been tuned to the test data and will perform unusually well, and the second because the comparison system is used in a substandard version and will perform unusually poorly. Then, two other types of errors are discussed, one when no time is spent verifying that the randomization of subjects was successful, and lastly when carelessly constructed Likert scales are used.