Abstract
New and old methods of analyzing two-choice experiments with confidence ratings are evaluated. These include the theory of signal detectability (TSD), Luce's choice theory,nonparametric techniques based on areas under receiver-operating characteristic (ROC) functions, and methods based onS′ and Ω, proposed by Balakrishnan and his colleagues. New methods for assessing the bias of a complete ROC function are proposed, together with an additional area-based measure of response bias. Area measures of both sensitivity and bias proved the most consistent. Response bias for a full ROC function was larger than bias at the cut point and also provided additional information. Participants showed voluntary control of bias for all measures except Ω. Unequal variance versions of TSD and choice models gave similar fits to data, with the choice model closer to an equal variance version. Discrimination data from Balakrishnan (1999) formed the empirical test bed.
Article PDF
Similar content being viewed by others
References
Agresti, A. (1996).An introduction to categorical data analysis. Chichester, U.K.: Wiley.
Balakrishnan, J. D. (1998a). Measures and interpretations of vigilance performance: Evidence against the detection criterion.Human Factors,40, 601–623.
Balakrishnan, J. D. (1998b). Some more sensitive measures of sensitivity and response bias.Psychological Methods,3, 68–90.
Balakrishnan, J. D. (1999). Decision processes in discrimination: Fundamental misrepresentations of signal detection theory.Journal of Experimental Psychology: Human Perception & Performance,25, 1189–1206.
Balakrishnan, J. D., &MacDonald, J. A. (2002). Decision criteria do not shift: Reply to Treisman.Psychonomic Bulletin & Review,9, 858–865.
Balakrishnan, J. D., & MacDonald, J. A. (2003).Alternatives to signal detection theory. Retrieved August 17, 2003, from www.psych .purdue.edu/~beowulf/dsdt/dsdt.html.
Balakrishnan, J. D., MacDonald, J. A., &Kohen, H. S. (2003). Is the area measure a historical anomaly?Canadian Journal of Experimental Psychology,57, 238–256.
Craig, A. (1979). Nonparametric measures of sensory efficiency for sustained monitoring tasks.Human Factors,21, 69–78.
Dusoir, A. E. (1975). Treatments of bias in detection and recognition models: A review.Perception & Psychophysics,17, 167–178.
Dusoir, A. E. (1983). Isobias curves in some detection tasks.Perception & Psychophysics,33, 403–412.
Egan, J. P., Schulman, A. I., &Greenberg, G. Z. (1959). Operating characteristics determined by binary decisions and by ratings.Journal of the Acoustical Society of America,31, 768–773.
Green, D. M., &Luce, R. D. (1973). Speed-accuracy trade off in auditory detection. In S. Kornblum (Ed.),Attention and performance IV (pp. 547–569). New York: Academic Press.
Grier, J. B. (1971). Nonparametric indexes for sensitivity and bias: Computing formulas.Psychological Bulletin,75, 424–429.
Heath, R. A., &Fulham, R. (1988). An adaptive filter model for recognition memory.British Journal of Mathematical & Statistical Psychology,41, 119–144.
Hodos, W. (1970). A nonparametric index of response bias for use in detection and recognition experiments.Psychological Bulletin,74, 351–354.
Irwin, R. J., Hautus, M. J., &Francis, M. A. (2001). Indices of response bias in thesame-different experiment.Perception & Psychophysics,63, 1091–1100.
Kornbrot, D. E. (1978). Theoretical and empirical comparison of Luce's choice model and logistic Thurstone model of categorical judgment.Perception & Psychophysics,24, 193–208.
Kornbrot, D. E. (1980). Attention bands.British Journal of Mathematical & Statistical Psychology,33, 1–16.
Kornbrot, D. E. (1984). Mechanisms for categorisation.British Journal of Mathematical & Statistical Psychology,37, 84–198.
Kornbrot, D. E. (1988). Random walk models of binary choice.Acta Psychologica,69, 109–127.
Kornbrot, D. E., Galanter, E. G., &Donnelly, M. (1981). Estimates of utility function parameters.Journal of Experimental Psychology: Human Perception & Performance,7, 441–458.
Laming, D. R. J. (1968).Information theory of choice reaction times. London: Academic Press.
Laming, D. R. J. (1979). A critical comparison of two random-walk models for two-choice reaction data.Acta Psychologica,43, 431–453.
Link, S. W. (1975). The relative judgment theory of two-choice reaction time.Journal of Mathematical Psychology,12, 114–135.
Luce, R. D. (1959).Individual choice behavior. New York: Wiley.
Luce, R. D. (1986).Response times. Oxford: Oxford University Press, Clarendon Press.
Macmillan, N. A. (2002). Signal detection theory. In H. Pashler & J. Wixted (Eds.),Stevens' handbook of experimental psychology: Vol. 4. Methodology in experimental psychology (pp. 43–90). New York: Wiley.
Macmillan, N. A., &Creelman, C. D. (1991).Detection theory: A user's guide. Cambridge: Cambridge University Press.
Macmillan, N. A., &Creelman, C. D. (1996). Triangles in ROC space: History and theory of “nonparametric” measures of sensitivity and response bias.Psychonomic Bulletin & Review,3, 164–170.
Macmillan, N. A., Rotello, C. M., &Miller, J. (2004). The sampling distributions of Gaussian ROC statistics.Perception & Psychophysics,66, 406–421.
McCarthy, D., &Davison, M. (1981). Towards a behavioral theory of bias in signal detection.Perception & Psychophysics,29, 371–382.
McCarthy, D., &Davison, M. (1984). Isobias and alloiobias functions in animal psychophysics.Journal of Experimental Psychology: Animal Behavior Processes,10, 390–409.
Pollack, I., &Norman, D. A. (1964). A non-parametric analysis of recognition experiments.Psychonomic Science,1, 125–126.
Smith, P. L., &Vickers, D. (1989). Modelling evidence accumulation with partial loss in expanded judgment.Journal of Experimental Psychology: Human Perception & Performance,15, 797–815.
Stone, G. C., &Callaway, E. (1964). Effects of stimulus probability on reaction time in a number-naming task.Quarterly Journal of Experimental Psychology,16, 47–55.
Swets, J. A. (1986). Form of empirical ROCs in discrimination and diagnostic tasks.Psychological Bulletin,99, 181–198.
Treisman, M. (2002). Is signal detection theory fundamentally flawed? A response to Balakrishnan (1998a, 1998b, 1999).Psychonomic Bulletin & Review,9, 845–857.
Vickers, D., Caudrey, D., &Willson, R. [J.] (1971). Discriminating between the frequency of occurrence of two alternative events.Acta Psychologica,35, 151–172.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kornbrot, D.E. Signal detection theory, the approach of choice: Model-based and distribution-free measures and evaluation. Perception & Psychophysics 68, 393–414 (2006). https://doi.org/10.3758/BF03193685
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03193685