Skip to main content
SpringerLink
Log in

Robust Modeling in Cognitive Science

  • Published:
Computational Brain & Behavior Aims and scope Submit manuscript

Abstract

In an attempt to increase the reliability of empirical findings, psychological scientists have recently proposed a number of changes in the practice of experimental psychology. Most current reform efforts have focused on the analysis of data and the reporting of findings for empirical studies. However, a large contingent of psychologists build models that explain psychological processes and test psychological theories using formal psychological models. Some, but not all, recommendations borne out of the broader reform movement bear upon the practice of behavioral or cognitive modeling. In this article, we consider which aspects of the current reform movement are relevant to psychological modelers, and we propose a number of techniques and practices aimed at making psychological modeling more transparent, trusted, and robust.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Source: https://www.bayesianspectacles.org/library/

Fig. 2

Similar content being viewed by others

Notes

  1. Note that registered reports involve more than preregistration: They also involve a journal’s guarantee that a paper will be published regardless of how the data turn out.

  2. https://www.kaggle.com/competitions

  3. https://cpc-18.com/

References

  • Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école Américaine. Econometrica, 21, 503–546.

    Article  Google Scholar 

  • Allais, M. (1979). The foundations of a positive theory of choice involving risk and a criticism of the postulates and axioms of the American School. In Allais, M., & Hagen, O. (Eds.) Expected utility hypothesis and the Allais paradox (pp. 27–145). Dordrecht: Riedel.

  • Alogna, V., Attaya, M.K., Aucoin, P., Bahník, Š., Birch, S., Birt, A.R., et al. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.

    Article  PubMed  Google Scholar 

  • Batchelder, W.H. (2010). Cognitive psychometrics: using multinomial processing tree models as measurement tools. In Embretson, S. (Ed.) Measuring psychological constructs: advances in model-based approaches. Washington, DC: American Psychological Association Books.

  • Baumeister, A.A., & Kellas, G. (1968). Reaction time and mental retardation. International Review of Research in Mental Retardation, 3, 163–193.

    Article  Google Scholar 

  • Baumgaertner, B., Devezer, B., Buzbas, E.O., Nardin, L.G. (2018). A model-centric analysis of openness, replication, and reproducibility. arXiv:1811.04525.

  • Beck, K. (2003). Test-driven development: by example. Addison-Wesley Professional.

  • Bell, R.M., Koren, Y., Volinsky, C. (2010). All together now: a perspective on the Netflix prize. Chance, 23, 24–29.

    Article  Google Scholar 

  • Birnbaum, M.H., & Quispe-Torreblanca, E.G. (2018). TEMAP2.R: true and error model analysis program in R. Judgment and Decision Making, 13, 428–440.

    Google Scholar 

  • Bones, A.K. (2012). We knew the future all along: scientific hypothesizing is much more accurate than other forms of precognition—a satire in one part. Perspectives on Psychological Science, 7, 307–309.

    Article  PubMed  Google Scholar 

  • Brown, G.D.A., Neath, I., Chater, N. (2007). A temporal ratio model of memory. Psychological Review, 114(1), 539–576.

    Article  PubMed  Google Scholar 

  • Brown, S.D., Marley, A.A.J., Donkin, C., Heathcote, A. (2008). An integrated model of choices and response times in absolute identification. Psychological Review, 115, 396–425.

    Article  PubMed  Google Scholar 

  • Busemeyer, J.R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171–189.

    Article  PubMed  Google Scholar 

  • Cavagnaro, D.R., Pitt, M.A., Gonzalez, R., Myung, I.J. (2013). Discriminating among probability weighting functions with adaptive design optimization. Journal of Risk and Uncertainty, 47, 255–289.

    Article  PubMed  PubMed Central  Google Scholar 

  • Chambers, C.D. (2013). Registered reports: a new publishing initiative at cortex. Cortex, 49, 609–610.

    Article  PubMed  Google Scholar 

  • Chambers, C.D., Dienes, Z., McIntosh, R.D., Rotshtein, P., Willmes, K. (2015). Registered reports: realigning incentives in scientific publishing. Cortex, 66, A1–A2.

    Article  PubMed  Google Scholar 

  • Cook, S.R., Gelman, A., Rubin, D.B. (2006). Validation of software for Bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics, 15(3), 675–692.

    Article  Google Scholar 

  • Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2011). Output interference in recognition memory. Journal of Memory and Language, 64(4), 316–326.

    Article  Google Scholar 

  • Dutilh, G., Vandekerckhove, J., Ly, A., Matzke, D., Pedroni, A., Frey, R., Wagenmakers, E.-J. (2017). A test of the diffusion model explanation for the worst performance rule using preregistration and blinding. Attention, Perception, & Psychophysics, 79, 713–725.

    Article  Google Scholar 

  • Dutilh, G., Annis, J., Brown, S.D., Cassey, P., Evans, N.J., Grasman, R.P.P.P., Donkin, C. (2018). The quality of response time data inference: a blinded, collaborative assessment of the validity of cognitive models. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-017-1417-2.

  • Epper, T., & Fehr-Duda, H. (2018). The missing link: unifying risk taking and time discounting (Tech. Rep. Nos. Department of Economics Discussion Paper 2018–12). St. Gallen: University of St. Gallen.

    Google Scholar 

  • Evans, N.J., Holmes, W.R., Trueblood, J.S. (2018). Response time data provides critical constraints on dynamic models of multi-alternative, multi-attribute choice. https://osf.io/h7e6v/ (Manuscript submitted for publication).

  • Farrell, S., & Lewandowsky, S. (2018). Computational modeling of cognition and behavior. Cambridge University Press.

  • Finkbeiner, S.D., Briscoe, A.D., Reed, R.D. (2014). Warning signals are seductive: relative contributions of color and pattern to predator avoidance and mate attraction in Heliconius butterflies. Evolution, 68, 3410–3420.

    Article  PubMed  Google Scholar 

  • Garner, W.R. (1953). An informational analysis of absolute judgments of loudness. Journal of Experimental Psychology, 46, 373–380.

    Article  PubMed  Google Scholar 

  • Gerrein, J.R., & Chechile, R.A. (1977). Storage and retrieval processes of alcohol-induced amnesia. Journal of Abnormal Psychology, 86(3), 285.

    Article  PubMed  Google Scholar 

  • Gould, S.J. (1996). The mismeasure of man. WW Norton & Company.

  • Guan, M., Lee, M.D., Vandekerckhove, J. (2015). A hierarchical cognitive threshold model of human decision making on different length optimal stopping problems. In Dale, R. et al. (Eds.) Proceedings of the 37th annual conference of the cognitive science society. Austin: Cognitive Science Society.

  • Hardwicke, T.E., & Ioannidis, J. (2018). Mapping the universe of registered reports. BITSS. https://doi.org/10.17605/OSF.IO/FZPCY, http://osf.io/preprints/bitss/fzpcy.

  • Heathcote, A., Brown, S.D., Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In Forstmann, B.U., & Agenmakers, E.-J. (Eds.) An introduction to model-based cognitive neuroscience (pp. 25–48): Springer.

  • Holland, M.K., & Lockhead, G.R. (1968). Sequential effects in absolute judgments of loudness. Perception & Psychophysics, 3, 409–414.

    Article  Google Scholar 

  • Huber, J., Payne, J.W., Puto, C. (1982). Adding asymmetrically dominated alternatives: violations of regularity and the similarity hypothesis. Journal of Consumer Research, 9(1), 90–98.

    Article  Google Scholar 

  • Jensen, A.R. (2006). Clocking the mind: mental chronometry and individual differences. Elsevier.

  • Kass, R.E., & Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377–395.

    Google Scholar 

  • Kerr, N.L. (1998). Harking: hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.

    Article  PubMed  Google Scholar 

  • Kılıč, A., Criss, A.H., Malmberg, K.J., Shiffrin, R.M. (2017). Models that allow us to perceive the world more accurately also allow us to remember past events more accurately via differentiation. Cognitive psychology, 92, 65–86.

    Article  PubMed  Google Scholar 

  • Klein, R.A., Ratliff, K.A., Vianello, M., Adams, R.B.J., Bahník, S., Bernstein, M.J., et al. (2014). Investigating variation in replicability: a ”many labs” replication project. Social Psychology, 45, 142–152.

    Article  Google Scholar 

  • Latty, T., & Beekman, M. (2010). Irrational decision-making in an amoeboid organism: transitivity and context-dependent preferences. Proceedings of the Royal Society B: Biological Sciences, 278(1703), 307–312.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lee, M.D. (2018). Bayesian methods in cognitive modeling. In Wixted, J., & Wagenmakers, E.-J. (Eds.) The Stevens’ handbook of experimental psychology and cognitive neuroscience methodology. 4th edn., Vol. 5: Wiley.

  • Lee, M.D., & Wagenmakers, E.-J. (2013). Bayesian cognitive modeling: a practical course. Cambridge University Press.

  • Liew, S.X., Howe, P.D., Little, D.R. (2016). The appropriacy of averaging in the study of context effects. Psychonomic Bulletin & Review, 23(5), 1639–1646.

    Article  Google Scholar 

  • Luce, R.D. (1959). Individual choice behavior: a theoretical analysis. New York: Wiley.

    Google Scholar 

  • MacCoun, R.J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Lilienfeld, S., & Waldman, I. (Eds.) Psychological science under scrutiny: recent challenges and proposed solutions (pp. 297–322): Wiley.

  • Marsh, H.W., Morin, A.J., Parker, P.D., Kaur, G. (2014). Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annual Review of Clinical Psychology, 10, 85–110.

    Article  PubMed  Google Scholar 

  • Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H.A., van der Molen, M.W., Wagenmakers, E.-J. (2015). The effect of horizontal eye movements on free recall: a preregistered adversarial collaboration. Journal of Experimental Psychology: General, 144, e1–e15.

    Article  Google Scholar 

  • Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., et al. (2014). Promoting transparency in social science research. Science, 343, 30–31.

    Article  PubMed  PubMed Central  Google Scholar 

  • Munafò, M. R., Nosek, B.A., Bishop, D.V., Button, K.S., Chambers, C.D., du Sert, N.P., Ioannidis, J.P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.

    Article  PubMed  PubMed Central  Google Scholar 

  • Murdock, B.B. (1960). The distinctiveness of stimuli. Psychological Review, 67, 16–31.

    Article  PubMed  Google Scholar 

  • Murdock, B.B. (1962). The serial position effect in free recall. Journal of Experimental Psychology, 64, 482–488.

    Article  Google Scholar 

  • Myung, I.J., Forster, M.R., Browne, M.W. (2000). A special issue on model selection. Journal of Mathematical Psychology, 44, 1–2.

    Article  PubMed  Google Scholar 

  • Navarro, D.J. (in press). Between the devil and the deep blue sea: tensions between scientific judgement and statistical model selection. Computational Brain & Behavior.

  • Noble, W.S. (2009). A quick guide to organizing computational biology projects. PLOS Computational Biology, 5(7), 1–5. https://doi.org/10.1371/journal.pcbi.1000424.

    Article  Google Scholar 

  • Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., et al. (2015). Promoting an open research culture. Science, 348, 1422–1425.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nosek, B.A., Ebersole, C.R., DeHaven, A.C., Mellor, D.T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 201708274.

  • Oberauer, K., Lewandowsky, S., Avh, E., Brown, G.D., Conway, A., Covan, N., et al. (2018). Benchmarks for models of short term and working memory. Psychological Bulletin.

  • Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.

    Article  Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.

    Article  Google Scholar 

  • Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: a crisis of confidence? Perspectives on Psychological Science, 7, 528– 530.

    Article  PubMed  Google Scholar 

  • Pitt, M.A., Kim, W., Navarro, D.J., Myung, J.I. (2006). Global model analysis by parameter space partitioning. Psychological Review, 113(1), 57.

    Article  PubMed  Google Scholar 

  • Platt, J.R. (1964). Strong inference. Science, 146(3642), 347–353.

    Article  PubMed  Google Scholar 

  • Popper, K.R. (1959). The logic of scientific discovery. London: Routledge.

    Google Scholar 

  • Ratcliff, R., Schmiedek, F., McKoon, G. (2008). A diffusion model explanation of the worst performance rule for reaction time and IQ. Intelligence, 36, 10–17.

    Article  PubMed  PubMed Central  Google Scholar 

  • Roe, R.M., Busemeyer, J.R., Townsend, J.T. (2001). Multialternative decision field theory: a dynamic connectionst model of decision making. Psychological Review, 108(2), 370.

    Article  PubMed  Google Scholar 

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641.

    Article  Google Scholar 

  • Rouder, J., Haaf, J.M., Snyder, H.K. (in press). Minimizing mistakes in psychological science. Advances in Methods and Practices in Psychological Science, https://psyarxiv.com/gxcy5/.

  • Rubin, D.C., Hinton, S., Wenzel, A. (1999). The precise time course of retention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 1161–1176.

    Google Scholar 

  • Shanks, D.R., Newell, B.R., Lee, E.H., Balakrishnan, D., Ekelund, L., Cenac, Z., Moore, C. (2013). Priming intelligent behavior: an elusive phenomenon. PloS ONE, 8, e56515.

    Article  PubMed  PubMed Central  Google Scholar 

  • Shiffrin, R.M., Lee, M.D., Kim, W.-J., Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.

    Article  PubMed  Google Scholar 

  • Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., et al. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1, 337–356.

    Article  Google Scholar 

  • Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.

    Article  PubMed  Google Scholar 

  • Simonsohn, U. (2013). Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888.

    Article  PubMed  Google Scholar 

  • Simonson, I. (1989). Choice based on reasons: the case of attraction and compromise effects. Journal of Consumer Research, 16(2), 158–174.

    Article  Google Scholar 

  • Spellman, B.A. (2015). A short (personal) future history of revolution 2.0. Perspectives on Psychological Science, 10, 886–899.

    Article  PubMed  Google Scholar 

  • Stewart, N., Brown, G.D.A., Chater, N. (2005). Absolute identification by relative judgment. Psychological Review, 112, 881–911.

    Article  PubMed  Google Scholar 

  • Sun, R. (2008). The Cambridge handbook of computational psychology. New York: Cambridge University Press.

    Book  Google Scholar 

  • Trueblood, J.S., Brown, S.D., Heathcote, A. (2015). The fragile nature of contextual preference reversals: reply to tsetsos, chater, and usher (2015). Psychological Review, 122(4), 848–853. https://doi.org/10.1037/a0039656.

    Article  PubMed  Google Scholar 

  • Turner, B.M., Schley, D.R., Muller, C., Tsetsos, K. (2017). Competing theories of multialternative, multiattribute preferential choice. Psychological Review, 125(3), 329–362. https://doi.org/10.1037/rev0000089.

    Article  PubMed  Google Scholar 

  • Tversky, A. (1972). Elimination by aspects: a theory of choice. Psychological Review, 79(4), 281.

    Article  Google Scholar 

  • Unsworth, N., Redick, T.S., Lakey, C.E., Young, D.L. (2010). Lapses in sustained attention and their relation to executive control and fluid abilities: an individual differences investigation. Intelligence, 38, 111–122.

    Article  Google Scholar 

  • Usher, M., & McClelland, J.L. (2004). Loss aversion and inhibition in dynamical models of multialternative choice. Psychological Review, 111(3), 757.

    Article  PubMed  Google Scholar 

  • Vandekerckhove, J., Matzke, D., Wagenmakers, E.-J. (2015). Model comparison and the principle of parsimony. In Busemeyer, J. R., Wang, Z., Townsend, J. T., Eidels, A. (Eds.) Oxford handbook of computational and mathematical psychology (pp. 300–317). New York: Oxford University Press.

  • Vanpaemel, W. (2010). Prior sensitivity in theory testing: an apologia for the Bayes factor. Journal of Mathematical Psychology, 54, 491–498.

    Article  Google Scholar 

  • Vanpaemel, W., Vermorgen, M., Deriemaecker, L., Storms, G. (2015). Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra, 1, 3. https://doi.org/10.1525/collabra.13.

    Article  Google Scholar 

  • Voss, A., Rothermund, K., Voss, J. (2004). Interpreting the parameters of the diffusion model: an empirical validation. Memory & Cognition, 32, 1206–1220.

    Article  Google Scholar 

  • Wagenmakers, E.-J. (2012). A year of horrors. De Psychonoom, 27, 12–13.

    Google Scholar 

  • Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11(1), 192–196. https://doi.org/10.3758/BF03206482.

    Article  Google Scholar 

  • Wagenmakers, E.-J., Grünwald, P., Steyvers, M. (2006). Accumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149–166.

    Article  Google Scholar 

  • Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H.L., Kievit, R.A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.

    Article  PubMed  Google Scholar 

  • Watts, D.J. (2017). Should social science be more solution-oriented? Nature Human Behaviour, 1, 1–5.

    Article  Google Scholar 

  • Wicherts, J.M., Borsboom, D., Kats, J., Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726–728.

    Article  PubMed  Google Scholar 

  • Zhang, S., & Lee, M.D. (2010). Optimal experimental design for a class of bandit problems. Journal of Mathematical Psychology, 54, 499–508.

    Article  Google Scholar 

  • Zwaan, R.A., Etz, A., Lucas, R.E., Donnellan, M.B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41.

Download references

Acknowledgments

This article is the product of the Workshop on Robust Social Science held in St. Petersburg, FL, in June 2018.

Funding

The workshop was made possible by generous funding from the National Science Foundation (grant no. BCS-1754205) to Joachim Vandekerckhove and Michael Lee of the University of California, Irvine. Alexander Etz was supported by NSF GRFP no. DGE-1321846. Berna Devezer was supported by NIGMS of the NIH under award no. P20GM104420. Dora Matzke was supported by a Veni grant (no. 451-15-010) from the Netherlands Organization of Scientific Research (NWO). Jennifer Trueblood was supported by NSF no. SES-1556325.

Author information

Authors and Affiliations

  1. Department of Cognitive Sciences, University of California, Irvine, Irvine, CA, USA

    Michael D. Lee, Alexander Etz, Jeffrey N. Rouder & Joachim Vandekerckhove

  2. Department of Psychology, Syracuse University, Syracuse, NY, USA

    Amy H. Criss

  3. Department of Business, University of Idaho, Moscow, ID, USA

    Berna Devezer

  4. School of Psychology, University of New South Wales, Sydney, Australia

    Christopher Donkin

  5. Department of Psychology, The Ohio State University at Lima, Lima, OH, USA

    Fábio P. Leite

  6. Department of Psychology, University of Amsterdam, Amsterdam, Netherlands

    Dora Matzke

  7. Department of Psychology, Vanderbilt University, Nashville, TN, USA

    Jennifer S. Trueblood

  8. Department of Psychology, Missouri Western State University, St Joseph, MO, USA

    Corey N. White

Authors
  1. Michael D. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amy H. Criss
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Berna Devezer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christopher Donkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexander Etz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fábio P. Leite
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dora Matzke
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jeffrey N. Rouder
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jennifer S. Trueblood
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Corey N. White
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Joachim Vandekerckhove
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joachim Vandekerckhove.

Ethics declarations

Disclaimer

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the National Science Foundation and Netherlands Organization of Scientific Research.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, M.D., Criss, A.H., Devezer, B. et al. Robust Modeling in Cognitive Science. Comput Brain Behav 2, 141–153 (2019). https://doi.org/10.1007/s42113-019-00029-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42113-019-00029-y

Keywords

Search

Navigation