Skip to main content
SpringerLink
Log in

A Holistic Approach to Empirical Analysis: The Insignificance of P, Hypothesis Testing and Statistical Significance*

  • Conference paper
  • First Online:
From Analysis to Visualization (JBCC 2017)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 313))

Included in the following conference series:

Abstract

It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.

Jon was one of the most creative, critical, energetic and open-minded academics I ever met and we hit it off almost immediately. Four years ago we started our collaboration on workshops and a book, on underlying fundamental problems to applied statistics, and math from a multidisciplinary perspective. What a severe loss his passing was to the community of critical thinkers and thought leaders. And, I lost a friend.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akerlof, G., Kranton, R.: Economics and identity. Q. J. Econ. 115, 715–753 (2000)

    Google Scholar 

  2. Akerlof, G.A., Shiller, R.J.: Animal Spirits: How Human Psychology Drives the Economy, and Why It Matters for Global Capitalism. Princeton University Press, Princeton (2009)

    Google Scholar 

  3. Altman, M.: The methodology of economics and the survivor principle revisited and revised: some welfare and public policy implications of modeling the economic agent. Rev. Soc. Econ. 57, 427–449 (1999)

    Google Scholar 

  4. Altman, M.: Introduction to special issue on statistical significance. J. Socio-Econ. 33, 523–525 (2004)

    Google Scholar 

  5. Altman, M.: Statistical significance, path dependency, and the culture of journal publication. J. Socio-Econ. 33, 651–663 (2004)

    Google Scholar 

  6. Altman, M.: Mental models, bargaining power and institutional change. In: Paper present at the, World Interdisciplinary Network for Institutional Research Conference, Old Royal Naval College, Greenwich, London, UK, 11–14 September 2014

    Google Scholar 

  7. Altman, M.: A bounded rationality assessment of the new behavioral economics. In: Frantz, R., Chen, S.-H., Dopfer, K., Heukelom, F., Mousavi, S. (eds.) Routledge Handbook of Behavioral Economics, pp. 179–193. Routledge, London (2017)

    Google Scholar 

  8. Altman, M.: A more scientific approach to applied economics: reconstructing statistical, analytical significance, and correlation analysis. Available at SSRN (2018). https://ssrn.com/abstract=3126147 or https://doi.org/10.2139/ssrn.3126147

  9. American Statistical Association. American statistical association releases statement on statistical significance and p-values: provides principles to improve the conduct and interpretation of quantitative science (2016). https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf

  10. Arrow, K.J.: Decision Theory and the Choice of a Level of Significance for the t-Test. In: Olkin, I., et al. (eds.) Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, pp. 70–78. Stanford University Press, Stanford (1960)

    Google Scholar 

  11. Baddeley, M.: Herding, social influence and expert opinion. J. Econ. Methodol. 20, 35–44 (2013)

    Google Scholar 

  12. Bailey, D.H., Borwein, J.M., Brent, R.P., Reisi, M.: Reproducibility in computational science: a case study: randomness of the digits of Pi. Exp. Math. (2016). https://doi.org/10.1080/10586458.2016.1163755

  13. Becker, G.: Accounting for Tastes. Harvard University Press, Cambridge (1996); Coase, R.: Essays on Economics and Economists. University of Chicago Press, Chicago (1994)

    Google Scholar 

  14. Coe, R.: Its the effect size, stupid: what effect size is and why it is important. In: Paper presented at the 2002 Annual Conference of the British Educational Research Association, University of Exeter, Exeter, Devon, England, September 12–14 (2002). http://www.leeds.ac.uk/educol/documents/00002182.htm

  15. Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Routledge, New York (1977)

    Google Scholar 

  16. Deaton, A.: Instruments, randomization, and learning about development. J. Econ. Lit. 48, 424–455 (2010)

    Google Scholar 

  17. Fanelli, D.: How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS One 4 (2009). https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738

  18. Fidler, F., Thomason, N., Cumming, G., Finch, S., Lee, J.: Editors can lead researchers to confidence intervals, but can’t make them think: statistical reform lessons from medicine. Psychol. Sci. 15, 119–126 (2004)

    Google Scholar 

  19. Friedman, M.: The methodology of positive economics. In: Friedman, M. (ed.) Essays in Positive Economics, pp. 3–43. University of Chicago Press, Chicago (1953)

    Google Scholar 

  20. Gallo, A.: A refresher on statistical significance. Harvard Business Review (2016). https://hbr.org/2016/02/a-refresher-on-statistical-significance#comment-section

  21. Harford, T.: Big data: are we making a big mistake? Financial Times (2014). https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0

  22. Keynes, J.M.: The General Theory of Employment, Interest and Money. Macmillan, London (1936)

    Google Scholar 

  23. Leibenstein, H.: Allocative efficiency vs. X-efficiency. Am. Econ. Rev. 56, 392–415 (1966)

    Google Scholar 

  24. Munafõ, M.R., Smith, G.D.: Replication is not enough. Nature: Int. J. Sci. 553, 400–401 (2018). https://www.nature.com/magazine-assets/d41586-018-01023-3/d41586-018-01023-3.pdf

  25. McCloskey, D.: The loss function has been mislaid: the rhetoric of significance tests. Am. Econ. Rev. 75, 201–205 (1985)

    Google Scholar 

  26. McCloskey, D.: The insignificance of statistical significance. Sci. Am. 72, 32–33 (1995)

    Article  Google Scholar 

  27. McCloskey, D.N., Ziliak, S.: The standard error of regressions. J. Econ. Lit. 34, 97–114 (1996)

    Google Scholar 

  28. McCrum-Gardner, E.: Sample size and power calculations made simple. Int. J. Ther. Rehabil. 17, 10–14 (2010)

    Google Scholar 

  29. Morrison, D.E., Henkel, R.E.: The Significance Test Controversy: A Reader. Aldine, Chicago (1970)

    Google Scholar 

  30. Qualtrics. Calculating sample size (2018). https://www.qualtrics.com/blog/calculating-sample-size/

  31. Simon, H.A.: Behavioral Economics. In: Eatwell, J., Millgate, M., Newman, P. (eds.) The New Palgrave: A Dictionary of Economics. Macmillan, London (1987)

    Google Scholar 

  32. Taleb, N.N.: The Black Swan: The Impact of the Highly Improbable. Random House, New York (2007)

    Google Scholar 

  33. Thompson, B.: Why encouraging effect size reporting is not working: the etiology of researcher resistance to changing practices. J. Psychol. 133, 133–141 (1999)

    Google Scholar 

  34. Wasserstein, R.L., Lazar, N.A.: The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133 (2016). https://doi.org/10.1080/00031305.2016.1154108

  35. Ziliak, S., McCloskey, D.N.: Size matters: the standard error of regressions in the American economic. Rev. J. Socio-Econ. 33, 527–546

    Google Scholar 

  36. Ziliak, S., McCloskey, D.N.: The Cult of Statistical Significance: How the Standard Error Costs Jobs, Justice, and Lives. University of Michigan Press, Ann Arbor (2008)

    Google Scholar 

Download references

Acknowledgements

Thanks to Hannah Altman and Louise Lamontagne for their helpful comments and suggestions.

Author information

Authors and Affiliations

  1. Dean, University of Dundee School of Business (UDSB), Dundee, Scotland, UK

    Morris Altman

  2. Chair Professor of Behavioural and Institutional Economics, and Co-operatives, Dundee, Scotland, UK

    Morris Altman

Authors
  1. Morris Altman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Morris Altman .

Editor information

Editors and Affiliations

  1. Lawrence Berkeley National Laboratory, Berkeley, CA, USA

    David H. Bailey

  2. Western University, London, ON, Canada

    Naomi Simone Borwein

  3. Mathematical Sciences Institute, Australian National University, Canberra, ACT, Australia

    Richard P. Brent

  4. School of IT and Mathematical Sciences, University of South Australia, Mawson Lakes, SA, Australia

    Regina S. Burachik

  5. School of Mathematical and Physical Sciences, University of Newcastle, Callaghan, NSW, Australia

    Judy-anne Heather Osborn

  6. School of Mathematical and Physical Sciences, University of Newcastle, Callaghan, NSW, Australia

    Brailey Sims

  7. Department of Mathematics, Western Michigan University, Kalamazoo, MI, USA

    Qiji J. Zhu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Altman, M. (2020). A Holistic Approach to Empirical Analysis: The Insignificance of P, Hypothesis Testing and Statistical Significance*. In: Bailey, D., et al. From Analysis to Visualization. JBCC 2017. Springer Proceedings in Mathematics & Statistics, vol 313. Springer, Cham. https://doi.org/10.1007/978-3-030-36568-4_16

Download citation

Publish with us

Policies and ethics

Search

Navigation