Abstract
The Programme for International Students Assessment (PISA) has become one of the most influential forces in global education. The growing influence has been accompanied by growing criticism. For nearly two decades since the first round of PISA was conducted in 2000, the global assessment program has been roundly scrutinized and criticized by education researchers all over the world. But the mounting criticism seems to have had little impact on PISA’s influence as evidenced by its growing power in global education policy and practice. The lack of impact of criticism does not mean the criticism is not valid or PISA has improved. It simply means that the criticism has been largely ignored. The lack of impact is no reason to give up exposing PISA as a flawed business that has great power to misguide education. The expanding influence of the PISA enterprise makes it more even more important to be critical of this juggernaut today. It is also important to consider more effective and more straightforward approaches to present the criticism. The purpose of this article is to present a summary of criticisms that reveal the most fundamental flaws of PISA in non-technical language in one place. Specifically, the article focuses on criticisms of PISA’s three fundamental deficiencies: its underlying view of education, its implementation, and its interpretation and impact on education globally.
Similar content being viewed by others
References
Arzarello, F., Garuti, R., & Ricci, R. (2015). The impact of PISA studies on the Italian national assessment system. In K. Stacey & R. Turner (Eds.), Assessing mathematical literacy (pp. 249–260). Berlin: Springer.
Baird, J.-A., Johnson, S., Hopfenbeck, T. N., Isaacs, T., Sprague, T., Stobart, G., et al. (2016). On the supranational spell of PISA in policy. Educational Research,58(2), 121–138.
Barber, M., & Mourshed, M. (2007). How the world’s best-performing school systems come out on top. Retrieved August 3, 2010 from New York https://www.mckinsey.com/industries/social-sector/our-insights/how-the-worlds-best-performing-school-systems-come-out-on-top.
Baroutsis, A., & Lingard, B. (2017). Counting and comparing school performance: An analysis of media coverage of PISA in Australia, 2000–2014. Journal of Education Policy,32(4), 432–449.
Berliner, D. C. (2011). The context for interpreting PISA results in the USA: Negativism, chauvinism, misunderstanding, and the potential to distort the educational systems of nations. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 77–96). New York: Springer.
Bieber, T., & Martens, K. (2011). The OECD PISA study as a soft power in education? Lessons from Switzerland and the US. European Journal of Education,46(1), 101–116. https://doi.org/10.1111/j.1465-3435.2010.01462.x.
Breakspear, S. (2012). The policy impact of Pisa: An exploration of the normative effects of international benchmarking in school system performance. Retrieved July 3, 2016 from Paris http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=EDU/WKP(2012)8&docLanguage=En.
Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education,33(1), 7–26.
Campbell, M. (2013). West vs Asia education rankings are misleading: Western schoolchildren are routinely outperformed by their Asian peers, but worrying about it is pointless. New Scientist (2898). Retrieved January 10, 2015 from https://www.newscientist.com/article/mg21728985-800-west-vs-asia-education-rankings-are-misleading/.
Carney, S., Rappleye, J., & Silova, I. (2012). Between faith and science: World culture theory and comparative education. Comparative Education Review,56(3), 366–393.
Domínguez, M., Vieira, M.-J., & Vidal, J. (2012). The impact of the programme for international student assessment on academic journals. Assessment in Education: Principles, Policy & Practice,19(4), 393–409.
Eivers, E. (2010). PISA: Issues in implementation and interpretation. The Irish Journal of Education/Iris Eireannach an Oideachais,38, 94–118.
Feniger, Y., & Lefstein, A. (2014). How not to reason with PISA data: An ironic investigation. Journal of Education Policy,29(6), 845–855. https://doi.org/10.1080/02680939.2014.892156.
Figazzolo, L. (2009). Impact of PISA 2006 on the education policy debate. Retrieved September 9, 2015 from http://download.ei-ie.org/docs/IRISDocuments/ResearchWebsiteDocuments/2009-00036-01-E.pdf.
Friedman, T. L. (2007). The world is flat: A brief history of the twenty-first century. New York: Farrar, Straus and Giroux.
Gewertz, C. (2018). PISA working toward exam to gauge creative thinking—Built by ACT. Education Week,38(6), 4.
Goldstein, H., Bonnet, G., & Rocher, T. (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics,32(3), 252–286. https://doi.org/10.3102/1076998606298042
Gould, S. J. (1996). The mismeasure of man. New York: Northon.
Green, A., & Mostafa, T. (2013). The dynamics of education systems: Convergent and divergent trends, 1990–2010. In J. Janmaat, M. Duru-Bellat, P. Méhaut, & A. Green (Eds.), The dynamics and social outcomes of education systems (pp. 15–45). Berlin: Springer.
Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy,24(1), 23–37.
Grisay, A., De Jong, J., Gebhardt, E., Berezner, A., & Halleux-Monseur, B. (2007). Translation equivalence across PISA countries. Journal of Applied Measurement,8(3), 249.
Grisay, A., & Gonzalez, E. (2009). Equivalence of item difficulties across national versions of the PIRLs and PISA reading assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments,2, 63–83.
Grönqvist, E., & Vlachos, J. (2016). One size fits all? The effects of teachers’ cognitive and social abilities on student achievement. Labour Economics,42, 138–150.
Gruber, K. H. (2006). The German ‘PISA-Shock’: Some aspects of the extraordinary impact of the OECD’s PISA study on the German education system. Paper presented at the Cross-national Attraction in Education: Accounts from England and Germany. Oxford: Symposium Books.
Hanushek, E. A. (2013). Economic growth in developing countries: The role of human capital. Economics of Education Review,37, 204–212.
Hanushek, E. A., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature,46, 607–668.
Hanushek, E. A., & Woessmann, L. (2010). The high cost of low educational performance: The long-run economic impact of improving PISA Outcomes. Retrieved March 20, 2011 from Paris http://books.google.com/books?id=k7AGPo0NvfYC&pg=PA33&lpg=PA33&dq=hanushek+pisa+gdp&source=bl&ots=2gCfzF-f1_&sig=wwe0XLL5EblVWK9e7RJfb5MyhIU&hl=en&sa=X&ei=MLPCUqaOD8-JogS6v4C4Bw&ved=0CGcQ6AEwBjgK-v=onepage&q=hanushek%20pisa%20gdp&f=false.
Hanushek, E. A., & Woessmann, L. (2012). Do better schools lead to more growth? Cognitive skills, economic outcomes, and causation. Journal of Economic Growth,17(4), 267–321.
Hargreaves, A., & Sahlberg, P. (2015). The tower of PISA is badly leaning. An argument for why it should be saved. Retrieved April 5, 2017 from https://www.washingtonpost.com/news/answer-sheet/wp/2015/03/24/the-tower-of-pisa-is-badly-leaning-an-argument-for-why-it-should-be-saved/?noredirect=on&utm_term=.69275346b5d3.
Harris, A., & Jones, M. S. (2015). Leading futures: Global perspectives on educational leadership. New Delhi: SAGE Publications India.
Hatzinikita, V., Dimopoulos, K., & Christidou, V. (2008). PISA test items and school textbooks related to science: A textual comparison. Science Education,92(4), 664–687.
Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research,62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726.
Hopmann, S. T. (2008). No child, no school, no state left behind: Schooling in the age of accountability. Journal of Curriculum Studies,40(4), 417–456.
Hopmann, S. T., Brinek, Gertrude, & Retzl, M. (Eds.). (2007). PISA zufolge PISA—PISA according to PISA. Berlin: Lit Verlag.
Jones, R. S. (2011). Education reform in Japan. Paris: OECD. https://doi.org/10.1787/5kg58z7g95np-en.
Kamens, D. H. (2015). A maturing global testing regime meets the world economy: Test scores and economic growth, 1960–2012. Comparative Education Review,59(3), 420–446.
Kjærnsli, M., & Lie, S. (2011). Students’ preference for science careers: International comparisons based on PISA 2006. International Journal of Science Education,33(1), 121–144.
Klees, S. J. (2016). Human capital and rates of return: Brilliant ideas or ideological dead ends? Comparative Education Review,60(4), 644–672.
Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on invalid statistics? Hanushek, Woessmann, PISA, and economic growth. Comparative Education,53(2), 166–191.
Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika. https://doi.org/10.1007/s11336-013-9347-z.
Kuramoto, N., & Koizumi, R. (2016). Current issues in large-scale educational assessment in Japan: Focus on national assessment of academic ability and university entrance examinations. Assessment in Education: Principles, Policy & Practice. https://doi.org/10.1080/0969594X.2016.1225667.
Labaree, D. (1997). Public goods, private goods: The American struggle over educational goals. American Educational Research Journal,34(1), 39–81.
Labaree, D. F. (2014). Let’s measure what no one teaches: PISA, NCLB, and the shrinking aims of education. Teachers College Record,116(9), 1–14.
Lewis, S. (2017). Policy, philanthropy and profit: The OECD’s PISA for schools and new modes of heterarchical educational governance. Comparative Education,53(4), 518–537. https://doi.org/10.1080/03050068.2017.1327246.
Lingard, B., Rezai-Rashti, G., Martino, W., & Sellar, S. (2015). Globalizing educational accountabilities. London: Routledge.
Loveless, T. (2012). The 2012 brown center report on American education: How well are American students learning? Retrieved January 20, 2015 from Washington DC http://www.brookings.edu/~/media/Files/rc/reports/2012/0216_brown_education_loveless/0216_brown_education_loveless.pdf.
Loveless, T. (2014). PISA’s China problem continues: A response to Schleicher, Zhang, and Tucker. Retrieved January 20, 2015 from http://www.brookings.edu/research/papers/2014/01/08-shanghai-pisa-loveless.
Lundgren, U. P. (2011). PISA as a political instrument: One history behind the formulating of the PISA programme. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 17–30). New York: Springer.
Meyer, H.-D., et. al. (2014). OECD and Pisa tests are damaging education worldwide—academics. Retrieved Februrary 3, 2015 from https://www.theguardian.com/education/2014/may/06/oecd-pisa-tests-damaging-education-academics.
Meyer, H.-D., & Benavot, A. (2013). PISA, power, and policy: The emergence of global educational governance. Oxford: Oxford University Press.
Morrison, H. (2013). Pisa 2012 major flaw exposed. Retrieved Februrary 3, 2015 from https://paceni.wordpress.com/2013/12/01/pisa-2012-major-flaw-exposed/.
Ninomiya, S. (2016). The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan. Assessment in Education: Principles, Policy & Practice,26, 1–20.
Nyborg, H. (2007). Do recent large-scale cross-national student assessment studies neglect general intelligence g for political reasons? European Journal of Personality,21, 739–741.
OECD. (1999). Measuring student knowledge and skills: A new framework for assessment. Retrieved March 15, 2019 from Paris: http://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/33693997.pdf.
OECD. (2010). OECD programme for international student assessment. Retrieved March 15, 2019 from http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_1_1,00.html.
OECD. (2013). Ready to learn: Students engagement, drive, and self-beliefs. Retrieved March 15, 2019 from Paris: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-III.pdf.
OECD. (2016a). PISA 2015 results (volume I): Excellence and equity in education. Retrieved March 15, 2019 from Paris: http://dx.doi.org/10.1787/9789264266490-en.
OECD. (2016b). PISA 2015: Results in focus. PISA. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf.
OECD. (2017). PISA 2015 results: Students’ well-being. Retrieved March 15, 2019 from Paris http://www.keepeek.com/Digital-Asset-Management/oecd/education/pisa-2015-results-volume-iii_9789264273856-en-.Wk1WGrQ-fOQ#page1.
OECD. (2018a). About PIAAC. Retrieved March 15, 2019 from http://www.oecd.org/skills/piaac/about/.
OECD. (2018b). PISA for development. Retrieved March 15, 2019 from https://www.oecd.org/pisa/pisa-for-development/.
OECD. (2018c). PISA for schools. Retrieved March 15, 2019 from http://www.oecd.org/PISA/pisa-for-schools/.
OECD. (2018d). PISA: About. Retrieved March 15, 2019 from http://www.oecd.org/pisa/aboutpisa/.
OECD. (nd). The international early learning and child well-being study—The study. Retrieved March 15, 2019 from http://www.oecd.org/education/school/the-international-early-learning-and-child-well-being-study-the-study.htm.
OECD/UNESCO. (2003). Literacy skills for the world of tomorrow—Further results from Pisa 2000. PISA. Retrieved October 23, 2017 from http://www.oecd.org/education/school/2960581.pdf.
Pence, A. (2016). Baby PISA: Dangers that can arise when foundations shift. Journal of Childhood Studies,41(3), 54–58.
Pereyra, M. A., Kotthoff, H.-G., & Cowen, R. (2011). PISA under examination. New York: Springer.
Popkewitz, T. (2011). PISA: Numbers, standardizing conduct, and the alchemy of school subjects. In M. A. Pereyra, H.-G. Kotthoff, & R. Cowen (Eds.), Pisa under examination (pp. 31–46). New York: Springer.
Prais, S. J. (2003). Cautions on OECD’s recent educational survey (PISA). Oxford Review of Education,29, 139–163.
Rindermann, H. (2007). The g-factor of international cognitive ability comparisons: The homogeneity of results in PISA, TIMSS, PIRLS and IQ-tests across nations. European Journal of Personality,21, 667–706.
Rutkowski, L., & Rutkowski, D. (2016). A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher,45(4), 252–257.
Sahlberg, P. (2011). Finnish lessons: What can the world learn from educational change in Finland?. New York: Teachers College Press.
Sahlberg, P. (2012). How GERM is infecting schools around the world. Retrieved October 23, 2017 from http://www.washingtonpost.com/blogs/answer-sheet/post/how-germ-is-infecting-schools-around-the-world/2012/06/29/gJQAVELZAW_blog.html-comments.
Sahlberg, P. (2017). FinnishED leadership. Thousand Oaks: Corwin.
Schleicher, A. (2013). Are the Chinese cheating in PISA or are we cheating ourselves? Retrieved Februrary 3, 2015 from http://oecdeducationtoday.blogspot.com/2013/12/are-chinese-cheating-in-pisa-or-are-we.html.
Schleicher, A. (2018). World class: How to build a 21st-century school system. Paris: OECD.
Schuelka, M. J. (2013). Excluding students with disabilities from the culture of achievement: The case of the TIMSS, PIRLS, and PISA. Journal of Education Policy,28(2), 216–230.
Sellar, S., & Lingard, B. (2014). The OECD and the expansion of PISA: New global modes of governance in education. British Educational Research Journal,40(6), 917–936.
Shirley, D. (2017). The new imperatives of educational change: Achievement with integrity. New York: Routledge.
Sjøberg, S. (2012). PISA: Politics, fundamental problems and intriguing results. Recherches en Education,14, 1–21.
Sjøberg, S. (2015a). OECD, PISA, and globalization: The influence of the international assessment regime. In C. H. Tienken & C. A. Mullen (Eds.), Education policy perils (pp. 114–145). London: Routledge.
Sjøberg, S. (2015b). PISA and global educational governance—A critique of the project, its uses and implications. Eurasia Journal of Mathematics, Science and Technology Education,11(1), 111–127.
Solheim, O. J., & Lundetræ, K. (2018). Can test construction account for varying gender differences in international reading achievement tests of children, adolescents and young adults? A study based on Nordic results in PIRLS, PISA and PIAAC. Assessment in Education: Principles, Policy and Practice,25(1), 107–126.
Stewart, W. (2013). Is Pisa fundamentally flawed? Retrieved Februrary 3, 2015 from http://www.tes.co.uk/article.aspx?storycode=6344672.
Stromquist, N. P. (2016). Using regression analysis to predict countries’ economic growth: Illusion and fact in education policy. Real-World Economics Review,76, 65–74.
Tröhler, D. (2013). The OECD and cold war culture: Thinking historically about PISA. In PISA, power, and policy: The emergence of global educational governance (pp. 141–161).
Tucker, M. (Ed.). (2011). Surpassing Shanghai: An agenda for American education built on the world’s leading systems. Boston: Harvard Education Press.
Tucker, M. (2014). Chinese lessons: Shanghai’s rise to the top of the PISA league tables. Retrieved October 23, 2017 from Washington DC. http://www.ncee.org/wp-content/uploads/2013/10/ChineseLessonsWeb.pdf.
Tucker, M. (2016). Asian countries take the U.S. to school. The Atlantic. Retrieved October 23, 2017 from https://www.theatlantic.com/education/archive/2016/02/us-asia-education-differences/471564/.
Uljens, M. (2007). The hidden curriculum of PISA: The promotion of neo-liberal policy by educational assessment. In S. T. Hopmann, G. Brinek, & M. Retzl (Eds.), PISA zufolge PISA—PISA According to PISA (pp. 295–303). Berlin: Lit Verlag.
Urban, M. (2017). We need meaningful, systemic evaluation, not a preschool PISA. Global Education Review,4(2), 18–24.
Wu, M. (2009). A comparison of PISA and TIMSS 2003 achievement results in mathematics. Prospects,39(1), 33.
Zhao, Y. (2009). Catching up or leading the way: American education in the age of globalization. Alexandria: ASCD.
Zhao, Y. (2012). World class learners: Educating creative and entrepreneurial students. Thousand Oaks: Corwin.
Zhao, Y. (2014). Who’s afraid of the big bad dragon: Why China has the best (and Worst) education system in the world. San Francisco: Jossey-Bass.
Zhao, Y. (2015). Lessons that matter: What we should learn from Asian school systems. Retrieved July 20, 2017 from Melbourne. http://www.mitchellinstitute.org.au/reports/lessons-that-matter-what-should-we-learn-from-asias-school-systems/.
Zhao, Y. (2016a). From deficiency to strength: Shifting the mindset about education inequality. Journal of Social Issues,72(4), 716–735.
Zhao, Y. (2016b). Who’s afraid of PISA: The fallacy of international assessments of system performance. In A. Harris & M. S. Jones (Eds.), Leading futures (pp. 7–21). Thousand Oaks: Sage.
Zhao, Y. (2017a). Fatal attraction: Why the west must stop copying China’s flawed education system. New Internationalist,505, 24–25.
Zhao, Y. (2017b). What works can hurt: Side effects in education. Journal of Educational Change,18(1), 1–19.
Zhao, Y. (2018a). The changing context of teaching and implications for teacher education. Peabody Journal of Education,93, 1–14.
Zhao, Y. (2018b). Shifting the education paradigm: Why international borrowing is no longer sufficient for improving education in China. ECNU Review of Education,1(1), 76–106.
Zhao, Y. (2018c). What works may hurt: Side effects in education. New York: Teachers College Press.
Zhao, Y., & Gearin, B. (2016). Squeezed out. In D. Ambrose & R. J. Sternberg (Eds.), Creative intelligence in the 21st century (pp. 121–138). Berlin: Springer.
Zhao, Y., & Wang, Y. (2018). Guarding the past or inventing the future: Education reforms in East Asia. In Y. Zhao & B. Gearin (Eds.), Imagining the future of global education: Dreams and nightmares (pp. 143–159). New York: Routledge.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhao, Y. Two decades of havoc: A synthesis of criticism against PISA. J Educ Change 21, 245–266 (2020). https://doi.org/10.1007/s10833-019-09367-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10833-019-09367-x