Skip to main content
SpringerLink
Log in

What do global university rankings really measure? The search for the X factor and the X entity

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. One exception is the study by Li et al. (2011a), which focuses on the effects of country and language.

  2. This principle is based on the functioning of the memory of individuals and the social processes that occur in science.

  3. As the sample is reduced, the variance of the variables increases and the t statistics associated with its parameters decrease. This problem occurs in Model 3 (n = 181) and particularly in Model 2 (n = 100).

References

  • Abramo, G., D’Angelo, C. A., & Caprasecca, A. (2009). Allocative efficiency in public research funding: Can bibliometrics help? Research Policy, 38(1), 206–215.

    Article  Google Scholar 

  • Adler, N. J., & Harzing, A. W. (2009). When knowledge wins: transcending the sense and nonsense of academic rankings. Academy of Management Learning & Education, 8(1), 72–95.

    Article  Google Scholar 

  • Altbach, P. (2006). The dilemmas of ranking. International Higher Education, 42, 1–2.

    Google Scholar 

  • Armstrong, J. S., & Sperry, T. (1994). Business school prestige-research versus teaching. Interfaces, 24(2), 13–43.

    Article  Google Scholar 

  • Bastedo, M. N., & Bowman, N. A. (2010). The U.S. News and World Report college rankings: Modeling institutional effects on organizational reputation. American Journal of Education, 116, 163–184.

    Article  Google Scholar 

  • Bentley, R., & Blackburn, R. (1990). Changes in academic research performance over time: A study of institutional accumulative advantage. Research in Higher Education, 31(4), 327–353.

    Article  Google Scholar 

  • Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? An MCDM view. Scientometrics, 84(1), 237–263.

    Article  Google Scholar 

  • Bornmann, L. (2011). Peer Review and Bibliometric: Potentials and Problems. In J. C Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  • Bowman, N. A., & Bastedo, M. N. (2011). Anchoring effects in world university rankings: exploring biases in reputation scores. High Education, 61, 431–444.

    Article  Google Scholar 

  • Brooks, R. (2005). Measuring University Quality. The Review of Higher Education, 29(1), 1–21.

    Article  MathSciNet  Google Scholar 

  • Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365.

    Article  Google Scholar 

  • Carpenter, G. S., & Nakamoto, K. (1989). Consumer Preference Formation and Pioneering Advantage. Journal of Marketing Research, 26(3), 285–298.

    Article  Google Scholar 

  • Chen, K., & Liao, P. (2012). A comparative study on world university rankings: a bibliometric survey. Scientometrics, 92, 89–103.

    Article  Google Scholar 

  • Cole, S., & Cole, J. R. (1967). Scientific output and recognition: A study in the operation of the reward system in science. American Sociological Review, 32(3), 377–390.

    Article  Google Scholar 

  • Crespi, G. (2007). The UK knowledge production function. In A. Bonaccorsi & C. Daraio (Eds.), Universities and strategic knowledge creation: Specialization and performance in Europe. Cheltenham, UK: Edward Elgar Publishing.

    Google Scholar 

  • Devinney, T., Dowling, G. R., & Perm-Ajchariyawong, N. (2006). The business schools rankings game. AACSB. Retrieved December 18, 2006, from www.aacsb.edu/resource_centers/Rankings.

  • Dey, E. L., Milem, J. F., & Berger, J. B. (1997). Changing patterns of publication productivity: Accumulative advantage or institutional isomorphism? Sociology of Education, 70, 308–323.

    Article  Google Scholar 

  • Dichev, L. D. (1999). How good are business schools rankings? Journal of Business, 72(2), 201–213.

    Article  Google Scholar 

  • Dill, D. D., & Soo, M. (2005). Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems. Higher Education, 49, 495–533.

    Article  Google Scholar 

  • Docampo, D. (2011). On using the Shanghai ranking to assess the research performance of university systems. Scientometrics, 86, 77–92.

    Article  Google Scholar 

  • Dyck, I. J. A. & Zingales, L. (2002). The Corporate Governance Role of the Media. CRSP Working Paper, No. 543. Retrieved October 19, 2005, from http://ssrn.com/abstract=335602.

  • Fombrun, C., & Shanley, M. (1990). What’s in a name? Reputation building and corporate strategy. Academy of Management Journal, 33(2), 233–258.

    Article  Google Scholar 

  • Gioia, D. A., & Corley, K. G. (2002). Being good versus looking good: Business schools rankings and the circean transformation from substance to image. Academy of Management Learning and Education, 1(1), 107–120.

    Article  Google Scholar 

  • Gómez, I., Bordons, M., Fernández, M. T., & Morillo, F. (2009). Structure and research performance of Spanish universities. Scientometrics, 79(1), 131–146.

    Article  Google Scholar 

  • Gómez-Sancho, J. M. & Pérez-Esparrells, C. (2012). International Higher Education rankings at a glance: How to valorize the research in Social Sciences and Cultural Studies. In A. López-Varela (Ed.), Social Sciences and Humanities. Applications and Theories. Book 1 (pp. 355–374). Rijeka: InTech.

  • Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate data analysis. NJ: Prentice Hall.

    Google Scholar 

  • Hazelkorn, E. (2009). Impact of Global Rankings on Higher Education Research and the Production of Knowledge, UNESCO Forum on Higher Education, Research and Knowledge, Occasional Paper No. 15.

  • Huang, M. H. (2012). Opening the black box of QS World University Rankings. Research Evaluation, 21, 71–78.

    Article  Google Scholar 

  • Jeon, Y., Miller, S. M., & Ray, S. C. (2003). MBA program reputation: Objective rankings for students, employers, and program administrators, Economics Working Paper. Paper 200328. Retrieved October 19, 2012, from http://digitalcommons.uconn.edu/econ_wpapers/200328..

  • Johnson, D. (1997). Getting noticed in economics: The determinants of academic citations. American Economist, 41(1), 43–52.

    Google Scholar 

  • Katz, J.S. (1999). The self-similar science system. Research Policy, 28(5), 501–517.

    Google Scholar 

  • Keith, B. (2001). Organizational contexts and university performance outcomes: The Limited Role of Purposive Action in the Management of Institutional Status. Research in Higher Education, 42(5), 493–516.

    Article  Google Scholar 

  • Kivinen, O., Hedman, J., & Kaipainen, P. (2013). Productivity analysis of research in natural sciences, technology and clinical medicine: An input–output model applied in comparison of Top 300 ranked universities of 4 North European and 4 East Asian countries. Scientometrics. 94(2), 683–699.

    Google Scholar 

  • Li, M., Shankar, S., & Tang, K. K. (2011a). Why does the USA dominate university league tables? Studies in Higher Education, 36(8), 923–937.

    Article  Google Scholar 

  • Li, M., Shankar, S., & Tang, K. K. (2011b). Catching up with Harvard: Results from regression analysis of world universities league tables. Cambridge Journal of Education, 41(2), 121–137.

    Article  Google Scholar 

  • Lieberman, M. B., & Montgomery, D. B. (1988). First-mover advantages. Strategic Management Journal, 9, 41–58.

    Article  Google Scholar 

  • Liu, N. C., & Li Liu, Y. C. (2005). Academic ranking of world universities using scientometrics. A comment to the Fatal Attraction. Scientometrics, 64(1), 101–109.

    Article  Google Scholar 

  • Locke, W., Verbik, L., Richardson, J., & King, R. (2008). Counting what is measured or measuring what counts? League tables and their impact on higher education institutions in England. Bristol, UK: Higher Education Funding Council for England.

    Google Scholar 

  • Longden, B. (2011). Ranking Indicators and Weights. In J. C. Shin, R. K. Toutkoushian, & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  • Marginson, S. (2007). Global university rankings: implications in general and for Australia. Journal of Higher Education Policy and Management, 29(2), 131–142.

    Article  Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159, 56–63.

    Article  Google Scholar 

  • Morgeson, F. P., & Nahrgang, J. D. (2008). Same as it ever was: Recognizing stability in the Business Week rankings. Academy of Management Learning & Education, 7, 26–41.

    Article  Google Scholar 

  • O’Connell, C. (2012). Research discourses surrounding global university rankings: exploring the relationship with policy and practice recommendations. Higher Education. doi: 10.1007/s10734-012-9572-x. Published on line: 30 September 2012.

  • Olivares, M., & Wetzel, H. (2011). Competing in the higher education market: Empirical evidence for economies of scale and scope in German higher education institutions (No. 223). University of Lüneburg Working Paper Series in Economics.

  • Ordorika, I., & Rodríguez, R. (2010). TIMES ranking in the reputation market/El ranking Times en el mercado del prestigio universitario. Perfiles Educativos, XXXII, 129, 8–29.

    Google Scholar 

  • Oswald, A. J. (2008). Can we test for bias in scientific peer-review? IZA DP No. 3665, Forschungsinstitut zur Zukunft der Arbeit Institute for the Study of Labor.

  • Patterson, W. (1993). First-mover advantage: the opportunity curve. Journal of Management Studies, 30(5), 759–777.

    Article  Google Scholar 

  • Power, M. (1997). The audit society. Oxford: Oxford University Press.

    Google Scholar 

  • Rauhvargers, A. (2011). Report on rankings 2011: Global university rankings and their impact. Brussels: European University Association.

    Google Scholar 

  • Rigney, D. (2010). The Matthew effect: How advantage begets further advantage. Columbia University Press.

  • Safón, V. (2012). Can the reputation of an established business school change? Management in Education, 26(4), 169–180.

    Google Scholar 

  • Saisana, M., & D’Hombres, B. (2008). Higher Education Rankings: Robustness Issues and Critical Assessment, Report 23487, European Commission, JRC-IPSC, Italy.

  • Saisana, M., d’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research Policy, 40(1), 165–177.

    Article  Google Scholar 

  • Salmi, J., & Saroyan, A. (2007). League tables as policy instruments: Uses and misuses. Higher Education Management and Policy, 19(2), 31–68.

    Article  Google Scholar 

  • Shin, J. C. (2011). Organizational Effectiveness and University Rankings, In J. C. Shin, R. K. Toutkoushian & U. Teichler, (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  • Shin, J. C., & Toutkoushian, R. K. (2011). The past, present, and future of University Rankings. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3), Dordrecht: Springer Science.

  • Smyth, D. J. (1999). The determinants of the reputations of economics departments: Pages published, citations and the Andy Rooney effect. American Economist, 43(2), 49–58.

    Google Scholar 

  • Stella, A. & Woodhouse, D. (2006). Ranking of Higher Education Institutions. AUQA Occasional Publications Number 6. Australian Universities Quality Agency.

  • Suarez, F., & Lanzolla, G. (2007). The role of environmental dynamics in building a first mover advantage theory. Academy of Management Review, 32(2), 377–392.

    Article  Google Scholar 

  • Taylor, P., & Braddock, R. (2007). International university ranking systems and the idea of university excellence. Journal of Higher Education Policy and Management, 29(3), 245–260.

    Article  Google Scholar 

  • Teichler, U. (2011a). Social contexts and systemic consequence of University Rankings: A meta-analysis of the ranking literature. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  • Teichler, U. (2011b). The future of university rankings. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective (Vol. 3). Dordrecht: Springer Science.

  • Toutkoushian, R. K., & Webber, K. (2011). Measuring the Research Performance of Postsecondary Institutions. In J. C. Shin, R. K. Toutkoushian & U. Teichler (Eds.), University Rankings, The Changing Academy: The Changing Academic Profession in International Comparative Perspective(Vol. 3). Dordrecht: Springer Science.

  • Trank, C. Q., & Rynes, S. L. (2003). Who moved our cheese? Reclaiming professionalism in business education. Academy of Management Learning and Education, 2(2), 189–205.

    Article  Google Scholar 

  • Usher, A., & Savino, M. (2006). A world of difference: A global survey of university league tables. Toronto, ON: Educational Policy Institute.

    Google Scholar 

  • Usher, A., & Savino, M. (2007). A Global Survey of Rankings and League Tables. In Institute for Higher Education Policy (Ed.), College and University Ranking Systems: Global Perspectives and American Challenges (pp. 23–35). www.ihep.org.

  • Van Dalen, H. P., & Henkens, K. N. (2005). Signals in science-On the importance of signaling in gaining attention in science. Scientometrics, 64(2), 209–233.

    Google Scholar 

  • Van Dyke, N. (2005). Twenty years of university reports cards. Higher Education in Europe, 30(2), 103–124.

    Article  Google Scholar 

  • Van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & Van Raan, A. F. J. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346.

    Article  Google Scholar 

  • Van Raan, A. F. J. (2005a). Fatal attraction: Ranking of universities by bibliometric methods. Scientometrics, 62, 133–145.

    Article  Google Scholar 

  • Van Raan, A. F. J. (2005b). Reply to the comments of Liu et al. Scientometrics, 64(1), 111–112.

    Article  Google Scholar 

  • Van Raan, A. F. (2006). Statistical properties of bibliometric indicators: Research group indicator distributions and correlations. Journal of the American Society for Information Science and Technology, 57(3), 408–430.

    Google Scholar 

  • Van Raan, A. F. J. (2007). Challenges in the ranking of universities. In J. Sadlak & L. N. Cai (Eds.), The world-class university and ranking: Aiming beyond status. Shanghai: UNESCO-CEPES, Shanghai Jiao Tong University, Cluj University Press.

    Google Scholar 

  • Van Vught, F., & Ziegele, F. (Eds). (2011). U-multirank. Design and Testing the Feasibility of a Multidimensional Global University Ranking. Final Report. Consortium for Higher Education and Research Performance assessment. CHERPA-Network.

  • Von Tunzelmann, N., Ranga, M., Martin, B. & Geuna, AL (2003). The effects of size on research performance: A SPRU review. Science and Technology Policy Research (SPRU), The Freeman Centre, University of Sussex Falmer, Brighton, BN1 9QE, UK.

  • Williams, R. (2007). Broadening the Criteria: Lessons from the Australian Rankings. In J. Sadlak & L. N. Cai (Eds.), The world-class university and ranking: Aiming beyond status. Shanghai: UNESCO-CEPES, Shanghai Jiao Tong University, Cluj University Press.

    Google Scholar 

  • Zitt, M., & Filliatreau, G. (2007). Big is (made) beautiful: Some comments about the Shanghai ranking of world-class universities. In J. Sadlak & N.C. Liu (Eds.), The world-class university and ranking: Aiming beyond status (pp. 141–160). Shanghai: UNESCO-CEPES, Institute of Higher Education, Shanghai Jiao Tong University, and Cluj University Press.

Download references

Author information

Authors and Affiliations

  1. Department of Management, Faculty of Economics, University of Valencia, Avenida de los Naranjos, s/n, 46022, Valencia, Spain

    Vicente Safón

Authors
  1. Vicente Safón
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vicente Safón.

Appendices

Appendix 1. Variables studied as possible X factors and their measures

University in U.S. 1 if the university is in the U.S., 0 if it is not.

English-speaking country (excluded U.S.). 1 if the official national language is English, 0 if it is not or the university is in the U.S.

Size (students). Variable measured on a 4-point scale, where 1 = small, 2 = medium-sized, 3 = large y 4 = very large. Taken from the Quacquarelli Symonds database. See description at www.topuniversities.com.

Age. Variable measured on a 5-point scale: 1-New (<10 years), 2-Young (<25 years), 3-Established (<50 years), 4-Mature (<100 years), 5-Historic (>100 years). Variable taken from the Quacquarelli Symonds database.

Annual income. Variable taken from the ARWU database. Coded in the form of a ranking, so that a smaller number indicates larger income.

Scope (number of disciplines). Variable measured on a 4-point scale, ranging from 1 = very few subjects offered to 4 = a wide variety of subjects. Variable taken from the Quacquarelli Symonds database.

Activity in hard sciences. This variable measures whether the university carries out activities in hard sciences. It is measured using a dichotomous variable with a value of 1 if the university published papers on Nature and Science between 2007 and 2011, and 0 if not (data taken from the ARWU database).

Orientation towards research. Variable taken from the Quacquarelli Symonds database. Coded in the form of a score from faculty to student ratio. See an explanation at: http://www.iu.qs.com/university-rankings/rankings-indicators/methodology-faculty-student. Student staff ratio is commonly used as a measure of teaching quality (Longden 2011). According to literature, a lower academic staff to student ratio indicates that the university is more teaching oriented than research oriented (Li et al. 2011b).

Reputation. Variable ranging from 0 = no reputation to 100 = the highest reputation among academics. Taken from the “Quacquarelli Symonds Global Academic Survey”, using a sample of 34,000 academics from a broad, balanced variety of subjects and countries. See an explanation at: http://www.iu.qs.com/university-rankings/world-university-rankings/2011-academic-survey-responses/.

Appendix 2. Correlations between the X factors and the affected THE indicators

See Table 5.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Safón, V. What do global university rankings really measure? The search for the X factor and the X entity. Scientometrics 97, 223–244 (2013). https://doi.org/10.1007/s11192-013-0986-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-013-0986-8

Keywords

Mathematics Subject Classification

JEL Classification

Search

Navigation