Skip to main content
Log in

Scholarly impact assessment: a survey of citation weighting solutions

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Scholarly impact assessment has always been a hot issue. It has played an important role in evaluating researchers, scientific papers, scientific teams, and institutions within science of science. Scholarly impact assessment is also used to address fundamental issues, such as reward evaluation, funding allocation, promotion and recruitment decision. Scholars generally agree that it is more reasonable to use weighted citations to assess the scholarly impact. Although a great number of researchers use weighted citations to access the scholarly impact, there is a lack of a systematic summary of citation weighting methods. To fill the gap, this paper summarizes the existing classical indicators and weighting methods used in measuring scholarly impact from the perspectives of articles, authors and journals. We also summarize the focus of the indicators involved in this paper and the weighting factors that involved in the weighting methods. Finally, we discuss the open issues to try to discover the hidden trends of citation weighting. Through this paper, we can not only have a clearer understanding of the weighting methods in the scholarly impact assessment, but also think more deeply about the weighting factors to be explored.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abujbara, A., Ezra, J., & Radev, D. R. (2013). Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies (pp. 596–606).

  • Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2010). hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices. Scientometrics, 82(2), 391–400.

    Google Scholar 

  • Anderson, T. R., Hankin, R. K. S., & Killworth, P. D. (2008). Beyond the Durfee square: Enhancing the h-index to score total publication output. Scientometrics, 76(3), 577–588.

    Google Scholar 

  • Bai, X., Feng, X., Lee, I., Zhang, J., & Ning, Z. (2016a). Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS ONE, 11(9), 1–15.

    Google Scholar 

  • Bai, X., Hou, J., Du, H., Kong, X., & Xia, F. (2017a). Evaluating the impact of articles with geographical distances between institutions. In International conference on world wide web companion (pp. 1243–1244).

  • Bai, X., Liu, H., Zhang, F., Ning, Z., Kong, X., Lee, I., et al. (2017b). An overview on evaluating and predicting scholarly article impact. Information, 8(3), 73–86.

    Google Scholar 

  • Bai, X., Zhang, J., Cui, H., Ning, Z., & Xia, F. (2016b). PNCOIRank: Evaluating the impact of scholarly articles with positive and negative citations. In International conference companion on world wide web (pp. 9–10).

  • Batista, P. D., Campiteli, M. G., & Kinouchi, O. (2006). Is it possible to compare researchers with different scientific interests. Scientometrics, 68(1), 179–189.

    Google Scholar 

  • Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College and Research Libraries News, 68(5), 314–316.

    Google Scholar 

  • Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.

    Google Scholar 

  • Bollen, J., Sompel, H. V. D., Hagberg, A., & Chute, R. (2009). A principal component analysis of 39 scientific impact measures. PLoS ONE, 4(6), e6022–e6032.

    Google Scholar 

  • Bornmann, L., & Marx, W. (2016). The journal impact factor and alternative metrics: A variety of bibliometric measures has been developed to supplant the impact factor to better assess the impact of individual research papers. EMBO Reports, 17(8), 1094–1097.

    Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169–173.

    Google Scholar 

  • Brown, R. J. C. (2009). A simple method for excluding self-citation from the h-index: The b-index. Online Information Review, 33(6), 1129–1136.

    Google Scholar 

  • Callaway, E. (2016). Publishing elite turns against impact factor. Nature, 535(7611), 210–211.

    Google Scholar 

  • Cantín, M., Muñoz, M., & Roa, I. (2015). Comparison between impact factor, eigenfactor score, and scimago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183–1188.

    Google Scholar 

  • Casadevall, A., Bertuzzi, S., Buchmeier, M. J., Davis, R. J., Drake, H., Fang, F. C., et al. (2016). Asm journals eliminate impact factor information from journal websites. Applied and Environmental Microbiology, 1(4), 2407–2408.

    Google Scholar 

  • Casadevall, A., & Fang, F. C. (2014). Causes for the persistence of impact factor mania. MBio, 5(2), e00,064–14.

    Google Scholar 

  • Cavalcanti, D. C., Prudncio, R. B. C., Pradhan, S. S., Shah, J. Y., & Pietrobon, R. S. (2011). Good to be bad? Distinguishing between positive and negative citations in scientific impact. In IEEE international conference on TOOLS with artificial intelligence (pp. 156–162).

  • Costas, R., & Bordons, M. O. (2007). The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics, 1(3), 193–203.

    Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2015). Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019.

    Google Scholar 

  • Davis, P. M. (2008). Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts? Journal of the American Society for Information Science and Technology, 59(13), 2186–2188.

    Google Scholar 

  • Dellavalle, R. P., Schilling, L. M., Rodriguez, M. A., van de Sompel, H., & Bollen, J. (2007). Refining dermatology journal impact factors using PageRank. Journal of the American Academy of Dermatology, 57(1), 116–119.

    Google Scholar 

  • Ding, Y., & Yan, E. (2010). Measuring scholarly impact in heterogeneous networks. Proceedings of the Association for Information Science and Technology, 47(1), 1–7.

    Google Scholar 

  • Dong, P., Loh, M., & Mondry, A. (2005). The “impact factor” revisited. Biomedical digital libraries, 2(1), 1–8.

    Google Scholar 

  • Eck, N. J. V., & Waltman, L. (2008). Generalizing the h- and g-indices. Journal of Informetrics, 2(4), 263–271.

    Google Scholar 

  • Egghe, L. (2006). An improvement of the H-index: The G-index. ISSI, 2(1), 8–9.

    MathSciNet  Google Scholar 

  • Elliott, D. B. (2014). The impact factor: A useful indicator of journal quality or fatally flawed? Ophthalmic and Physiological Optics, 34(1), 4–7.

    Google Scholar 

  • Fersht, A. R. (2009). The most influential journals: Impact factor and eigenfactor. Proceedings of the National Academy of Sciences of the United States of America, 106(17), 6883–6884.

    Google Scholar 

  • Fiala, D. (2012). Time-aware pagerank for bibliographic networks. Journal of Informetrics, 6(3), 370–388.

    Google Scholar 

  • Fiala, D. (2014). Current index: A proposal for a dynamic rating system for researchers. Journal of the American Society for Information Science and Technology, 65(4), 850–855.

    Google Scholar 

  • Fiala, D., Rousselot, F., & Jezek, K. (2008). Pagerank for bibliographic networks. Scientometrics, 76(1), 135–158.

    Google Scholar 

  • Franceschet, M. (2010). The difference between popularity and prestige in the sciences and in the social sciences: A bibliometric analysis. Journal of Informetrics, 4(1), 55–63.

    Google Scholar 

  • Franceschet, M., & Colavizza, G. (2018). Timerank: A dynamic approach to rate scholars using citations. Journal of Informetrics, 11(4), 1128–1141.

    Google Scholar 

  • Fujimagari, H., & Fujita, K. (2015). Detecting research fronts using neural network model for weighted citation network analysis. Journal of Information Processing, 23(6), 753–758.

    Google Scholar 

  • Fujita, K., Kajikawa, Y., Mori, J., & Sakata, I. (2014). Detecting research fronts using different types of weighted citation networks. Journal of Engineering and Technology Management, 32, 129–146.

    Google Scholar 

  • Galam, S. (2011). Tailor based allocations for multiple authorship: A fractional gh-index. Scientometrics, 89(1), 365–379.

    Google Scholar 

  • Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA, 295(1), 90–93.

    Google Scholar 

  • Gómez-Núñez, A. J., Batagelj, V., Vargas-Quesada, B., Moya-Anegón, F., & Chinchilla-Rodríguez, Z. (2014). Optimizing SCImago journal & country rank classification by community detection. Journal of Informetrics, 8(2), 369–383.

    Google Scholar 

  • Gonzalezpereira, B., Guerrerobote, V. P., & De Moyaanegon, F. (2010). A new approach to the metric of journals scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.

    Google Scholar 

  • Guerrero-Bote, V. P., & Moya-Anegón, F. (2012). A further step forward in measuring journals scientific prestige: The SJR2 indicator. Journal of Informetrics, 6(4), 674–688.

    Google Scholar 

  • Habibzadeh, F., & Yadollahie, M. (2008). Journal weighted impact factor: A proposal. Journal of Informetrics, 2(2), 164–172.

    Google Scholar 

  • Hagen, N. T. (2010). Harmonic publication and citation counting: Sharing authorship credit equitably not equally, geometrically or arithmetically. Scientometrics, 84(3), 785–793.

    Google Scholar 

  • Hall, C. M., & Page, S. J. (2015). Following the impact factor: Utilitarianism or academic compliance? Tourism Management, 51, 309–312.

    Google Scholar 

  • Harzing, A. W., & van der Wal, R. (2009). A google scholar h-index for journals: An alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41–46.

    Google Scholar 

  • Ha, T. C., Tan, S. B., & Soo, K. C. (2006). The journal impact factor: Too much of an impact? Annals-Academy of Medicine Singapore, 35(12), 911–916.

    Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.

    MATH  Google Scholar 

  • Jha, R., Jbara, A. A., Qazvinian, V., & Radev, D. R. (2017). NLP-driven citation analysis for scientometrics. Natural Language Engineering, 23(1), 93–130.

    Google Scholar 

  • Jin, B. (2006). H-Index: An evaluation indicator proposed by scientist. Science Focus, 1(1), 8–9.

    Google Scholar 

  • Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.

    Google Scholar 

  • Kianifar, H., Sadeghi, R., & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and scimago journal rank indicator of pediatric neurology journals. Acta Informatica Medica, 22(2), 103–106.

    Google Scholar 

  • Kong, X., Mao, M., Wang, W., Liu, J., & Xu, B. (2018). Voprec: Vector representation learning of papers with text information and structural identity for recommendation. IEEE Transactions on Emerging Topics in Computing, PP(99), 1.

    Google Scholar 

  • Kosmulski, M. (2006). A new Hirsch-type index saves time and works equally well as the original h-index. ISSI Newsletter, 2(3), 4–6.

    Google Scholar 

  • Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62(2), 217–229.

    Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646.

    Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2011). Remaining problems with the “New Crown Indicator” (MNCS) of the CWTS. Journal of Informetrics, 5(1), 224–225.

    Google Scholar 

  • Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of Informetrics, 1(2), 145–154.

    Google Scholar 

  • Luo, D., Gong, C., Hu, R., Duan, L., & Ma, S. (2016). Ensemble enabled weighted PageRank. arXiv preprint arXiv:160405462.

  • Mathur, V. P., Sharma, A., et al. (2009). Impact factor and other standardized measures of journal citation: A perspective. Indian Journal of Dental Research, 20(1), 81–85.

    Google Scholar 

  • Mingers, J., & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323–337.

    MathSciNet  MATH  Google Scholar 

  • Moed, H. F. (2010a). CWTS crown indicator measures citation impact of a research group’s publication oeuvre. Journal of Informetrics, 4(3), 436–438.

    Google Scholar 

  • Moed, H. F. (2010b). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.

    Google Scholar 

  • Moustafa, K. (2015). The disaster of the impact factor. Science and Engineering Ethics, 21(1), 139–142.

    MathSciNet  Google Scholar 

  • Nykl, M., Campr, M., & Jezek, K. (2015). Author ranking based on personalized pagerank. Journal of Informetrics, 9(4), 777–799.

    Google Scholar 

  • Nykl, M., Jezek, K., Fiala, D., & Dostal, M. (2014). Pagerank variants in the evaluation of citation networks. Journal of Informetrics, 8(3), 683–692.

    Google Scholar 

  • Opthof, T., & Leydesdorff, L. (2010). Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance. Journal of Informetrics, 4(3), 423–430.

    Google Scholar 

  • Ordunamalea, E., & Lopezcozar, E. D. (2014). Google scholar metrics evolution: An analysis according to languages. Scientometrics, 98(3), 2353–2367.

    Google Scholar 

  • Pajić, D. (2015). On the stability of citation-based journal rankings. Journal of Informetrics, 9(4), 990–1006.

    Google Scholar 

  • Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact on the social web. First Monday, 15(7), 1.

    Google Scholar 

  • Ramin, S., & Shirazi, A. S. (2012). Comparison between Impact factor, SCImago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132–136.

    Google Scholar 

  • Roldan-Valadez, E., Orbe-Arteaga, U., & Rios, C. (2018). Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: A linear mixed design model analysis of radiology, nuclear medicine and medical imaging journals. La Radiologia Medica, 123, 1–11.

    Google Scholar 

  • Rousseau, R. (2008). Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics, 2(4), 335–340.

    Google Scholar 

  • Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics, 9(1), 102–117.

    Google Scholar 

  • Saad, G. (2006). Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics, 69(1), 117–120.

    MathSciNet  Google Scholar 

  • Sayyadi, H., & Getoor, L. (2009). Futurerank: Ranking scientific articles by predicting their future PageRank. In Siam international conference on data mining (pp. 533–544).

  • Schreiber, M. (2008). EDITORIAL: To share the fame in a fair way, hm modifies h for multi-authored manuscripts. New Journal of Physics, 10(4), 1131–1137.

    MathSciNet  Google Scholar 

  • Schreiber, M. (2010). The influence of self-citation corrections and the fractionalised counting of multi-authored manuscripts on the Hirsch index. Annalen Der Physik, 18(9), 607–621.

    Google Scholar 

  • Schubert, A., Glänzel, W., & Thijs, B. (2006). The weight of author self-citations. A fractional approach to self-citation counting. Scientometrics, 67(3), 503–514.

    Google Scholar 

  • Sekercioglu, C. H. (2008). Quantifying coauthor contributions. Science, 322(5900), 371–371.

    Google Scholar 

  • Smolinsky, L. (2016). Expected number of citations and the crown indicator. Journal of Informetrics, 10(1), 43–47.

    Google Scholar 

  • Su, C., Pan, Y. T., Zhen, Y. N., Ma, Z., Yuan, J. P., Guo, H., et al. (2011). PrestigeRank: A new evaluation method for papers and journals. Journal of Informetrics, 5(1), 1–13.

    Google Scholar 

  • Vaccario, G., Medo, M., Wider, N., & Mariani, M. S. (2017). Quantifying and suppressing ranking bias in a large citation network. Journal of Informetrics, 11(3), 766–782.

    Google Scholar 

  • Valenzuela, M., Ha, V., & Etzioni, O. (2015). Identifying meaningful citations. In Association for the advancement of artificial intelligence WS-15-13 (pp. 21–26).

  • Van, H. B. A., Phelps, J., Barnes, M., & Suk, W. A. (2000). Evaluating scientific impact. Environmental Health Perspectives, 108(9), 392–393.

    Google Scholar 

  • Walker, D., Xie, H., Yan, K. K., & Maslov, S. (2007). Ranking scientific publications using a model of network traffic. Journal of Statistical Mechanics: Theory and Experiment, 6(6), 1–5.

    Google Scholar 

  • Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.

    Google Scholar 

  • Waltman, L., & van Eck, N. J. (2008). Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie. Journal of Informetrics, 2(4), 369–372.

    Google Scholar 

  • Waltman, L., & Van Eck, N. J. (2010). The relation between eigenfactor, audience factor, and influence weight. Journal of the Association for Information Science and Technology, 61(7), 1476–1486.

    Google Scholar 

  • Waltman, L., & van Eck, N. J. (2012). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology, 63(12), 2378–2392.

    Google Scholar 

  • Waltman, L., & van Eck, N. J. (2013a). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.

    Google Scholar 

  • Waltman, L., & Van Eck, N. J. (2013b). A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 7(4), 833–849.

    Google Scholar 

  • Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285.

    Google Scholar 

  • Waltman, L., Van Eck, N. J., Van Leeuwen, T. N., Visser, M. S., & Van Raan, A. F. J. (2011a). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.

    Google Scholar 

  • Waltman, L., Yan, E., & van Eck, N. J. (2011b). A recursive field-normalized bibliometric performance indicator: An application to the field of library and information science. Scientometrics, 89(1), 301–314.

    Google Scholar 

  • Wang, l. (2013). A new h-type index A+ index. Information Magazine, 1, 55–58.

    Google Scholar 

  • Wang, Y., Tong, Y., & Zeng, M. (2013). Ranking scientific articles by exploiting citations, authors, journals, and time information. In AAAI conference on artificial intelligence (pp. 933–939).

  • Wang, S., Xie, S., Zhang, X., Li, Z., Yu, P. S., & Shu, X. (2016). Future influence ranking of scientific literature. Computer Science, 9, 1–9.

    Google Scholar 

  • Wan, X., & Liu, F. (2014). Are all literature citations equally important? Automatic citation strength estimation and its applications. Journal of the Association for Information Science & Technology, 65(9), 1929–1938.

    Google Scholar 

  • Wesley-Smith, I., Bergstrom, C. T., & West, J. D. (2016). Static ranking of scholarly papers using article-level eigenfactor (ALEF). arXiv preprint arXiv:160608534.

  • West, J. D., Bergstrom, T. C., & Bergstrom, C. T. (2010). The eigenfactor metrics™: A network approach to assessing scholarly journals. College and Research Libraries, 71(3), 236–244.

    Google Scholar 

  • West, J. D., Jensen, M. C., Dandrea, R. J., Gordon, G. J., & Bergstrom, C. T. (2013). Author-level eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the Association for Information Science and Technology, 64(4), 787–801.

    Google Scholar 

  • Wu, Q. (2010). The w-index: A measure to assess scientific impact by focusing on widely cited papers. Journal of the American Society for Information Science and Technology, 61(3), 609–614.

    Google Scholar 

  • Yan, E., & Ding, Y. (2011). Discovering author impact: A PageRank perspective. Information Processing and Management, 47(1), 125–134.

    Google Scholar 

  • Zhang, C. T. (2009a). A proposal for calculating weighted citations based on author rank. Embo Reports, 10(5), 416–417.

    Google Scholar 

  • Zhang, C. T. (2009b). The e-index, complementing the h-index for excess citations. PLoS ONE, 4(5), e5429–e5432.

    Google Scholar 

  • Zhou, D., Orshanskiy, S. A., Zha, H., & Giles, C. L. (2007). Co-ranking authors and documents in a heterogeneous network. In IEEE international conference on data mining (pp. 739–744).

  • Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015). Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66(2), 408–427.

    Google Scholar 

  • Zitt, M. (2010). Citing-side normalization of journal impact: A robust variant of the Audience Factor. Journal of Informetrics, 4(3), 392–406.

    Google Scholar 

  • Zitt, M., Ramanana-Rahary, S., & Bassecoulard, E. (2005). Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation. Scientometrics, 63(2), 373–401.

    Google Scholar 

  • Zitt, M., & Small, H. (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59(11), 1856–1860.

    Google Scholar 

  • Zyczkowski, K. (2010). Citation graph, weighted impact factors and performance indices. Scientometrics, 85(1), 301–315.

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Natural Science Foundation of China (61872054), the Fund for Promoting the Reform of Higher Education by Using Big Data Technology, Energizing Teachers and Students to Explore the Future (2017A01002), and the Fundamental Research Funds for the Central Universities (DUT18JC09).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xiangjie Kong.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cai, L., Tian, J., Liu, J. et al. Scholarly impact assessment: a survey of citation weighting solutions. Scientometrics 118, 453–478 (2019). https://doi.org/10.1007/s11192-018-2973-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-018-2973-6

Keywords

Navigation