Abstract
The evidence base in environmental sciences is increasing steadily. Environmental researchers have been challenged to handle massive volumes of data to support more comprehensive studies, assess the current status of science, and move research towards future progress. Bibliometrics can provide important insights into the research directions by providing summarized information for several end users. Here, we present an in-depth discussion on the use of bibliometric indicators to evaluate research outputs through four case studies comprising disciplines in environmental sciences. We discuss four big challenges researchers may face when conducting bibliometric reviews and how to deal with them. We also address some primary questions researchers may answer with bibliometric mapping, drawing lessons from the case studies. Lastly, we clarify some misuses of review concepts and suggest methodological principles of systematic reviews and maps to improve the overall quality of bibliometric studies.
Similar content being viewed by others
Data availability
All data generated or analyzed during this study are included in this published article.
Abbreviations
- Systematic review:
-
“rigorous summary of research literature on a given topic that has been conducted using structured, transparent, and reproducible methods. The term could be used to indicate any review that uses approaches involved in a systematic review (i.e., systematic review approach)” (Nakagawa et al. 2018)
- Systematic map:
-
“literature summary conducted using strict, systematic standards. It summarizes the characteristics of studies from a broad research field in a database, figure, or graph. Can identify knowledge gaps and knowledge clusters”. (Nakagawa et al. 2018)
References
Abbasi A, Altmann J, Hossain L (2011) Identifying the effects of co-authorship networks on the performance of scholars: a correlation and regression analysis of performance measures and social network analysis measures. J Inform 5(4):594–607. https://doi.org/10.1016/j.joi.2011.05.007
Andrade DF, Romanelli JP, Pereira-Filho ER (2019) Past and emerging topics related to electronic waste management: top countries, trends, and perspectives. Environ Sci Pollut Res 26(17):17135–17151. https://doi.org/10.1007/s11356-019-05089-y
Ashley P, Boyd WE (2006) Quantitative and qualitative approaches to research in environmental management. Australasian J Environ Manag 13(2):70–78. https://doi.org/10.1080/14486563.2006.10648674
Avenell A, Handoll HH, Grant AM (2001) Lessons for search strategies from a systematic review, in The Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr 73(3):505–510. https://doi.org/10.1093/ajcn/73.3.505
Bayliss HR, Beyer FR (2014) Information retrieval for ecological syntheses. Res Synth Methods 6(2):136–148. https://doi.org/10.1002/jrsm.1120
Boanares D, Azevedo CS (2014) The use of nucleation techniques to restore the environment: a bibliometric analysis. Nat Conservação 12(2):93–98. https://doi.org/10.1016/j.ncon.2014.09.002
Boudry C, Baudouin C, Mouriaux F (2018) International publication trends in dry eye disease research: a bibliometric analysis. Ocular Surf 16(1):173–179. https://doi.org/10.1016/j.jtos.2017.10.002
Cabeza-Ramírez L, Sánchez Cañizares S, Fuentes-García F (2020) From bibliometrics to entrepreneurship: a study of studies. Revista española de Documentación Científica 43(3):268. https://doi.org/10.3989/redc.2020.3.1702
Calver MC, Goldman B, Hutchings PA, Kingsford RT (2017) Why discrepancies in searching the conservation biology literature matter. Biol Conserv 213(Part A):19–26. https://doi.org/10.1016/j.biocon.2017.06.028
CEE. Collaboration for Environmental Evidence (2018). Guidelines and standards for evidence synthesis in environmental management. https://environmentalevidence.org/information-for-authors/. Accessed 30 April 2020
Cobo MJ, López-Herrera AG, Herrera-Viedma E, Herrerab F (2011) Science mapping software tools: review, analysis, and cooperative study among tools. J Am Soc Inf Sci Technol 62:1382–1402. https://doi.org/10.1002/asi.21525
Fortuin KPJ, Van Koppen CSA, Leemans R (2011) The value of conceptual models in coping with complexity and interdisciplinarity in environmental sciences education. BioScience 61(10):802–814. https://doi.org/10.1525/bio.2011.61.10.10
Fu HZ, Wang MH, Ho YS (2013) Mapping of drinking water research: a bibliometric analysis of research output during 1992 - 2011. Sci Total Environ 443:757–765. https://doi.org/10.1016/j.scitotenv.2012.11.061
Garfield E (1972). The design and production of a citation index, In: Citation indexing-its theory and application in science, technology and humanities, John Wiley and Sons, New York, pp.19-36.
Gibert K, Horsburgh J, Athanasiadis I, Holmes G (2018) Environmental data science. Environ Model Softw 106:4–12. https://doi.org/10.1016/j.envsoft.2018.04.005
Glanville J (2017) Searching bibliographic databases. In: Cooper HC, Hedges LV, Valentine JC (eds) The handbook of research synthesis and metaanalysis, 3rd edn. Russell Sage Foundation, New York
Glanzel W, Moed HF (2002) Journal impact measures in bibliometric research. Scientometrics 53(2):171–193. https://doi.org/10.1023/A:1014848323806
Glanzel W, Schubert A (2004) Analyzing scientific networks through co-authorship. In: Glanzel W, Schubert A (eds) Handbook of quantitative science and technology research. Springer, Dordrecht, pp 257–276. https://doi.org/10.1007/1-4020-2755-9_12
Godin B (2006) On the origins of bibliometrics. Scientometrics 68(1):109–133. https://doi.org/10.1007/s11192-006-0086-0
Gonçalves MCP, Kieckbusch TG, Perna RF, Fujimoto JT, Morales SAV, Romanelli JP (2019) Trends on enzyme immobilization researches based on bibliometric analysis. Process Biochem 76:95–110. https://doi.org/10.1016/j.procbio.2018.09.016
Grames E, Elphick C (2020) Use of study design principles would increase the reproducibility of reviews in conservation biology. Biol Conserv 241:108385. https://doi.org/10.1016/j.biocon.2019.108385
Grames EM, Stillman AN, Tingley MW, Elphick CS (2019) An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods Ecol Evol 10:1645–1654. https://doi.org/10.1111/2041-210x.13268
Guan Y, Kang R, Liu J (2018) Evolution of the field of ecological restoration over the last three decades: a bibliometric analysis. Restor Ecol 27(3):647–660. https://doi.org/10.1111/rec.12899
Haddaway NR (2017) Response to collading science-based evidence to inform public opinion on the environmental effects of marine drilling platforms in the Mediterranean Sea. J Environ Manag 203(Part 1):612–614. https://doi.org/10.1016/j.jenvman.2017.03.043
Haddaway N, Woodcock P, Macura B, Collins A (2015) Making literature reviews more reliable through application of lessons from systematic reviews. Conserv Biol 29(6):1596–1605. https://doi.org/10.1111/cobi.12541
Higgins JP, Green S (2011) Cochrane handbook for systematic reviews of interventions. Wiley, Chichester
Hirsch HG, Hoffmann-Riem H, Biber-Klemm S, Grossenbacher-Mansuy W, Joye D, Pohl C, Wiesmann U (eds) (2008) Handbook of transdisciplinary research. Bern, Springer
Huang L, Zhang Y, Guo Y, Zhu D, Porter AL (2014) Four dimensional science and technology planning: a new approach based on bibliometrics and technology road mapping. Technol Forecast Soc Chang 81:39–48. https://doi.org/10.1016/j.techfore.2012.09.010
Koseoglu MA (2016) Mapping the institutional collaboration network of strategic management research: 1980–2014. Scientometrics 109(1):203–226. https://doi.org/10.1007/s11192-016-1894-5
Larsen P, von Ins M (2010) The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84(3):575–603. https://doi.org/10.1007/s11192-010-0202-z
Lázaro-Lobo A, Ervin G (2019) A global examination on the differential impacts of roadsides on native vs. exotic and weedy plant species. Global Ecol Conserv 17:e00555. https://doi.org/10.1016/j.gecco.2019.e00555
Leimu R, Koricheva J (2004) Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc R Soc Lond B 271:1961–1966
Lélé S, Norgaard RB (2005) Practicing Interdisciplinarity. BioScience 55(11):967. https://doi.org/10.1641/0006-3568(2005)055[0967:pi]2.0.co;2
Leydesdorff L, Wouters P, Bornmann L (2016) Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—astate-ofthe-art report. Scientometrics 109(3):2129–2150. https://doi.org/10.1007/s11192-016-2150-8
Livoreil B, Glanville J, Haddaway NR, Bayliss H, Bethel A, de Lachapelle FF, Robalino S, Savilaakso S, Zhou W, Petrokofsky G, Frampton G (2017) Systematic searching for environmental evidence using multiple tools and sources. Environ Evid 6(1). https://doi.org/10.1186/s13750-017-0099-6
Lokers R, Knapen R, Janssen S, van Randen Y, Jansen J (2016) Analysis of big data technologies for use in agro-environmental science. Environ Model Softw 84:494–504. https://doi.org/10.1016/j.envsoft.2016.07.017
Nakagawa S, Samarasinghe G, Haddaway NR, Westgate MJ, O’Dea RE, Noble DWA, Lagisz M (2018) Research weaving: visualizing the future of research synthesis. Trends Ecol Evol 2464:224–238. https://doi.org/10.1016/j.tree.2018.11.007
Okubo Y (1997) Bibliometric indicators and analysis of research systems: methods and examples. Organization for Economic Co-operation and Development (OECD), Paris
Ostrom E (1990) Governing the commons: the evolution of institutions for collective. Cambridge University Press, Cambridge
Palencia C, Castro A, Giaiotti D, Stel F, Vinet F, Fraile R (2009)Hailpad-based research: a bibliometric review. Atmos Res 93(1-3):664–670. https://doi.org/10.1016/j.atmosres.2008.09.025
Perz S, Brilhante S, Brown I, Michaelsen A, Mendoza E, Passos V et al (2010) Crossing boundaries for environmental science and management: combining interdisciplinary, interorganizational and international collaboration. Environ Conserv 37(4):419–431. https://doi.org/10.1017/s0376892910000810
Pilkington A, Chai KH (2008) Research themes, concepts and relationships: a study of International Journal of Service Industry Management (1990-2005). Int J Serv Ind Manag 19(1):83–110. https://doi.org/10.1108/09564230810855725
Ponce FA, Lozano AM (2010) The most cited works in Parkinson’s disease. Mov Disord 26(3):380–390. https://doi.org/10.1002/mds.23445
Pullin A, Stewart G (2006) Guidelines for systematic review in conservation and environmental management. Conserv Biol 20(6):1647–1656. https://doi.org/10.1111/j.1523-1739.2006.00485.x
Romanelli JP, Boschi RS (2019) The legacy of Elinor Ostrom on common forests research assessed through bibliometric analysis. Cerne 25(4):332–346. https://doi.org/10.1590/01047760201925042658
Romanelli JP, Fujimoto JT, Ferreira MD, Milanez DH (2018) Assessing ecological restoration as a research topic using bibliometric indicators. Ecol Eng 120:311–320. https://doi.org/10.1016/j.ecoleng.2018.06.015
Romanelli J, Meli P, Naves R, Alves M, Rodrigues R (2020a) Reliability of evidence-review methods in restoration ecology. Conserv Biol 35:142–154. https://doi.org/10.1111/cobi.13661
Romanelli J, Boschi R, Almeida D, Rodrigues R (2020b) Is the methodology used in reviews of restoration outcomes reliable? A systematic map protocol. Ecol Solut Evid 1(2). https://doi.org/10.1002/2688-8319.12030
Roudgarmi P (2011) Qualitative research for environmental sciences: a review. J Food, Agric Environ 9(3):871–879
Testa J (2009) The Thomson Reuters Journal Selection Process. Transnat Corp Rev 1(4):59–66. https://doi.org/10.1080/19186444.2009.11658213
van Eck N, Waltman L (2010) Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 84(2):523–538. https://doi.org/10.1007/s11192-009-0146-3
van Eck N, Waltman L (2014) CitNetExplorer: a new software tool for analyzing and visualizing citation networks. J Inform 8(4):802–823. https://doi.org/10.1016/j.joi.2014.07.00
Viana J, Santos J, Neiva R, Souza J, Duarte L, Teodoro A, Freitas A (2017) Remote sensing in human health: a 10-year bibliometric analysis. Remote Sens 9(12):1225. https://doi.org/10.3390/rs9121225
Vincenot CE (2018) How new concepts become universal scientific approaches: insights from citation network analysis of agent-based complex systems science. Proc R Soc B Biol Sci 285(1874):20172360. https://doi.org/10.1098/rspb.2017.2360
Vincent S, Focht W (2009) US higher education environmental program managers’ perspectives on curriculum design and core competencies: implications for sustainability as a guiding framework. Int J Sustain High Educ 10(2):164–183. https://doi.org/10.1108/14676370910945963
Wildgaard L (2015) A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar. Scientometrics 104(3):873–906. https://doi.org/10.1007/s11192-015-1608-4
Zupic I, Cater T (2015) Bibliometric methods in management and organization. Organ Res Methods 18(3):429–472. https://doi.org/10.1177/1094428114562629
Funding
The researchers are grateful to the “Fundação de Amparo à Pesquisa do Estado de São Paulo” (Fapesp, grants 2013/50718-5, 2018/18416-2, 2016/17304-0, 2019/23908-4 and 2019/08533-4), in party by “Coordenação de Aperfeiçoamento de Pessoal de Nível Superior” (CAPES - Finance Code 001), and “Conselho Nacional de Desenvolvimento Científico e Tecnológico” (CNPq, grant 141304/2019-7), for providing financial support.
Author information
Authors and Affiliations
Contributions
JPR, MCPG, and DFA conceived the ideas and designed methodology; JPR, MCPG, and DFA collected the data. All the authors led to the writing of the manuscript. All the authors contributed critically to the drafts and gave final approval for publication.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Responsible Editor: Philippe Garrigues
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Romanelli, .P., Gonçalves, M.C.P., de Abreu Pestana, L.F. et al. Four challenges when conducting bibliometric reviews and how to deal with them. Environ Sci Pollut Res 28, 60448–60458 (2021). https://doi.org/10.1007/s11356-021-16420-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11356-021-16420-x