Abstract
The systematic literature reviews (SLRs) are a formally planned approach in finding, evaluating and summarising all available evidence on a specific research question. The objective of this paper is to compare formal SLRs and informal literature reviews in software engineering. For this purpose, a SLR has been conducted to compare the results with a previously conducted informal literature review of software process improvement success factors. Previous research using an informal literature review approach found 47 articles via the snow balling technique. For formal literature reviews, the SLR was conducted by applying customised search strings covering the time to 30 June 2004 (this is the deadline of the informal literature review). In total, 38 relevant articles were identified via the formal literature review. The results show that the data extraction process in the formal literature review enabled more success factors to be extracted (i.e. 34) than the informal literature review (i.e. 18). In the formal literature review, the publication inclusion and exclusion criteria and selecting primary studies helped in identifying the right list of publications. A real challenge in the formal literature review was to define a search string. In general it was observed that the SLR methodology is better than the informal literature review with respect to the planning for literature review, the design of search string, sources to be searched, publication inclusion and exclusion criteria, publication quality assessment and the data extraction process. However, the informal literature review has identified more articles than the formal literature review. Snow balling technique should be used with the formal literature review in order to identify all the relevant articles.
Similar content being viewed by others
References
Kitchenham, B.; Charters, C.: Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report—EBSE 2007-001 (2007)
Niazi M., Wilson D., Zowghi D.: Critical success factors for software process improvement: an empirical study. Softw. Process Improv. Pract. J. 11(2), 193–211 (2006)
Dyba T., Kitchenham B., Jorgensen M.: Evidence based software engineering for practitioners. IEEE Softw. 22(1), 158–165 (2005)
Sackett D.L., Straus S.E., Richardson W.S., Rosenberg W., Haynes R.B.: Evidence-Based Medicine: How to Practice and Teach EBM, 2nd edn. Churchill Livingstone, Edinburgh (2000)
Khan S.U., Niazi M., Ahmad R.: Factors influencing clients in the selection of offshore software outsourcing vendors: an exploratory study using a systematic literature review. J. Syst. Softw. 84(4), 686–699 (2011)
Smite D., Wohlin C., Gorscheck T., Feldt R.: Empirical evidence in global software engineering: a systematic review. Empir. Softw. Eng. 15(1), 91–118 (2010)
Niazi M., Ikram N., Bano M., Imtiaz S., Khan S.U.: Establishing trust in offshore software outsourcing relationships: an exploratory study using a systematic literature review. IET Softw. 7(5), 283–293 (2013)
Dyba T., Dingsoyr T.: Empirical studies of agile software development: a systematic review. Inf. Softw. Technol. 50(9), 833–859 (2008)
Staples M., Niazi M.: Systematic review of organizational motivations for adopting CMM-based SPI. Inf. Softw. Technol. J. 50(7–8), 605–620 (2008)
Verner J., Brereton O.P., Kitchenham B., Turner M., Niazi M.: Risks and risk mitigation in global software development: a tertiary study. Inf. Softw. Technol. J. 56(1), 54–78 (2014)
Kitchenham B., Mendes E., Travassos G.H.: Cross vs within-company cost estimation studies: a systematic review. IEEE Trans. Softw. Eng. 33(5), 316–329 (2007)
Staples M., Niazi M.: Experiences using systematic review guidelines. J. Syst. Softw. 80(9), 1425–1437 (2007)
Imtiaz, S.; Bano, M.; Ikram, N.; Niazi, M.: Experiences of conducting systematic literature reviews in software engineering: a tertiary study. In: 17th International Conference on Evaluation and Assessment in Software Engineering, EASE 2013 (2013)
Fabio, D.S.; André, S.; Sérgio, S.; César, F.; Cleviton, M.: A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (2010)
Ali Barbar, M.; Zhang, H.: Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: Third International Symposium on Empirical Software Engineering and Measurement (2009)
Kitchenham B., Brereton P., Budgen D., Turner M., Bailey J., Linkman S.: Systematic literature reviews in software engineering: a systematic literature review. Inf. Softw. Technol. 51(1), 7–15 (2009)
Kitchenham B., Pretorius R., Brereton P., Budgen D., Turner M., Niazi M., Linkman S.: Systematic literature reviews in software engineering—a tertiary study. Inf. Softw. Technol. J. 52(8), 792–805 (2010)
Verner, J.; Brereton, O.P.; Kitchenham, B.; Turner, M.; Niazi, M.: Systematic literature reviews in global software development: a tertiary study. In: International Conference on Evaluation & Assessment in Software Engineering (EASE 2012) (2012)
Kitchenham B., Pretorius R., Brereton P., Budgen D., Turner M., Niazi M., Linkman S.: Refining the systematic literature review process—two participant-observer case studies. Empir. Softw. Eng. J. 15(6), 618–653 (2010)
Niazi M.: An exploratory study of software process improvement implementation risks. J. Softw. Evol. Process 24(8), 877–894 (2012)
Khan S.U., Niazi M., Rashid A.: Barriers in the selection of offshore software development outsourcing vendors: an exploratory study using a systematic literature review. J. Inf. Softw. Technol. 53(7), 693–706 (2011)
Niazi M., Wilson D., Zowghi D.: A maturity model for the implementation of software process improvement: an empirical study. J. Syst. Softw. 74(2), 155–172 (2005)
Niazi, M.: Software process improvement: a road to success. In: The 7th International Conference on Product Focused Software Process Improvement, LNCS (2006)
Cox K., Niazi M., Verner J.: An empirical study of Sommerville’s requirements engineering practices. IET Softw. J. 3(5), 339–355 (2009)
Niazi, M.; Cox, K.; Verner, J.: An empirical study identifying high perceived value requirements engineering practices. In: Advances in Information Systems Development, Fourteenth International Conference on Information Systems Development (ISD’2005) Karlstad University, Sweden 15–17 Aug. Springer (2006)
McCabe G.P., Moore D.S.: Introduction to the Practice of Statistics, 4th edn. W.H.Freeman and Co, New York (2002)
Chen, L.: Systematic literature review: experiences and observations from the trenches. Keele University UK Seminar series (2009)
Hossain, E.; Ali-Babar, M.; Paik, H.-Y.: Using scrum in global software development: a systematic literature review. In: IEEE International Conference on Global Software Engineering, ICGSE09. Lero, Limerick, Ireland (2009)
Alshayeb M.: The impact of refactoring to patterns on software quality attributes. Arab. J. Sci. Eng. 36(7), 1241–1251 (2011)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Niazi, M. Do Systematic Literature Reviews Outperform Informal Literature Reviews in the Software Engineering Domain? An Initial Case Study. Arab J Sci Eng 40, 845–855 (2015). https://doi.org/10.1007/s13369-015-1586-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13369-015-1586-0