Skip to main content
SpringerLink
Log in

Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

As a typical multi-criteria group decision making (MCGDM) problem, research and development (R&D) project selection involves multiple decision criteria which are formulated by different frames of discernment, and multiple experts who are associated with different weights and reliabilities. The evidential reasoning (ER) rule is a rational and rigorous approach to deal with such MCGDM problems and can generate comprehensive distributed evaluation outcomes for each R&D project. In this paper, an ER rule based model taking into consideration experts’ weights and reliabilities is proposed for R&D project selection. In the proposed approach, a utility based information transformation technique is applied to handle qualitative evaluation criteria with different evaluation grades, and both adaptive weights of criteria and utilities assigned to evaluation grades are introduced to the ER rule based model. A nonlinear optimisation model is developed for the training of weights and utilities. A case study with the National Science Foundation of China is conducted to demonstrate how the proposed method can be used to support R&D project selection. Validation data show that the evaluation results become more reliable and consistent with reality by using the trained weights and utilities from historical data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Agarski, B., Budak, I., Kosec, B., & Hodolic, J. (2012). An approach to multi-criteria environmental evaluation with multiple weight assignment. Environmental Modeling & Assessment, 17(3), 255–266.

    Article  Google Scholar 

  • Amiri, M. P. (2010). Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods. Expert Systems with Applications, 37(9), 6218–6224.

    Article  MathSciNet  Google Scholar 

  • Baker, N., & Freeland, J. (1975). Recent advances in R&D benefit measurement and project selection methods. Management Science, 21(10), 1164–1175.

    Article  Google Scholar 

  • Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1), 197–245.

    Article  Google Scholar 

  • Chiang, T. A., & Che, Z. H. (2010). A fuzzy robust evaluation model for selecting and ranking NPD projects using Bayesian belief network and weight-restricted DEA. Expert Systems with Applications, 37(11), 7408–7418.

    Article  MathSciNet  Google Scholar 

  • Collan, M., & Luukka, P. (2014). Evaluating R&D projects as investments by using an overall ranking from four new fuzzy similarity measure-based TOPSIS variants. IEEE Transactions on Fuzzy Systems, 22(3), 505–515.

    Article  Google Scholar 

  • Dempster, A. P. (1967). Upper and lower probabilities induced by a multivalued mapping. The Annals of Mathematical Statistics, 38, 325–339.

    Article  MathSciNet  MATH  Google Scholar 

  • Feng, B., Ma, J., & Fan, Z. P. (2011). An integrated method for collaborative R&D project selection: Supporting innovative research teams. Expert Systems with Applications, 38(5), 5532–5543.

    Article  Google Scholar 

  • Heidenberger, K., & Stummer, C. (1999). Research and development project selection and resource allocation: A review of quantitative modelling approaches. International Journal of Management Reviews, 1(2), 197–224.

    Article  Google Scholar 

  • Henriksen, A. D., & Traynor, A. J. (1999). A practical R&D project-selection scoring tool. IEEE Transactions on Engineering Management, 46(2), 158–170.

    Article  Google Scholar 

  • Hsu, Y. G., Tzeng, G. H., & Shyu, J. Z. (2003). Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Management, 33(5), 539–551.

    Article  Google Scholar 

  • Huang, C. C., Chu, P. Y., & Chiang, Y. H. (2008). A fuzzy AHP application in government-sponsored R&D project selection. Omega, 36(6), 1038–1052.

    Article  Google Scholar 

  • Jackson, B. (1983). Decision methods for selecting a portfolio of R&D projects. Research Management, 26(5), 21–26.

    Google Scholar 

  • Jung, U., & Seo, D. W. (2010). An ANP approach for R&D project evaluation based on interdependencies between research objectives and evaluation criteria. Decision Support Systems, 49(3), 335–342.

    Article  Google Scholar 

  • Južnič, P., Pečlin, S., Žaucer, M., Mandelj, T., Pušnik, M., & Demšar, F. (2010). Scientometric indicators: Peer-review, bibliometric methods and conflict of interests. Scientometrics, 85(2), 429–441.

    Article  Google Scholar 

  • Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.

    Article  Google Scholar 

  • Liu, W., Hu, G., Tang, L., & Wang, Y. (2015a). China's global growth in social science research: Uncovering evidence from bibliometric analyses of SSCI publications (1978–2013). Journal of Informetrics, 9(3), 555–569.

    Article  Google Scholar 

  • Liu, W., Tang, L., Gu, M., & Hu, G. (2015b). Feature report on China: A bibliometric analysis of China-related articles. Scientometrics, 102(1), 503–517.

    Article  Google Scholar 

  • Liu, X. B., Pei, F., Yang, J. B., & Yang, S. L. (2010). An MAGDM approach combining numerical values with uncertain linguistic information and its application in evaluation of R&D projects. International Journal of Computational Intelligence Systems, 3(5), 575–589.

    Article  Google Scholar 

  • Mahmoodzadeh, S., Shahrabi, J., Pariazar, M., & Zaeri, M. S. (2007). Project selection by using fuzzy AHP and TOPSIS technique. International Journal of Human and social sciences, 1(3), 135–140.

    Google Scholar 

  • Meade, L. M., & Presley, A. (2002). R&D project selection using the analytic network process. IEEE Transactions on Engineering Management, 49(1), 59–66.

    Article  Google Scholar 

  • Poh, K. L., Ang, B. W., & Bai, F. (2001). A comparative analysis of R&D project evaluation methods. R&D Management, 31(1), 63–75.

    Article  Google Scholar 

  • Provost, F., & Kohavi, R. (1998). Guest editors’ introduction: On applied research in machine learning. Machine Learning, 30(2), 127–132.

    Article  Google Scholar 

  • Saaty, T. L. (1980). The analytic hierarchy process: Planning, priority setting, resources allocation. New York: McGraw.

    MATH  Google Scholar 

  • Schmidt, R. L., & Freeland, J. R. (1992). Recent progress in modeling R&D project-selection processes. IEEE Transactions on Engineering Management, 39(2), 189–201.

    Article  Google Scholar 

  • Shafer, G. (1976). A mathematical theory of evidence (Vol. 1). Princeton: Princeton University Press.

    MATH  Google Scholar 

  • Smarandache, F., Dezert, J., & Tacnet, J. M. (2010). Fusion of sources of evidence with different importances and reliabilities. In 2010 13th Conference on information fusion (FUSION) (pp. 1–8). IEEE.

  • Taroun, A., & Yang, J. B. (2011). Dempster–Shafer theory of evidence: Potential usage for decision making and risk analysis in construction project management. The Built & Human Environment Review, 4(Special Issue 1), 155–166.

    Google Scholar 

  • Tian, Q., Ma, J., Liang, J., Kwok, R. C., & Liu, O. (2005). An organizational decision support system for effective R&D project selection. Decision Support Systems, 39(3), 403–413.

    Article  Google Scholar 

  • van Raan, A. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36(3), 397–420.

    Article  Google Scholar 

  • Wessely, S. (1998). Peer review of grant applications: What do we know? The Lancet, 352, 301–305.

    Article  Google Scholar 

  • Xiao, Z., Yang, X., Pang, Y., & Dang, X. (2012). The prediction for listed companies’ financial distress by using multiple prediction methods with rough set and Dempster–Shafer evidence theory. Knowledge-Based Systems, 26, 196–206.

    Article  Google Scholar 

  • Xu, D. L. (2012). An introduction and survey of the evidential reasoning approach for multiple criteria decision analysis. Annals of Operations Research, 195(1), 163–187.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J. B. (2001). Rule and utility based evidential reasoning approach for multiattribute decision analysis under uncertainties. European Journal of Operational Research, 131(1), 31–61.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J. B., & Singh, M. G. (1994). An evidential reasoning approach for multiple-attribute decision making with uncertainty. IEEE Transactions on Systems, Man, and Cybernetics, 24(1), 1–18.

    Article  Google Scholar 

  • Yang, J. B., & Xu, D. L. (2002). On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 32(3), 289–304.

    Article  Google Scholar 

  • Yang, J. B., & Xu, D. L. (2013). Evidential reasoning rule for evidence combination. Artificial Intelligence, 205, 1–29.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J. B., & Xu, D. L. (2014). A study on generalising Bayesian inference to evidential reasoning. In International conference on belief functions (pp. 180–189). Springer.

  • Zhu, W. D., Liu, F., Chen, Y. W., Yang, J. B., Xu, D. L., & Wang, D. P. (2015). Research project evaluation and selection: An evidential reasoning rule-based method for aggregating peer review information with reliabilities. Scientometrics, 105(3), 1469–1490.

    Article  Google Scholar 

Download references

Acknowledgements

This research is partially supported by the National Natural Science Foundation of China under Grant No. 71071048 and 71601060 and the Scholarship from China Scholarship Council under Grant No. 201306230047.

Author information

Authors and Affiliations

  1. School of Accounting, Zhejiang University of Finance and Economics, 18 Xueyuan Road, Hangzhou, 310018, Zhejiang, China

    Fang Liu

  2. School of Economics, Hefei University of Technology, 193 Tunxi Road, Hefei, 230009, Anhui, China

    Wei-dong Zhu

  3. Alliance Manchester Business School, The University of Manchester, Manchester, M15 6PB, UK

    Yu-wang Chen, Dong-ling Xu & Jian-bo Yang

Authors
  1. Fang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei-dong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu-wang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dong-ling Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian-bo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fang Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, F., Zhu, Wd., Chen, Yw. et al. Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach. Scientometrics 111, 1501–1519 (2017). https://doi.org/10.1007/s11192-017-2278-1

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-017-2278-1

Keywords

Search

Navigation