Skip to main content
SpringerLink
Log in

Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Research project evaluation and selection is mainly concerned with evaluating a number of research projects and then choosing some of them for implementation. It involves a complex multiple-experts multiple-criteria decision making process. Thus this paper presents an effective method for evaluating and selecting research projects by using the recently-developed evidential reasoning (ER) rule. The proposed ER rule based evaluation and selection method mainly includes (1) using belief structures to represent peer review information provided by multiple experts, (2) employing a confusion matrix for generating experts’ reliabilities, (3) implementing utility based information transformation to handle qualitative evaluation criteria with different evaluation grades, and (4) aggregating multiple experts’ evaluation information on multiple criteria using the ER rule. An experimental study on the evaluation and selection of research proposals submitted to the National Science Foundation of China demonstrates the applicability and effectiveness of the proposed method. The results show that (1) the ER rule based method can provide consistent and informative support to make informed decisions, and (2) the reliabilities of the review information provided by different experts should be taken into account in a rational research project evaluation and selection process, as they have a significant influence to the selection of eligible projects for panel review.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Agarski, B., Budak, I., Kosec, B., & Hodolic, J. (2012). An approach to multi-criteria environmental evaluation with multiple weight assignment. Environmental Modeling and Assessment, 17(3), 255–266.

    Article  Google Scholar 

  • Bulathsinhala, N. A. (2014), Ex-ante evaluation of publicly funded R&D projects: Searching for exploration. Science and Public Policy, scu035, 1-14.

  • Carlsson, C., Fullér, R., Heikkilä, M., & Majlender, P. (2007). A fuzzy approach to R&D project portfolio selection. International Journal of Approximate Reasoning, 44(2), 93–105.

    Article  MathSciNet  MATH  Google Scholar 

  • Chen, X. T. (2009). National science fund and management sciences (1986–2008). Beijing: Science Press.

    Google Scholar 

  • Coffin, M. A., & Taylor, B. W, I. I. I. (1996). Multiple criteria R&D project selection and scheduling using fuzzy logic. Computers & Operations Research, 23(3), 207–220.

    Article  Google Scholar 

  • Feng, B., Ma, J., & Fan, Z. P. (2011). An integrated method for collaborative R&D project selection: Supporting innovative research teams. Expert Systems with Applications, 38(5), 5532–5543.

    Article  Google Scholar 

  • Henriksen, A. D., & Traynor, A. J. (1999). A practical R&D project-selection scoring tool. IEEE Transactions on Engineering Management, 46(2), 158–170.

    Article  Google Scholar 

  • Horrobin, D. F. (1996). Peer review of grant applications: A harbinger for mediocrity in clinical research? The Lancet, 348(9037), 1293–1295.

    Article  Google Scholar 

  • Hsu, Y. G., Tzeng, G. H., & Shyu, J. Z. (2003). Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Management, 33(5), 539–551.

    Article  Google Scholar 

  • Huang, C. C., Chu, P. Y., & Chiang, Y. H. (2008). A fuzzy AHP application in government-sponsored R&D project selection. Omega, 36(6), 1038–1052.

    Article  Google Scholar 

  • Jayasinghe, U. W., Marsh, H. W., & Bond, N. (2006). A new reader trial approach to peer review in funding research grants: An Australian experiment. Scientometrics, 69(3), 591–606.

    Article  Google Scholar 

  • Južnič, P., Pečlin, S., Žaucer, M., Mandelj, T., Pušnik, M., & Demšar, F. (2010). Scientometric indicators: Peer-review, bibliometric methods and conflict of interests. Scientometrics, 85(2), 429–441.

    Article  Google Scholar 

  • Khalili-Damghani, K., Sadi-Nezhad, S., & Tavana, M. (2013). Solving multi-period project selection problems with fuzzy goal programming based on TOPSIS and a fuzzy preference relation. Information Sciences, 252, 42–61.

    Article  MathSciNet  Google Scholar 

  • Lawson, C. P., Longhurst, P. J., & Ivey, P. C. (2006). The application of a new research and development project selection model in SMEs. Technovation, 26(2), 242–250.

    Article  Google Scholar 

  • Linton, J. D., Walsh, S. T., & Morabito, J. (2002). Analysis, ranking and selection of R&D projects in a portfolio. R&D Management, 32(2), 139–148.

    Article  Google Scholar 

  • Mahmoodzadeh, S., Shahrabi, J., Pariazar, M., & Zaeri, M. S. (2007). Project selection by using fuzzy AHP and TOPSIS technique. World Academy of Science, Engineering and Technology, 30, 333–338.

    Google Scholar 

  • Meade, L. M., & Presley, A. (2002). R&D project selection using the analytic network process. IEEE Transactions on Engineering Management, 49(1), 59–66.

    Article  Google Scholar 

  • Olsson, N. O., Krane, H. P., Rolstadås, A., & Veiseth, M. (2010). Influence of reference points in ex post evaluations of rail infrastructure projects. Transport Policy, 17(4), 251–258.

    Article  Google Scholar 

  • Oral, M., Kettani, O., & Çınar, Ü. (2001). Project evaluation and selection in a network of collaboration: A consensual disaggregation multi-criterion approach. European Journal of Operational Research, 130(2), 332–346.

    Article  MATH  Google Scholar 

  • Provost, F., & Kohavi, R. (1998). Guest editors’ introduction: On applied research in machine learning. Machine Learning, 30(2), 127–132.

    Article  Google Scholar 

  • Shafer, G. (1976). A mathematical theory of evidence (Vol. 1). Princeton: Princeton University Press.

    MATH  Google Scholar 

  • Silva, T., Guo, Z., Ma, J., Jiang, H., & Chen, H. (2013). A social network-empowered research analytics framework for project selection. Decision Support Systems, 55(4), 957–968.

    Article  Google Scholar 

  • Silva, T., Jian, M., & Chen, Y. (2014). Process analytics approach for R&D project selection. ACM Transactions on Management Information Systems (TMIS), 5(4), 21.

    Google Scholar 

  • Smarandache F, Dezert J, Tacnet J. (2010). Fusion of sources of evidence with different importances and reliabilities. In: 2010 13th conference on information fusion (FUSION). IEEE (pp. 1–8).

  • Solak, S., Clarke, J. P. B., Johnson, E. L., & Barnes, E. R. (2010). Optimization of R&D project portfolios under endogenous uncertainty. European Journal of Operational Research, 207(1), 420–433.

    Article  MathSciNet  MATH  Google Scholar 

  • Tavana, M., Khalili-Damghani, K., & Sadi-Nezhad, S. (2013). A fuzzy group data envelopment analysis model for high-technology project selection: A case study at NASA. Computers & Industrial Engineering, 66(1), 10–23.

    Article  Google Scholar 

  • Tian, Q., Ma, J., Liang, J., Kwok, R. C., & Liu, O. (2005). An organizational decision support system for effective R&D project selection. Decision Support Systems, 39(3), 403–413.

    Article  Google Scholar 

  • Wang, J., & Hwang, W. L. (2007). A fuzzy set approach for R&D portfolio selection using a real options valuation model. Omega, 35(3), 247–257.

    Article  MathSciNet  Google Scholar 

  • Wang, D. P., Zhu, W. D., Chen, B., & Liu, F. (2015). Analysis of the expert judgments in the NSFC peer review from the perspective of human cognition: A research based on the ER rule. Science of science and management of S. & T., 36(4), 22–35.

    Google Scholar 

  • Xu, D. L. (2012). An introduction and survey of the evidential reasoning approach for multiple criteria decision analysis. Annals of Operations Research, 195(1), 163–187.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J. B. (2001). Rule and utility based evidential reasoning approach for multiattribute decision analysis under uncertainties. European Journal of Operational Research, 131(1), 31–61.

    Article  MathSciNet  MATH  Google Scholar 

  • Yang, J. B., & Singh, M. G. (1994). An evidential reasoning approach for multiple-attribute decision making with uncertainty. IEEE Transactions on Systems, Man and Cybernetics, 24(1), 1–18.

    Article  Google Scholar 

  • Yang, J. B., & Xu, D. L. (2002). On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, 32(3), 289–304.

    Article  Google Scholar 

  • Yang, J. B., & Xu, D. L. (2013). Evidential reasoning rule for evidence combination. Artificial Intelligence, 205, 1–29.

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This research is partially supported by the National Natural Science Foundation of China under Grant No. 71071048 and the Scholarship from China Scholarship Council under Grant No. 201306230047.

Author information

Authors and Affiliations

  1. School of Economics, Hefei University of Technology, 193 Tunxi Road, Hefei, 230009, Anhui, China

    Wei-dong Zhu

  2. School of Management, Hefei University of Technology, 193 Tunxi Road, Hefei, 230009, Anhui, China

    Fang Liu, Jian-bo Yang, Dong-ling Xu & Dong-peng Wang

  3. Manchester Business School, The University of Manchester, Manchester, M15 6PB, UK

    Fang Liu, Yu-wang Chen, Jian-bo Yang & Dong-ling Xu

Authors
  1. Wei-dong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu-wang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jian-bo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dong-ling Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dong-peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fang Liu.

Appendices

Appendix 1: Reliabilities of experts for “Results and comparative analysis” section

As the reliabilities of some experts are not available in the data set, the average true positive rate of 0.2726 and the average true negative rate of 0.9592 are used as their reliabilities accordingly.

 

No. of projects

Negative

Positive

TN

TP

True positive rate

True negative rate

Project 1/Expert 1

2

 

2

  

0

 

Expert 2

12

5

7

4

 

0

0.8

Expert 3

11

9

2

7

1

0.5

0.7778

Expert 4

19

4

15

4

6

0.4

1

Expert 5

4

1

3

1

 

0

1

Project 3/Expert 1

11

8

3

5

 

0

0.625

Expert 2

11

6

5

6

1

0.2

1

Expert 3

18

3

15

3

4

0.2667

1

Expert 4

15

11

4

11

1

0.25

1

Expert 5

10

4

6

4

 

0

1

Project 4/Expert 1

       

Expert 2

15

4

11

4

1

0.0910

1

Expert 3

16

13

3

12

 

0

0.9231

Expert 4

7

1

6

1

3

0.5

1

Expert 5

12

7

5

7

2

0.4

1

Project 5/Expert 1

       

Expert 2

       

Expert 3

       

Expert 4

6

1

5

1

 

0

1

Expert 5

7

5

2

5

1

0.5

1

Project 6/Expert 1

       

Expert 2

14

4

10

4

3

0.3

1

Expert 3

18

2

16

2

3

0.1875

1

Expert 4

       

Expert 5

12

5

7

5

4

0.5714

1

Project 7/Expert 1

       

Expert 2

3

1

2

1

2

1

1

Expert 3

23

7

16

7

3

0.1875

1

Expert 4

       

Expert 5

       

Project 8/Expert 1

15

11

4

11

1

0.25

1

Expert 2

18

3

15

3

4

0.2667

1

Expert 3

11

6

5

6

1

0.2

1

Expert 4

10

4

6

4

 

0

1

Expert 5

11

8

3

5

 

0

0.625

Project 9/Expert 1

10

2

8

2

3

0.375

1

Expert 2

11

5

6

4

2

0.3333

0.8

Expert 3

5

1

4

1

1

0.25

1

Expert 4

20

6

14

6

6

0.4286

1

Expert 5

11

8

3

5

 

0

0.625

Project 10/Expert 1

18

3

15

3

4

0.2667

1

Expert 2

15

11

4

11

1

0.25

1

Expert 3

11

8

3

5

 

0

0.625

Expert 4

10

4

6

4

 

0

1

Expert 5

11

6

5

6

1

0.2

1

Appendix 2

 

Expert 1

Expert 2

Expert 3

Expert 4

Expert 5

Original evaluation information of project R 2 by experts

 Comprehensive evaluation level

Average

Good

Poor

Excellent

Excellent

 Funding recommendation

Not fund

Fund

Not fund

Fund with priority

Fund

 

Expert 1

Expert 2

Expert 3

Expert 4

Expert 5

Original evaluation information of project R 3 by experts

 Comprehensive evaluation level

Average

Good

Good

Good

Average

 Funding recommendation

Fund

Fund

Fund

Fund

Fund

 

Experts

Projects

Negative

Positive

TN

TP

True positive rate

True negative rate

Reliabilities of experts for project R 2 and R 3

 Project R 2

Expert 3

13

4

9

4

4

0.4444

1

 Project R 3

Expert 3

19

6

13

6

6

0.461538462

1

Expert 5

12

2

10

2

3

0.3

1

  1. The original data set is available for research use with request

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, Wd., Liu, F., Chen, Yw. et al. Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities. Scientometrics 105, 1469–1490 (2015). https://doi.org/10.1007/s11192-015-1770-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1770-8

Keywords

Search

Navigation