Abstract
This chapter describes the practice of federal research impact evaluation. Evaluation of research impact is described for three cases: Research selection, where the work has not yet been performed; Research review, where work and results are ongoing; and ex-post research assessment, where research has been completed and results can be tracked. Qualitative methods (such as peer review), semi-quantitative methods (such as historiographic tracings of accomplishments and critical scientific events), and quantitative methods (such as cost-benefit analysis and bibliometrics) are described. While peer review in its broadest sense is the most widely used method in research selection, review, and ex-post assessment, it has its deficiencies, and there is no single method which provides a complete impact evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Bibliography
AFOSR, “Recent Research Accomplishments of the Air Force Office of Scientific Research”, AFOSR, 1989.
Averch, H., “Policy Uses of ‘Evaluation of Research’ Literature”, OTA Contractor Report, July 1990.
Averch, H., “The Practice of Research Evaluation in the United States,”, Research Evaluation, 1: 2, 1991.
Cozzens, S. E., “Expert Review in Evaluating Programs”, Science and Public Policy, 14: 2, April 1987.
DOD, “Project Hindsight”, Office of the Director of Defense Research and Engineering, DTIC Report No. AD495905, October 1969.
DOE, “An Assessment of the Basic Energy Sciences Program”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER-0123, March 1982.
DOE, “Health and Environmental Research: Summary of Accomplishments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER0194, May 1983
DOE, “Health and Environmental Research: Summary of Accomplishments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER0275, August 1986.
DOE, “Multiprogram Laboratory Appraisals”, DOE ORDER 5000. 2A, September 1988.
DOE, “Procedures for Peer Review Assessments”, Office of Energy Research, Office of Program Analysis, Report No. DOE/ER-0491P, April 1991.
Frazier, S. P., “University Funding: Information on the Role of Peer Review at NSF and NIH”, U.S. General Accounting Office Report No. GAO/RCED-8787FS, March 1987.
Griliches, Z., “Research Costs and Social Returns: Hybrid Corn and Related Innovations,” Journal of Political Economy, 66, 1958.
Hall, D., and Nauda, A., “An Interactive Approach for Selecting IRD Projects,” IEEE Transactions on Engineering Management, 37: 2, May 1990.
IITRI, “Technology in Retrospect and Critical Events in Science”, Illinois Institute of Technology Research Institute Report, December, 1968.
Irvine, J. and Martin, B. R., “Foresight in Science: Picking the Winners,” Frances Pinter, London, 1984.
Johnston, R., “Project Selection Mechanisms: International Comparisons”, OTA Contractor Report, July 1990.
Kerpelman, L. C., and Fitzsimmons, S. J. J., “Methods for the Strategic Evaluation of Research Programs: The State-of-the Art”, and “Annotated Bibliography”, NSF Contract No. PRA 8400688, Abt Associates Inc., 1985.
Kostoff, R. N., “A Cost/ Benefit Analysis of Commercial Fusion-Fission Hybrid Reactor Development”, Journal of Fusion Energy, 3: 2, 1983.
Kostoff, R. N., “Evaluation of Proposed and Existing Accelerated Research Programs by the Office of Naval Research”, IEEE Transactions of Engineering Management, 35: 4, November 1988.
Kostoff, R. N. and Stanford, L. B., “Program Funding Profiles under Budgetary Constraints”, Research Evaluation, 1: 1, Apr. 1991a.
Kostoff, R. N., “A Quantitative Approach to Determining the Impact of Research”, Presented at Twenty-Second Annual Pittsburgh Conference on Modeling and Simulation, May 2–3, 1991b.
Kostoff, R. N., “Research Impact Assessment”, Presented at Third International Conference on Management of Technology, February 17–21, 1992.
Logsdon, J. M., and Rubin, C. B., “An Overview of Federal Research Evaluation Activities”, Program of Policy Studies in Science and Technology Report, George Washington Univ., April 1985.
Luukonen-Gronow, T., “Scientific Research Evaluation: A Review of Methods and Various Contexts of Their Application,” RD Management, 17: 3, 1987.
Martin, B. R. et al, “Recent trends in the Output and Impact of British Science”, Science and Public Policy, 17:1, Feb., 1990.
Narin, F., “Bibliometric Techniques in the Evaluation of Research Programs”, Science and Public Policy, 14: 2, April 1987b.
Narin, F., “The Impact of Different Modes of Research Funding”, in: Evered, David and Harnett, Sara, Eds., The Evaluation of Scientific Research, John Wiley and Sons, Chichester, UK, 1989.
NIST, “Annual Report, 1990,” Visiting Committee on Advanced Technology, January 1991a.
NIST, “An Assessment of the National Institute of Standards and Technology Programs: FY 1990,” Board on Assessment of NIST Programs, National Research Council, National Academy Press, 1991b.
NSF, “Science and Engineering Indicators - 1989”, National Science Board Report NSB 89–1, GPO, Wash., D.C., 1989.
ONR, “1989 ONR Significant Accomplishments: Office of Naval Research Sponsored Programs”, ONR, January 1989.
Ormala, E., “Nordic Experiences of the Evaluation of Technical Research and Development”, Research Policy, 18, 1989.
OTA, “Research Funding as an Investment: Can We Measure the Returns”, U. S. Congress, Office of Technology Assessment, OTA-TM-SET-36 ( Wash., DC: U. S. GPO, April 1986 ).
OTA, “Federally Funded Research: Decisions for a Decade”, U. S. Congress, Office of Technology Assessment, OTA-SET-490 ( Wash., DC: U. S. GPO, May 1991 ).
Salasin, J. et al, “The Evaluation of Federal Research Programs”, MITRE Technical Report MTR-80W123, June 1980.
Wirt, J.G. et al, “RD Management: Methods Used by Federal Agencies”, Rand Report No. R-1156-HEW, January 1974.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1993 Springer Science+Business Media New York
About this chapter
Cite this chapter
Kostoff, R. (1993). Evaluating Federal R&D in the United States. In: Bozeman, B., Melkers, J. (eds) Evaluating R&D Impacts: Methods and Practice. Springer, Boston, MA. https://doi.org/10.1007/978-1-4757-5182-6_9
Download citation
DOI: https://doi.org/10.1007/978-1-4757-5182-6_9
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-5135-9
Online ISBN: 978-1-4757-5182-6
eBook Packages: Springer Book Archive