Abstract
Most benchmarks are smaller than actual application programs. One reason is to improve benchmark universality by demanding resources every computer is likely to have. However, users dynamically increase the size of application programs to match the power available, whereas most benchmarks are static and of a size appropriate for computers available when the benchmark was created; this is particularly true for parallel computers. Thus, the benchmark overstates computer performance, since smaller problems spend more time in cache. Scalable benchmarks, such as HINT, examine the full spectrum of performance through various memory regimes, and express a superset of the information given by any particular fixed-size benchmark. Using 5,000 experimental measurements, we have found that performance on the NAS Parallel Benchmarks, SPEC, LINPACK, and other benchmarks is predicted accurately by subsets of HINT performance curve. Correlations are typically better than 0.995. Predicted ranking is often perfect.
Similar content being viewed by others
References
A. AburtoJr. Benchmarks developed and maintained at the Naval Command Control and Ocean Surveillance Center RDTE Division NRaD, San Diego, CA. ftp site: ftp.nosc.mil in directory pub r aburto. Mirrored at the NIST Web site.
D. Bailey, J. Barton, T. Lasinski, and H. Simon. “The NAS Parallel Benchmarks.” Report RNR-91-002, NASA r Ames Research Center, January 1991.
H. J. Curnow and B. A. Wichmann. “A synthetic benchmark.” Computer Journal, 191, February 1976.
J. Dongarra. “Performance of various computers using standard linear equations software in a Fortran environment.” ORNL, updated periodically.
J. Dongarra and W. Gentzsch, Eds..Computer Benchmarks, North Holland: Amsterdam, 1993.
J. Gustafson and Q. Snell. “HINT: A new way to measure computer performance,” in Proceedings of the Twenty-Eighth Hawaii International Conference on System Sciences, Wailea, Maui, Hawaii, January 1995.
J. Gustafson, Q. Snell, R. Todi. HINT web site: http:// www.scl.ameslab.gov/ HINT.
F. H. McMahon.“L.L.N.L FORTRAN KERNELS: MFLOPS.” Lawrence Livermore National Laboratory, benchmark tape available, 1983.
S. Saini and D. Bailey. “NAS Parallel Benchmark results 12-95,” Report NAS-95-021, NASA r Ames Research Center, December 1995.
Silicon Graphics Inc. “Origin 2000 & Onyx 2 4 MB cache performance report.” Revision 1.07, March 17, 1997.
The SPEC web site is http:// www.specbench.org/ osg / cpu95 /.
The STREAM web site is http:// reality.sgi.com/ mccalpin/ papers / bandwidth/ bandwidth.html.
R. Weicker. “Dhrystone: A synthetic systems programming benchmark.” Communications of the ACM, Volume 27, Number 10, October 1984.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Gustafson, J.L., Todi, R. Conventional Benchmarks as a Sample of the Performance Spectrum. The Journal of Supercomputing 13, 321–342 (1999). https://doi.org/10.1023/A:1008013204871
Issue Date:
DOI: https://doi.org/10.1023/A:1008013204871