SPEC/HPG: Measuring real-world supercomputing performance

As far back as the early 80s, David Kuck and his supercomputing colleagues knew there needed to be something better than kernel-level benchmarks. In response to this need, he formed the PERFECT Club in mid-1980s.

PERFECT stands for PERFormance Evaluation for Cost-Effective Transformations. The Club comprised a couple dozen members from various organizations united in developing a better supercomputing benchmark. The result was a benchmark representing 13 Fortan-based engineering and scientific applications containing more than 50,000 lines of code.

But, Kuck had a problem. He realized he couldn't keep the benchmark moving forward without a lot of help. That's when he and Rudi Eigenmann, a member of Kuck's research team, realized that SPEC would provide the ideal home for continued benchmark development.

In 1996, the SPEC High-Performance Group (SPEC/HPG) released its first benchmark, SPEChpc®96. It contained two applications: a seismic processing application with 20,000 lines of code and a computational chemistry program comprising 110,000 lines of code.

Currently, SPEC/HPG covers the supercomputing arena with three benchmark suites:

• The SPEC MPI® benchmark suite for measuring performance of compute-intensive applications using the Message-Passing Interface (MPI) across a wide range of cluster and SMP hardware.
• The SPEC OMP® benchmark suite for measuring performance and energy efficiency using applications based on the OpenMP standard for shared-memory parallel processing.
• The SPEC ACCEL® benchmark suite for measuring hardware-accelerated performance for highly parallel compute-intensive applications using the OpenCL, OpenACC and OpenMP standards.

In March 2018, SPEC/HPG made a bold move designed to provide easy access to tools for research vital to the ongoing development of supercomputing technology: It began offering its benchmarks free of charge to qualified non-profit organizations worldwide.

"SPEC/HPG benchmarks provide a framework that helps to ensure the rigor, reproducibility and application-based versatility that are critical to sound research efforts."

-- Torsten Hoefler, who leads the Scalable Parallel Computing Laboratory (SPCL) at ETH Zurich and co-chairs the MPI Forum's collective operations working group.

Standard Performance Evaluation Corporation