SPEC Establishes Machine Learning Committee to Develop Vendor-Agnostic Benchmarks

New benchmarks to measure end-to-end system performance of ML training and inference tasks

GAINESVILLE, Va., March 10, 2022 — (SPEC) today announced the formation of the SPEC Machine Learning Committee. The SPEC ML Committee will develop practical methodologies for benchmarking artificial intelligence (AI) and machine learning (ML) performance in the context of real-world platforms and environments. The Committee will also work with other SPEC committees to update their benchmarks for ML environments. Current members include AMD, Dell, Inspur, Intel, NetApp, NVIDIA and Red Hat.

"IDC expects enterprises to spend nearly $342 billion on AI in 2021, and it's essential that these companies understand what that money will buy," said Arthur Kang, Chair of the SPEC ML Committee. "This new committee will design and develop the vendor agnostic benchmarks that vendors need to prove their solutions in a competitive market and enterprises need to make an informed buying decision. I encourage anyone interested in the future of ML processing to join the SPEC ML Committee and help shape these invaluable benchmarks."

The SPEC ML Committee is initially developing benchmarks to measure end-to-end performance of a system under test (SUT) handling ML training and inference tasks. The goal of these benchmarks is to better represent industry practices compared to other existing benchmarks by including major parts of the end-to-end ML/DL pipeline, including data prep and training/inference. This vendor-neutral third-party benchmark will enable ML system designers to benchmark their offerings against those of their competitors and allow ML users, such as enterprises and scientific research institutions, to better understand how solutions will perform in real-world environments, enabling them to make better purchasing decisions.

The SPEC ML Committee welcomes any current SPEC members willing to join the development and management of the SPEC ML benchmark, as well as new SPEC members, especially ML/DL end users and manufacturers. Visit membership or contact [email protected] for more information.

About SPEC

SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks and tools to evaluate performance for the newest generation of computing systems. Its membership comprises more than 120 leading computer hardware and software vendors, educational institutions, research organizations, and government agencies worldwide.

###

Media contact: Brigit Valencia
360.597.4516
[email protected]

Images available upon request.

SPEC® is a trademark of the Standard Performance Evaluation Corporation. All other product and company names herein may be trademarks of their registered owners.