The SPECjbb2015 Benchmark - Frequently Asked Questions (FAQ) - December 9, 2020

Common questions are addressed in this FAQ or in the benchmark documentation on the SPECjbb2015 Benchmark Information web page.

This version of the FAQ is aligned with the 1.03 version of the SPECjbb2015 benchmark.
Subsequent updates will be added to SPECjbb2015 FAQ.

For all other questions, send your query via e-mail to: [email protected] with "SPECjbb" in the subject line.

About the Benchmark

  1. The SPECjbb2015 benchmark replaces SPECjbb2013 as the next generation Java server business benchmark.
  2. Can the SPECjbb2015 documents have different versioning numbers than the latest SPECjbb2015 kit?
    • Yes, each document has its own version number. Check the SPECjbb2015 website for the latest documents.
  3. Can SPECjbb2015 be used to produce a compliant result within a cloud environment?
    • Yes, such a submission may be compliant depending on the test sponsor's knowledge of the test-bed and the conditions under which SPECjbb2015 is tested. Sections "2.6. General Availability", "2.7.3. Hardware" and "2.7.4. Software" of the SPECjbb2015 Benchmark Run and Reporting Rules define the details about each software and hardware component used within the test-bed that the test sponsor is required to disclose as part of the submission. Additionally, the intent of section "2.7.4. Software" is that the SUT (System Under Test) cannot be shared while testing. This is very important as SPECjbb2015 requires undisturbed steady state availability on the SUT to accurately determine both metrics of max-jOPS and critical-jOPS.
Hardware Requirements
  1. What is the minimum hardware required?
    • At minimum, a server with 8GB of RAM is required. 24GB and above is recommended for performance runs
Software Requirements
  1. What Operating Systems are supported?
    • On the SUT side, the workload is designed to run on any OS for which a Java SE 7 VM or higher is available.
  2. What other software will I need?
    • A Java SE 7 (or higher) virtual machine (JVM).
    • The workload is written in Java to enable the highest degree of portability across software operating environments.
  3. Which Java runtime environments (JREs) can be used for a compliant run?
    • Any JRE can be used as long as it adheres to the SPECjbb2015 Benchmark Run and Reporting Rules. Notably, but not limited to:
      1. Section 2.6 General Availability, which includes support requirements
      2. Section 2.8 Java Specifications, which outlines JVM implementation requirements
Acquiring the Benchmark
  1. How much does the benchmark cost?
  2. How do I order the benchmark?
Issues
  1. Where can I look to address issues?
Result Submission
  1. Where can I indicate, within the Full Disclosure Report (FDR), that the software or hardware has support?
  2. How do I show that a community supported Java meets the General Availability and Support requirements?
    • If you are using a community supported Java, it is required that you provide a link(s) to page(s) from which the committee can determine whether the community product meets the General Availability and Support requirements. The requirements for community supported products are outlined in Appendix C. Guidelines for General Availability
      • Enter the link(s) in the JVM Notes section of the FDR.
    • Here are example entries for some common community supported Java. Please ensure the links are up-to-date before using. This is not meant to be an exhaustive list.
      • OpenJDK:
        Community Support:
        • Community URL: http://openjdk.java.net/projects/jdk/
        • Support URL: http://bugs.openjdk.java.net/secure/Dashboard.jspa
      • AdoptOpenJDK:
        Community Support: https://adoptopenjdk.net/support.html
      • Red Hat OpenJDK:
        Community Support: https://access.redhat.com/articles/1299013
  3. Do I need to specify the BIOS version?
    • Yes, with firmware fixes to address hardware security vulnerabilities (ex. Meltdown, Spectre), it is required to disclose the BIOS version within the HW Version section of the FDR.
  4. Where can I specify, within the FDR, that my submission is made with or without security mitigations (Meltdown, Spectre v1 and Spectre v2)?
    • It is required to report security mitigation information in the HW Notes section of the FDR.
    • See that section for details on what to report.
Contacting SPEC
  1. I have a question or problem that isn't covered in the documentation.


Disclaimer

Product and service names mentioned herein may be the trademarks of their respective owners.
Copyright (c) 2007-2020 Standard Performance Evaluation Corporation (SPEC).
All Rights Reserved.