Standard Performance Evaluation Corporation

Software Requirements

1. What Operating Systems are supported?
   • On the SUT side, the workload will run on any OS for which a JVM is available.
   • On the CCS side, the CCS software and the PTDaemon has been tested on common Linux distribution, Microsoft Windows XP and Server 2003/2008 (32 and 64 bit), and Sun Solaris.
     • The power and temperature measurement devices you choose may or may not have drivers for your intended operating environment. Best to check.
2. What other software will I need?
   1. A Java virtual machine (JVM).
      • The workload and the CCS are written in Java to enable the highest degree of portability across software operating environments.

Acquiring the Benchmark

1. How much does the benchmark cost?
   • The cost of the benchmark and the order form are on the web at: http://www.spec.org/order.html
2. How do I order the benchmark?
   • You will find an order form at: http://www.spec.org/order.html

Contacting SPEC

1. I have a question or problem that isn't covered in the the documentation.
   • Send your query via e-mail to: [email protected] with "SPECpower" in the subject line.

Running / Troubleshooting the SPECpower_ssj2008 Benchmark

Set Up and Starting

1. What do I have to change just to get started?
   • See the document "Quick Start Guide"
2. How many JVM instances should I use?
   • Typically, with 1 to four cores per processor socket (or package), best parallelism is achieved with one JVM per socket.
   • Each JVM instance will require a dedicated Java "heap"
   • Good practice is to look at published SPECpower_ssj2008 reports for many common server platforms at: SPECpower_ssj2008 Results Reports (http://www.spec.org/power_ssj2008/results/power_ssj2008.html)
3. What do the terms "Xmx" and "Xms" mean?
   • They are Java command line options that control Java heap size.
   • "Xms" is used to specify the minimum heap size
   • "Xmx" is used to specify the maximum heap size
   • For further information consult with the JVM vendor.
4. How important are the options used with the JVM in the JAVAOPTIONS_SSJ environment variable in the runnssj.bat (Windows) or runssj.sh (Linux) file?
   • Java options can be critical to the efficient operation of Java on a particular hardware platform.
   • We have seen performance difference of up to 50% between zero options and a full set of heap size and optimization flags.
   • Good practice is to look at published SPECpower_ssj2008 reports for many common server platforms on the SPECpower_ssj2008 Results Reports (http://www.spec.org/power_ssj2008/results/power_ssj2008.html)
   • Look for "JVM Command Line Options" in the Software Configuration section.
   • Note that options used with one brand of JVM are NOT expected to work with another brand, and sometimes not with a different version of the same JVM.
   • Options prefixed by "-X" are not standard.
   • Consult with the vendor of your JVM.
   • SPEC can not offer specific tuning recommendations.

Controller System

1. Can I use a laptop or desktop system for the Controller System?
   • Yes; with either type of system, on Windows OS, make sure to:
     1. set your Power Scheme to "Maximum Performance" and,
     2. screen saver to "none" and,
     3. set "Power Options" (plugged in) to System Standby"Never"
2. What are the minimum system requirements?
   • Most modern machines are suitable
   • Minimum processor speed of ~1GHz,
   • Minimum 1GB memory,
   • Ethernet network capability with TCP/IP installed,
   • ~10MB free disk space for code and log files,
   • No special graphics requirements.
3. Can I change the names of properties files used by the benchmark?
• Yes. You will need to specify the particular property file name in the script files either through an environment variable, a command line parameter, or both (depends on the script construction).
4. Does CCS have command line parameters?
• Yes. One; the name of the CCS properties file.
• The default name is "ccs.props" and is not required to appear on the command line.
• If you choose to use another name, simply add that name to the command line that starts CCS (assuring there is only one filename).
5. Is the file SPECpower_ssj_EXPERT.props used in the benchmark?
• No. This file contains 'properties' that can change the default behavior of the benchmark.
• For instance, the number of load levels, duration of load levels, delay between load levels, idle period settings and many other options.
• Use these parameters with caution.
• Chosen parameters can be inserted into your controlling properties file.
• Changes from the defaults will produce warnings in the reports.
• Results from runs where defaults were changed are not compliant and should not be considered comparable to compliant runs.
6. Can I change the duration of the load levels? (from the default of 240 seconds)
• Yes.
• Add the line below to your "SPECpower_ssj.props" file to set the length of each load level to 60 seconds
• input.load_level.length_seconds=60
• Idle and Calibration load levels are controlled by other properties.
  • Load level durations of less than 30 seconds are not recommended.
• This and other changes to the defaults can be found in the file
"SPECpower_ssj_EXPERT.props" in the "\ssj" directory.
• Results from runs where defaults were changed are not compliant and should not be considered comparable to compliant runs.

Power Analyzers

1. Is a power analyzer required to run the benchmark?
   • For compliant runs, yes.
   • For any other runs, the SPEC PTDaemon (Power Temperature Daemon) has a device type "dummy" for these occasions.
   • You may also remove the power analyzer data source from the ccs.props file.
2. Can the SPECpower_ssj2008 benchmark support multiple power analyzers?
• Yes. Multiple power analyzers and multiple temperature sensors are supported.
• Data collection devices are defined in the "ccs.props" file. The default ccs.props file contains an example of defining a second power analyzer.
• See the "CCS Design Guide"; the section on "Data Sources" provides instructions: CCS Design Document (http://www.spec.org/power/docs/SPECpower_ssj2008-Design_ccs.pdf)
• Note that an instance of the SPEC PTDaemon is required for each power analyzer or temperature sensor.
• Each PTDaemon instance must have a TCP/IP port number that matches that defined in the ccs.props file.
3. What is the difference between *tested* and *accepted* power measurement devices?
• Tested devices have operated successfully with the PTDdaemon. Not all are accepted for compliant benchmark runs. They are available for _research_ purposes.
• SPEC Accepted devices have been qualified by SPEC for use in producing compliant benchmark reports and submissions to SPEC. These devices have passed the Power Analyzer Acceptance Test.
4. What power analyzers have been *accepted* for use with the SPEC Power Benchmark?
   • SPEC Accepted power measuring devices are listed on the SPEC web site at: Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)
   • This same list will be displayed by the PTDaemon if run with no parameters; the PTDaemon will also display "help" text.
   • Under "Power Analyzer device types" are listed those power analyzers compliant with the SPEC accuracy and functionality requirements.
   • Run PTDaemon with the -u parameter (-u = unsupported) to see the complete list of power and temperature analyzers (and help text).
5. What power analyzers have been tested with the SPEC Power Benchmark?
   • A list of SPEC Accepted (supported and compliant) power measuring devices are documented on the SPEC web site at: Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)
   • The PTDaemon program can be used to display a list of power and temperature measuring devices along with their device codes;
   • For SPEC Accepted devices: Run the PTDaemon with the -h parameter, or zero parameters.
   • Listed devices (except 0/1000. Dummy) are valid for compliant runs of the benchmark.
   • To view a list of SPEC Accepted (supported) devices and unsupported devices, run the PTDaemon program with the -u parameter (-u = unsupported).
   • Listed devices annotated with an * are:
     • unsupported and not valid for compliant results
     • used with caution
6. Will my power analyzer work with the SPECpower PTDaemon even though my device does not appear on the accepted devices list shown on the SPEC web site at Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html)?
   • No, only devices on the SPEC Accepted Device List are expected to produce valid results.
   • If your device is not present on the devices list, it is unlikely that PTDaemon will successfully read data.
   • Communication protocols are quite different among the various manufacturers and models of power analyzers.
7. Will the SPECpower PTDaemon (Power Temperature Daemon) connect to and read my power analyzer even though my device is not on the tested list?
   • It is unlikely that PTDaemon will successfully read a meter not on the "tested" list.
   • See the responses to the FAQ just above.
8. How can I get my power analyzer included in the SPEC accepted devices list?
   • The process for getting a new measurement device accepted for use with SPECpower_ssj2008 is described briefly in section 2.13.5 of the run rules.
   • Basically, documentation must show that the device has sufficient accuracy, code must be written to enable SPEC's PTDaemon software to read measured values from the device, and then testing must be performed to show that the device does meet SPEC's requirements in actual usage. Even if the device doesn't meet the SPEC requirements, it may still be possible to add software support such that the device could be used for internal (non-publishable) measurements.
9. I have (or plan to buy) a multi-channel power analyzer. Can I use all three channels with the SPECpower benchmark?
• No. Multiple simultaneous independent channel operation (reads from) is not yet supported due to the design of the power analyzers.
• This feature is planned for a future release.
10. Will a power analyzer calibration certificate for one device apply to other power analyzers of exactly the same model?
• No. The calibration certificate issued is valid only for the specific device (one serial number) calibrated.
• A recently calibrated power analyzer is required to produce and report compliant results. "Recent" means that the calibration certificate must be dated within one year of the test date.
11. Is the test certificate I received with my instrument sufficient to meet the calibration requirements of the power analyzer requirements section of the run rules?
• If the certificate is entitled "Certificate of Calibration" or "Certificate of Traceable Calibration" and includes a unique calibration number or sticker, lab accreditation to ISO or other standards, and test equipment certification with traceability to a national standards institute, it probably meets the requirements. National standards institutes include NIST (USA), PTB (Germany), and NML (ROC - Taiwan) among others.
• See the SPECpower_ssj2008 "Run and Reporting Rules" section 2.13.2
• Power Analyzer Requirements (http://www.spec.org/power/docs/SPECpower_ssj2008-Run_Reporting_Rules.html#2.13.2)
12. I can not change the voltage and current ranges once PTDaemon has been run with a Yokogawa WT210 power analyzer.
• Once PTDaemon 'connects' to the WT210, the front panel is 'locked' to avoid inadvertent changes during a SPECpower_ssj2008 run.
• To re-enable the front panel,
• Exit PTDaemon
• On the WT210 front panel, press the Stop (integrate) button then the Reset button.
• Front panel operations should be enabled.
13. Do I have to configure the power measurement device manually?
    • This depends on the device brand and/or model.
    • Some devices need to be configured manually via the front panel; for example communication settings or current and voltage ranges.
    • Other devices will be controlled by the SPEC PTDaemon software
    • For more details on device setup, see the document Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
14. No data cable came with my power analyzer (RS-232), what now?
• Each brand/model power analyzer may require a different cable. some experimentation might be required.
• Some power analyzers will work with a "null modem" cable.
• One combination that has worked with some devices is a standard "Modem" cable with the addition of a "NULL MODEM" adapter.
• The above combination can also be plugged into a RS-232 to USB converter.
• Some data cable information can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
15. My power analyzer supports only RS-232 for data logging; my control system has no serial ports; neither DB-9 nor DB-25.
    - What do I do?
    • There are devices that convert RS-232 to USB, that cost $10-14 (US$).
    • Several brands of serial to USB converters have been found to work well.
    • With Windows, install the software per manufacturer instructions.
    • Then, use the device manager to see what "COM" port number was assigned.
    • Edit the runpower.bat file; change the default DEVICE_PORT number.
    • Note that the COM port assignment on your system may change if you disconnect and re-connect a device or change USB connections.
16. The PTDaemon software can't connect to my power analyzer.
    - What is wrong?
    • First, assure the analyzer is communicating on the data cable with the host machine.
      • The best way to verify this is to first use the vendor supplied software to read the analyzer.
    • Most problems can be resolved with this method, problems such as cable not connected, wrong cable, incorrect device settings and so on.
    • If the vendor supplied software shows the correct readings, assure that the following are correct:
      • In the ccs.props file; the IP address or hostname - usually "localhost"
      • In the ccs.props file the network (TCP/IP) port number must match that in the "runpower.bat" file.
      • The default is NETWORK_PORT=8888 (for power analyzer 1)
      • The analyzer device type used when the PTDaemon is started.
      • The default is DEVICE=0 (the dummay power analyzer)
      • If your problems persist, send e-mail to SPECpower support at this e-mail address: [email protected] with a subject of SPECpower.
17. Why does my power analyzer display unexpected watts readings?
    • This is likely to be traced back to how the analyzer is connected or the device settings.
    • More details on device setup can be found in the Measurement Setup Guide (http://www.spec.org/power/docs/SPEC-Power_Measurement_Setup_Guide.pdf)
    • Instructions will also be found in the power analyzer documentation or on the vendor web site. You may also contact the device vendor for further technical support.
18. Why do I see -1 (or -2) in the watts column in the ssj.nnnn.ccs-log.csv log file?
    • This indicates an error acquiring data.
    • -1 indicates the data is unreadable
    • -2 indicates a communication or timing error
19. What is wrong when I see very very large numbers recorded for watts (I have a Yokogawa WT210 Power Analyzer)?
• Your analyzer is in the "wrong mode".
• Reset your analyzer to factory default settings by following these steps:
• To reset the Yokogawa WT210 to initial values:
  1. turn the device off.
  2. press and hold enter key (on front panel), while powering on the analyzer, holding the enter key down until errors display (which is normal) finishes.
  3. done.
• Next it is best to assure the RS-232 (serial) communications are set up properly .
• This is not required for GPIB devices
• Yokogawa WT210 Power Analyzer - Front Panel Setup
  1. Press and release the “local” key
  2. in display window “C” use the Up/down arrow keys to select 488.2; Press enter
  3. In display window “A” use the Up/down arrow keys to select “hand 0”; Press enter
  4. In display window “B” use the Up/down arrow keys to select “For 0”; Press enter
  5. In display window “C” use the Up/down arrow keys to select “b 9600”; Press enter
  6. In display window “C” use the Up/down arrow keys to select “CR”. Press enter
20. What is wrong when I see errors relating to uncertainty not being calculated?
    You may have autoranging enabled on your analyzer, which is not supported and may not be selected.

Temperature Sensors

1. How do I run without a temperature sensor?
   • The PTDaemon (Power Temperature Daemon) has a sensor type of "dummy".
     • this is the best temporary fix if you will add or use a sensor in other runs.
     • The device code for "dummy" temperature sensor is 1001.
   • You may also remove the temperature sensor data source from the ccs.props file.
   • comment the lines that define the temperature sensor (# as first character)
2. What temperature sensors have been tested with the SPEC Power Benchmark?
• A list of successfully tested and compliant temperature measuring devices are listed on the SPEC web site on the Accepted Measurement Devices (http://www.spec.org/power/docs/SPECpower-Device_List.html) page.
3. How do I find the "device code" for a temperature sensor?
• Run the PTDaemon program with zero command line parameters or -h.
• A list of temperature sensors and power analyzers will be displayed, along with the 'help' information.
4. Help! My Digi Watchport/H meter worked fine last week, this week I get this set of errors from the power daemon.
   -- Serial Port Create File failed with error 2, serial port open failed with error 0, Failed to connect to meter
   -- When I look at the Ports with the device manager, I see this message in the box on the "general" tab: Windows cannot identify this hardware because it does not have a valid hardware identification number. (Code 9)
   --- FYI, Digi Watchport driver is version 4.2 for Windows XP.
   • Answer:
   • Unplug the Watchport USB connection, count to 10, plug it back in.
   • This problem sometimes occurs after the system has been re-booted.
   • At this time (27 Aug '07), we are not sure the root cause.

General Troubleshooting

1. CCS terminates with error such as: Can't find "SCmain".
   • This problem is most likely due to an issue with resolving the path to the JVM.
   • Check the path to "java.exe"
2. What if the PTDaemon can't connect to the power analyzer?
   • See FAQ in the Power Analyzer section
   • Check COM port and the analyzer type passed to the power daemon (ptd_xxxx.exe)
   • Assure the analyzer is connected via serial or USB cable and the analyzer is powered on.
3. One of the benchmark components is "waiting for connection" - forever.
   • Check the IP address of the "director" in the ccs.props file.
   • On the SUT, check the IP address of the "director" in the runssj.bat (or .sh) script file
   • This environment variable: set DIRECTOR_HOST=192.168.1.2
   • The CCS system and the SUT system should be able to 'ping' each other.
   • Check the IP address and TCP/IP port number in the ccs.props file and assure it is the same port number that is passed to the PTDaemon (runpower.bat).
     • In most cases the CCS software and the PTDaemon will be running on the same system. There should be no changes made to the ccs.props file in this case.
   • Kill or terminate all the SPECpower software and re-start.
4. The Command window immediately closes (Windows) when I execute a batch file (by double clicking on the file name).
   - How do I debug this?
• If you execute a runxxx.bat by double clicking on the file name , a command window will appear then close upon program termination.
• If an error message is generated, it disappears with the command windows.
• Instead, first open a command window (using a script file as suggested below) then, in that window, enter runxxx.bat.
• The window will stay open and allow examination of the activity.
5. How can I quickly open a command window in the current directory?
• One easy way is to create a script file in each directory in which you need to open command windows.
• On Windows systems put the following (example) in a script file:
• start "- SSJ -" cmd
• The above will open a command window in the directory from which the script file is run, and, as a bonus, the window will be "labeled" by the text in the string within the quote marks.
6. What's wrong when my benchmark run seems to complete, but no reports are created?
   - I find a "raw.partial" file in the SPECpower_ssj2008\results directory.
• See the following FAQs on Out of Memory Errors.
7. The benchmark failed during report generation, my reports are blank.
   - What's wrong?
• Look at the messages generated by the Director - the most common cause of this issue is an Out Of Memory Error.
• This is the case where "raw.partial" file is found in the results directory
• the Director messages can be found in the log file named: ssj.nnnn.director.log
• If this is the case, the maximum heap size (JVM Options -Xmx) is likely too small.
• A quick fix is to set the JVM Options to blank in the script file that runs the director.
• Set JAVAOPTIONS_DIRECTOR=""
• This will cause the JVM to dynamically change heap size.
• If the director is running on the CCS system, benchmark performance will not be affected.
• The director is run from script files rundirector.bat / rundirector.sh or in the runssj.bat (.sh)
• For run setups where the director runs on the SUT, edit the file runssj.bat /runssj.sh.
• The memory requirements of Director are related to the number of JVMs it is directing. The exact memory requirements depend on the type of JVM, operating system, number of JVMs, number of nodes and other factors.
8. Why I am getting "Out of Memory Errors" from Java?
• A benchmark run may fail in the "reporter stage" if the JVM heap size options for the "director" are set too low and it runs short of memory.
• You may find "out of memory errors" in the command window - scroll back.
• One solution is to remove the Xmx and Xms parameters. This enables dynamic heap allocation:
• in the file ssj\runssj.bat (runssj.sh on Linux)
• find this line "set JAVAOPTIONS_DIRECTOR=-Xms64m -Xmx256m"
• Change it to: "set JAVAOPTIONS_DIRECTOR="
• this will allow the JVM to dynamically allocate heap space.
• Save the script file and run again.
• You may have too little physical memory.
• You should have ~512MB to ~1GB free memory on the CCS system before starting the benchmark components.
9. My results report shows one or more warnings referencing a discrepancy between elapsed nanoTime and elapsed currentTimeMillis.
   - Does this warning disallow publication of the result?
   - What is the recommended action?
• Discrepancies between nanoTime() and currenTimeMillis() arise from the differences in the way these calls are implemented by various JVMs and platform hardware architectures.
• The discrepancies are reported by the benchmark to alert you to a possible area of concern, especially when reviewing the disclosure (reports)
• and, for the benefit of those testing the benchmark on new hardware or software.
• As warnings, they do not invalidate the benchmark run.
• You may submit a benchmark disclosure (with these warnings) for acceptance by SPEC. It will not necessarily be found invalid due to the presence of these warnings alone.
  • If the disclosure is accepted by SPEC and published at spec.org, the warnings will not appear in the published report.
• While minor discrepancies between nanoTime() and currentTimeMillis() have not been considered an obstacle to disclosure publication in the past, each result is reviewed independently and reviewers have the right to question discrepancies if they so choose.
• If the warnings are a concern to you, you may find that less aggressive performance tuning may reduce their incidence.
10. After submitting results I received feedback notifying me that I needed to change one or more parameters and re-run the test. However, when I did so I got the same results as before � SPECpower_ssj2008 ignored my changes.
    Changing the contents of the configuration file SPECpower_ssj.props changes how the test will execute. Changing the contents of SPECpower_ssj_config.props and SPECpower_ssj_config_sut.props only changes how the final report will be generated and has no effect on the actual execution of the tests.

SPECpower_ssj2008 Results Reports

1. Where are SPECpower_ssj2008 results cataloged on the SPEC web site?
   • Results disclosures are at this URL: http://www.spec.org/power_ssj2008/results/
2. How often are new results posted?
   • New disclosures are added approximately every two weeks.
3. Can I "turn off" compliance checking?
   - My results are only non-compliant because I do not have a temperature sensor.
   • No. This is to prevent public use of a Full Disclosure Report that is or might be non-compliant.
4. On the SPECpower report, if I compute my own performance to power ratio, why do I sometimes get different results for some load levels?
   - I've seen a difference of up to two points.
• This is due to the higher precision of the measured results used to compute the ratios that appear in the report. This is an artifact of rounding.
• The ratios in the reports are correct.

Results Publication