General Information

1. What is SPECjbb2000?
2. Who will use SPECjbb2000 and how will they use it?
3. Why is the benchmark important for vendors?
4. Why did SPEC decide to develop SPECjbb2000?
5. Does this benchmark measure end-to-end Java performance?
6. Does SPECjbb2000 represent real applications?
7. What specific aspects of performance does SPECjbb2000 measure?
8. What factors affecting Java performance are not measured by SPECjbb2000?
9. Why doesn't SPECjbb2000 cover AWT/graphics performance?
10. Is this benchmark suitable for measuring client performance?
11. Does SPECjbb2000 use Enterprise Java Beans (EJBs), Servlets or Java Server Pages (JSPs)?
12. Why doesn't the benchmark measure network and database performance?
13. Does this benchmark replace any other benchmark?
14. Can I compare my SPECjbb2000 results with my TPC-C results?
15. Can SPECjbb2000 results be compared to results from other SPEC benchmarks?
16. Is this related to any of IBM's BOB benchmarks?
17. Can I use this application for non-benchmarking purposes?
18. Do you provide source code for the benchmark?
19. Does floating point precision matter for this benchmark?
20. Are SPECjbb2000 results repeatable?
21. Where can I find out more about the benchmark?
22. What metrics does SPECjbb2000 use to report performance?

    Setting up SPECjbb2000

23. What are the requirements for running SPECjbb2000?
24. What other software do I need? Web server, database, proxy server?
25. Can SPECjbb2000 run under Java 2?
26. Can I use this benchmark for embedded processor platforms?
27. Can I run SPECjbb2000 on a uni-processor?
28. Can I run SPECjbb2000 on a shared memory processor (SMP) configuration and will it scale?
29. How long does it take to run the SPECjbb2000 benchmarks?
30. How much JVM heap memory is required?
31. How much system memory is needed to run the benchmarks?
32. Why can't I recompile this benchmark?

    Running SPECjbb2000

33. What do you mean by warehouses, points and threads?
34. Is there a warm-up period before each measured point?
35. Where can official SPECjbb2000 results be obtained?
36. Can I run SPECjbb2000 on a cluster configuration and compare results with SMP configurations? If not, why?
37. Can SPECjbb2000 help users compare Java and C++ performance?
38. Does garbage collection affect SPECjbb2000 results?
39. I see a lot of paging and disk I/O but I had a compliant run. Can I publish my results?
40. Why is it necessary to look at the SPECjbb2000 graph to get a good idea of performance?
41. My graph looks strange. Are there two graph formats?

    Ordering SPECjbb2000

42. What do you get when you order SPECjbb2000?
43. What is the price of a SPECjbb2000 license and when will it be available?
44. How do I contact SPEC for more information?

General Information

A1: SPECjbb2000 is a Java program emulating a 3-tier system with emphasis on the middle tier. Random input selection represents the 1st tier user interface. SPECjbb2000 fully implements the middle tier business logic. The 3rd tier database is replaced by binary trees.

SPECjbb2000 is inspired by the TPC-C benchmark and loosely follows the TPC-C specification for its schema, input generation, and transaction profile. SPECjbb2000 replaces database tables with Java classes and replaces data records with Java objects. The objects are held by either binary trees (also Java objects) or other data objects.

SPECjbb2000 runs in a single JVM in which threads represent terminals in a warehouse. Each thread independently generates random input (tier 1 emulation) before calling transaction-specific business logic. The business logic operates on the data held in the binary trees (tier 3 emulation). The benchmark does no disk I/O or network I/O.

For additional details, refer to the SPECjbb2000 white paper.

A2: Buyers of middle-tier servers can compare different platforms both in terms of differences in hardware and software (e.g., JVM, JIT, and OS).

A3: SPECjbb2000 represents a middleware application written in Java. Hardware vendors can use the benchmark's results to analyze their platforms' scalability when running Java applications. Software vendors can evaluate the efficiency of their JVMs, JITs, garbage collectors and thread implementations.

A4: SPEC has an existing Java client benchmark, SPECjvm98, and wanted to offer a Java benchmark from the server end. There are a few Java server benchmarks available, but SPECjbb2000 is the only one that provides SPEC's methodology for ensuring consistent, repeatable results.

A5: No. Please see the answers that follow for areas not covered by SPECjbb2000.

A6: Yes. Many companies have developed Java application frameworks for deploying large commercial systems. SPECjbb2000 models the business logic and object caches that make up the middle tier of these frameworks. The emulation of the other tiers isolates this tier and simplifies the benchmark by not requiring user emulation or a database. The implication is that combining a JVM with a high SPECjbb2000 throughput and a database tuned for online transaction processing will provide a business with a fast and robust multi-tier environment.

In developing this benchmark, SPEC worked toward a relatively flat method profile to avoid overly good optimization candidates. SPEC also compared the method profiles to other Java server applications within the member companies to make sure that SPECjbb2000 behaves like a server application.

A7: SPECjbb2000 measures the implementation of Java Virtual Machine (JVM), Just-in-time compiler (JIT), garbage collection, threads and some aspects of the operating system. It also measures the performance of CPUs, caches, memory hierarchy and the scalability of Shared Memory Processors (SMPs) platforms on the specified commercial workload.

A8: AWT, network, I/O and graphics performance are not measured by the benchmark.

A9: SPECjbb2000 doesn't cover AWT/graphics because this is not considered server work.

A10: No.

A11: No.

A12: A two- or three-tier configuration would be necessary to measure network and database performance. This would be a complex and expensive configuration to test. Many other benchmarks exist to measure database and network performance.

A13: No.

A14: No, absolutely not. SPECjbb2000 is memory-resident, uses totally different data-set sizes and workload mixes, performs no I/O to disks, and has no think times. It has a different set of run and reporting rules, a different measure of throughput, and a different metric. Such comparison would be a SERIOUS violation of SPEC's run and reporting rules and of TPC's "fair use policy." Violations are subject to penalties.

A15: No. There is no logical way to translate results from one benchmark to another.

A16: SPECjbb2000 is descended from IBM's pBOB benchmark. However, SPEC has extensively modified both the design and the code during the development of SPECjbb2000. Therefore, SPECjbb2000 is definitely not comparable to IBM's pBOB or any other BOB benchmark you may have gotten from IBM.

P.S. Many thanks to IBM for providing the original code.

A17: No.

A18: Yes, but you are required to run with the jar files provided with the benchmark. Recompilation is forbidden in the run rules and will invalidate your results.

A19: No. The SPECjbb2000 workload contains primarily integer computation.

A20: Yes. Measurement on a variety of platforms shows the SPECjbb2000 metric has a 95% confidence interval of +/- 2% of the score.

A21: The SPECjbb2000 kit comes with a User's Guide, overview of the SPECJBB2000 architecture and implementation, and Run and Reporting Rules. These same documents are also on the SPECjbb2000 page on the SPEC web site.

A22: SPECjbb2000 ops/second is a composite throughput measurement representing the averaged throughput over a range of points. It is described in detail in the document "SPECjbb2000 Run and Reporting Rules."

Setting up SPECjbb2000

Q23: What are the requirements for running SPECjbb2000?

A23: The user needs a machine with a minimum of 256 MB physical memory, and a JVM environment supporting the Java or Java 2 APIs. JVMs generally need 32 MB or more of disk space for the installed software. The benchmark itself requires 10 MB of disk space. For competitive numbers, more than 196 to 300 MB of physical memory is almost certainly needed, depending on your JVM.

A24: None of these. The benchmark does not use a web server, database or any other additional software besides the JVM.

A25: Yes.

A26: No.

A27: Yes, but you will need to run all points from one to eight warehouses to get a reportable result.

A28: Yes. In theory, the peak throughput point in the SPECjbb2000 benchmark will occur when the number of warehouses (or threads) equals the number of CPUs in an SMP system. This may not always be the case depending your JVM, operating system and hardware. During the development of SPECjbb2000, scaling up to 12 CPUs was measured on more than one hardware platform and JVM.

A29: A one-to-eight warehouse SPECjbb2000 run takes close to half an hour. Specifically, it takes just about three minutes per point measured, where a point is a number of warehouses. It measures throughput in this fixed amount of time, so faster machines do more work in the allotted time.

A30: Of the JVMs tested, the heap sizes ranged between 196 MB and 300 MB to run up to 8 warehouses. Typically, however, performance increases as you add more heap.

A31: Your system memory should always exceed the maximum heap size for the JVM. Most systems will need about 64 MB memory in addition to the java heap in order to run SPECjbb2000 without swapping.

A32: The intent of this benchmark is to measure the performance of the Java platform, given a known set of application bytecodes. Therefore, you may not recompile the source code if you want to produce a valid result.

If, however, you wish to experiment with the code, be aware that to successfully compile the spec.reporter package you will need a Java 2 developer kit that contains the com.sun.image.codec.jpeg package.

Running SPECjbb2000

Q33: What do you mean by warehouses, points and threads?

A33: A warehouse is a unit of stored data. It contains roughly 22 MB of data stored in many objects in many Btrees.

A thread represents a terminal user within a warehouse. There is a one-to-one mapping between warehouses and threads plus a few threads for SPECjbb2000 main and various JVM functions. As the number of warehouses increases during the full benchmark run, so does the number of threads.

By "point", we mean a two-minute measurement at a given number of warehouses. A full benchmark run consists of a sequence of measurement points with an increasing number of warehouses (and thus an increasing number of threads).

A34: For each point, the benchmark creates data for that many warehouses, then starts the threads making requests as fast as possible. The threads run for 30 seconds (the warm-up) before starting the two-minute measurement. This warm-up may get records into memory, but is probably not as influential as warm-up time on an actual database benchmark.

A35: Results are available at http://www.spec.org/osg/jbb2000/results/ SPEC licensees may also publish their own results in compliance with the latest SPEC Run and Reporting Rules on the web site.

A36: No, SPECjbb2000 was not designed to be an appropriate benchmark for clustered environments.

A37: No. SPECjbb2000 is a JVM server benchmark, not a language benchmark.

A38: Yes. Garbage collection definitely affects throughput, especially as the throughput increases. Increasing heap may lessen the effects of garbage collection in your tests.

Based on the throughput, the heap size, and whether input.forcegc is set to true or false, it is possible to decide how active the garbage collection is when measuring the SPECjbb2000 performance.

For SPECjbb2000, the size of the garbage generated during each warehouse run is determined by the throughput in terms of OPS/sec. Based on our measurements, there are about 270KB of garbage for every SPECjbb2000 OPS/sec during the 120 seconds when the performance is actually measured. For example, for a test running at 10,000 OPS/sec, there is about 2.7GB of garbage that needs collection during the 120 seconds when the performance is being measured.

The garbage is made up of short lived objects created during each warehouse run, and can be effectively collected during and after the run. You may find that forcing garbage collection between run points influences performance. You may set the property input.forcegc to "true" or "false" to control whether the test will force a garbage collection between each run point. If there is a forced garbage collection in between warehouse runs, the garbage is practically cleaned up before each warehouse run.

Your JVM may offer you flags to specify garbage collector algorithm or parameters. The -verbosegc switch will tell you more about the characteristics of your garbage collector during this benchmark.

A39: Yes, but it would be unwise. This is a memory-resident server benchmark and you should have adequate memory for your OS, JVM, the benchmark and heap. If you observe a lot of disk activity, your results will be non-competitive and should not be published.

A40: SPECjbb2000 measures the efficiency of the JVM, JIT compiler and garbage collector. As the amount of parallelism increases with more warehouses (and therefore more threads), the balance between these components changes, particularly when you get to more threads than CPUs. The shape of the curve shows you whether the system handles overload gracefully.

A41: The reporter tools can make two types of html: html with an html graph, and html with a jpg graph. The jpg graph creation requires Java 2 and, under UNIX, the DISPLAY environment variable set to a working X display (this may require setting xhost + on the display). The html graph is provided for those without Java 2 or without the display device.

If, initially, the result appears in the html with an html graph format, you can make the other format later (after adding Java 2 or an X host display) by rerunning the reporter. See the User's Guide for instructions. Compliant results sent for publication to SPEC will appear on the SPECjbb2000 website in the html with a jpg graph format.

Ordering SPECjbb2000

Q42: What do you get when you order SPECjbb2000?

A42: You get a CD-ROM of the SPEC-licensed product.

A43: SPECjbb2000 is available for $400. Academic institutions may be eligible for a discounted price of $200. Contact the SPEC office to order a copy or use the online product order inquiry form

A44:
SPEC (Standard Performance Evaluation Corporation) 
6585 Merchant Place, Suite 100
Warrenton, VA 20187, USA
Phone: 1 540 349 7878
FAX: 1 540 349 5992
E-Mail: [email protected]

• Kaivalya Dixit
• Anne Troop
• Rajiv Arora
• Robert Reynolds
• Steve Munroe
• Walter Bays
• Ruth Morgenstein

and others.