Standard Performance Evaluation Corporation

Code Defect Identified in the Windows Port of the SPECweb99_SSL Test Harness

July 22, 2003 -- SPEC has identified problems with the port of the SPECweb99_SSL test harness to the MS Windows operating system. The code defects prevent the client systems from generating the correct SPECweb99_SSL workload. It should be noted that the problems are solely within the the port of the SPECweb99_SSL code base to the Windows OS and not due to either the Windows OS or the OpenSSL library. Specifically, the WIN32 client_ssl.exe fails to generate any of the required SSL ClientKeyExchanges. The ClientKeyExchanges are a fundamental part of the SPECweb99_SSL workload. Therefore, test results generated using this version of the SPECweb99_SSL client code are not comparable to any other SPECweb99_SSL results.

If Windows clients are used to drive a SPECweb99_SSL test, then the updated version of client_ssl.exe must be used to produce a publishable SPECweb99_SSL result. To obtain this updated version, licensees should download the alternate source kit and updated client_ssl.exe from SPEC. Licensees may continue to use the existing code base for UNIX or LINUX based clients (e.g. AIX, HP-UX, LINUX, Solaris).

What Happened:

During the development of SPECweb99_SSL, an incomplete port for WIN32 was integrated into the code base. This code appeared to work under light loads. Just recently, it was reported that the client_ssl.exe was unstable when load driving SPECweb99_SSL tests for larger web servers. Further investigations showed that several important macros were not defined causing the path through the code to differ significantly from expected path. The path taken caused the WIN32 client_ssl.exe to fail to negotiate the SSL ClientKeyExchanges that the design intended. This significantly altered the workload presented to the server being tested. Also, the code failed to setup a locking callback routine for use by the OpenSSL API causing instability.

Remedy:

Licensees are advised to contact the SPEC office to arrange access to SPEC's file download system to obtain the updated version of the WIN32 client. Licensees should download the updated client_ssl.exe and alternate source kit (from which an updated client_ssl.exe can be generated) and utilize that version in all future SPECweb99_SSL tests using client systems running a Windows OS (XP, 2000, 2003 Server). A detailed description of the changes are included in the kit along with installation instructions.

SPEC regrets any inconvenience. If you have additional questions, please contact SPEC.