SPEC logo

SPECweb99_SSL 1.00 User's Guide

1 Introduction
2 Installing SPECweb99_SSL

2.1 Pre-Installation Checklist
2.2 Client Setup

2.2.1 Installing and running the UNIX client
2.2.2 Compiling UNIX client software (if necessary)
2.2.3 Windows Client Setup
2.2.4 Compiling Windows client software (if necessary)

2.3 Server Setup

3 Running SPECweb99_SSL

3.1 Customizing the rc file

3.1.1 Changeable benchmark parameters
3.1.2 Configuration description parameters
3.1.3 Benchmark Constants

3.2 Running the benchmark

4 Understanding the benchmark screen display

4.1 Test setup summary
4.2 Operational validation
4.3 Test Warnings
4.4 Results from each iteration
4.5 Screen display of ASCII output file

5 SPECweb99_SSL Result Pages and Raw File

5.1 Result pages
5.2 Raw file

6 Troubleshooting

6.1 Server system
6.2 Client system

7 Performance Tuning

8 Submitting results

1 Introduction

SPECweb99_SSL is a client/server benchmark for measuring the maximum number of simultaneous connections that a secure web server is able to support. The benchmark load is generated by client software on client machines networked to server machines running secure HTTP (HTTPS) server software.

This document is a practical guide for setting up and running a SPECweb99_SSL test. This user's guide covers some, but not all of the rules and restrictions pertaining to SPECweb99_SSL. Before running a test, you should read the complete "Run and Reporting Rules" contained in the kit. For an overview of the benchmark architecture, see the SPECweb99_SSL Whitepaper also contained in the kit.

2 Installing SPECweb99_SSL

2.1 Pre-Installation Checklist

Here is a checklist of steps to complete before installing the SPECweb99_SSL benchmark software.

  1. Decide on the hardware configuration to use for the benchmark. You need to decide on both server and client systems. There are no requirements on your choice of clients. For optimal performance, you will need enough clients and networks to saturate the server. (The client and server software can run on the same system, however this will cause poor performance)
  2. Choose HTTP server software that supports SSL; specifically HTTP over SSLv3 or TLS.  The user is responsible for obtaining the HTTPS server software; the SPECweb99_SSL benchmark CD does not include one. Apache with mod_ssl and Apache-SSL are free HTTPS daemons that could be used for this benchmark.  Your HTTPS server may require additional steps to enable SSL; please consult the HTTPS server documentation.
  3. Make sure you have enough disk space on all the machines.
    1. Client machines. The kit takes about 150MB to install. In addition, you'll need some space for various output files.
    2. Server machine(s). You will need space for the SPECweb99_SSL file set, space for the POST data log, and space for the web server log.

      3. Peak file size formulas for SPECweb99_SSL server (in Mbytes)

      Files Size formula (approximate)
      file_set (25 + ( simultaneous_connections * .66)) * 4.88 Mbytes
      post log simultaneous_connections * 0.06 Mbytes
      web server log consult server documentation

      The equations given are based on a speed of 400K bits/second and an average request size of 122K bits (based on the file sizes and the Zipf distribution used to select files). Using these values, a single connection can process a maximum of about 3.3 HTTPS operations per second.

      The post log formula assumes the default run time of 20 minutes warmup, 20 minutes run-time and 5 minutes rampdown. The post log gets zeroed out between iterations. When calculating space requirements for the web server logs, remember that the server logs do not get truncated between the 3 iterations.

      Note: Logs must be written to stable storage for a valid SPECweb99_SSL benchmark run. Stable storage refers to non-volatile storage media. In the case of solid state disks, they should have battery back-up.

2.2 Client Setup

This section describes the steps necessary to setup SPECweb99_SSL clients. Detailed instructions are provided for the two major operating system types: UNIX and Windows. The setup instructions for most of the hardware platforms are very similar to the generic setup instructions for the two operating system versions. The SPEC website may contain additional instructions for particular platforms.

Note: You may use a mix of Windows and UNIX-based clients in your setup, although this is not recommended.

Designate one of your client machines as the prime client machine. The prime client will control the entire test and the other clients, so it needs to be able to establish network connections to all the other clients and to the server. The prime client can be used just as the test controller or can serve as both controller and client. The following instructions assume it does both functions.

2.2.1 Installing and running the UNIX client

On all clients, do the following:

  1. Log in as a user with the appropriate permissions to install SPECweb99_SSL. Consult your operating system's documentation for instructions.
  2. Insert the SPECweb99_SSL CD into a CD-ROM drive and mount the drive.  You may need to become a superuser to mount the CD; consult your operating system's documentation for instructions. After mounting the CD, do an ls to make sure the CD is readable.
  3. Install the software  UNIX users have three choices of how to install SPECweb99_SSL:
     

     The CD also has several tar-and-gzipped precompiled kits that you can unpack to get the client software and tools. The precompiled kits have the suffix .tar.gz. The source files have the suffix .tgz (in order to make them acceptable to Windows).

    Once you have installed the pre-built software or built it from sources you will have the following compiled executables on your client:

    Client executable

    • client_ssl

    Server software that is built with the client executables

    • Cadgen99/cadgen99
    • Upfgen99/upfgen99
    • Wafgen99/wafgen99

    prime client tools

    • bin/specperl
    • bin/specssldump (for troubleshooting)
       
  4. Set Maximum Segment Size (MSS) of your network: Make sure that the TCP MSS is set to 1460 bytes (or less) on your system and the Maximum Transmission Unit (MTU) is 1500 bytes. This needs to be accomplished by platform-specific means outside the benchmark code itself. On many platforms, setting the MTU to 1500 automatically sets the MSS to 1460, which is (MTU - 40) bytes.

    5. Note: For a result to be valid the connections between a SPECweb99_SSL load generating machine and the System Under Test (SUT) must not use a TCP MSS greater than 1460 bytes. An MTU of 1500 bytes is the standard packet-size for IP over Ethernet.

  5. Start the client daemon: You will need to start the client daemon on all the clients: 
    6. Usage: ./client_ssl [options]
           -h    #   this help screen
       -D    #   debugging level
       -g    #   logical IP address to listen to (test traffic)
       -C    #   logical IP address to listen to (control port)
       -p    #   control port to listen on
       -i        use to indicate client is being run out of inetd
       -t    #   daemon idle timeout (set larger than anticipated run)
       -c        start a control session on stdin
       -d        start in background as a daemon
       -m    #   size of shared memory segment
       -w   dir  work directory
       -s    #   Deck size
       -S    #   Deck cache size

2.2.2 Compiling UNIX client software (if necessary)

The SPECweb99_SSL kit includes sources for all the tools and software needed to run SPECweb99_SSL. It also includes precompiled executables for many platforms. Occasionally, however, you may need to build or rebuild the client tools or client_ssl executable. You will need your own C compiler to build the SPECweb99_SSL tools and executables.

First, install the source code from the SPECweb99_SSL CD.  If the install.sh has trouble extracting the .tgz archives, you must either use GNU tar/gzip to manually extract the two archives in the root of the CD, or use the Java-based installer.

To build the prime client tools, follow the instructions in the <installation-directory>/tools/src/README file.

To build the client executables, you must first run ./configure (and any architecture-specific options if necessary) to build the Makefiles, then make to build the executables.

SPECweb99_SSL has been written to run on multiple platforms. You need to use the configure script to create Makefiles for your specific platform. For the most part, the configure script is smart enough to make the right decisions. But there are cases when the script cannot make the right decisions or the user may want to disable the use of certain features used in the code and instead choose an alternative method.

The following features are available in SPECweb99_SSL and can be used as arguments to the configure script to enable or disable the feature. Enabling or disabling these features affects the way the client_ssl program is built.

Type './configure --help' to see the generic options supported by configure.
Note: It should not be assumed that all these options are available or not available on a particular platform. Please check your product documentation before using these features.

--enable-rlimit
Enable the functions getrlimit() and setrlimit() to view/limit/control the consumption of a variety of system resources by a process and its children. In the case of SPECweb99_SSL, if this feature is enabled, it will allow the client_ssl program to have more open files and or sockets in a single process. The default behavior is to not use these functions
--enable-posix-threads
If a platform supports POSIX threads, the client program will be built using POSIX threads semantics. This will result in having a multi-threaded version of client instead of a multi-process version.
--enable-pthread_scope_system
When using POSIX threads, enabling or disabling this feature affects the contention-scope of the thread. Enabling this flag will enable a property of a thread (when the thread gets created) by which it will contend among system-wide resources. The default behavior is to not use this feature. Only use with --enable_posix_threads.
--enable-safe-gethostbyname
Enabling this feature means that gethostbyname() is thread-safe and can safely be used if the client is multi-threaded. Enabling this feature will not require enabling the next feature. The default behavior is to not assume the thread safe version of gethostbyname() is available.
--enable-gethostbyname_r
Enabling this feature means the reentrant/thread-safe version of gethostbyname() function, gethostbyname_r() is available on a particular platform and can be used instead of using gethostbyname() . The default behavior is to assume the existence of gethostbyname_r() .
Note: It makes sense to choose only one of safe-gethostbyname or gethostbyname_r depending on your platform.
--enable-safe-usleep
This feature needs to be considered if a platform does not support nanosleep() . In such a situation if usleep() is available on a platform and is thread-safe, it can be used in combination with sleep() . Otherwise the client program will use select() to emulate the sleep-feature. If nanosleep() is not present on a particular platform and if this argument is not passed to the configure script then the source code will use usleep() / sleep() regardless of whether it is thread-safe or not.
--disable-nanosleep
This feature is needed to disable the use of nanosleep() explicitly in order to override the default behavior.

 

Compile the benchmark software by typing the following command. 

make all

This will build the following executables:

./client_ssl
./Cadgen99/cadgen99
./Upfgen99/upfgen99
./Wafgen99/wafgen99

2.2.3 Windows Client Setup

On all clients, do the following steps:

  1. Insert the SPECweb99_SSL CD into the CD-ROM drive.
  2. Run SPECweb99_SSL.exe. This is a GUI-based installer.
  3. Alternative installation method: The CD contains 2 .tgz files with sources. web99_ssl-v100.tgz contains the client code and web99_ssl-v100-toolsource.tgz contains the SPEC tools. The files are tarred and gzipped, so you will need an archiving utility such as WinZip to extract them.  You'd then need to rebuild the files using the workspaces in the Win32 subdirectory.
  4. Set Maximum Segment Size (MSS) of your network: Make sure that the TCP MSS is set to 1460 bytes (or less) on your system and the Maximum Transmission Unit (MTU) is 1500 bytes.  The default MTU size for Ethernet on Windows is 1500 bytes, and this must not be changed.  Certain gigabit network adapters may have an option for "jumbo frames" (non-standard Ethernet frames above 1500 bytes), but this must be explicitly disabled for a compliant SPECweb99_SSL run.
  5. Start the client daemon. If you chose not to run the client on upon logon, start the client via Start/Programs/SPECweb99_SSL/SPECweb99_SSL Client.  You could also type client_ssl -Z from a command prompt.

2.2.4 Compiling Windows client software (if necessary)

The SPECweb99_SSL kit includes sources for all the tools and software needed to run SPECweb99_SSL. It also includes precompiled executables for many platforms. Occasionally, however, you may need to build or rebuild the client tools or SPECweb99_SSL client_ssl.exe executable. You will need Microsoft Visual C++ to build the SPECweb99_SSL executables and tools.

Note: in the following directions, %SPEC% refers to the installation directory.

To build the tools, such as specperl, get the tools sources from the SPECweb99_SSL CD (see alternative installation methods, above) then follow the directions in the README file in the %SPEC%\tools\src directory.

To rebuild the client software, run the SPECweb99_SSL.exe from the CD and check 'Source Code' during the GUI installer on the CD.  The installation directory will be hereafter referred to %SPEC%. This is one of the environment variables you get after running shrc.bat.

The kit contains 4 separate executables that can be rebuilt. The 4 workspace files are in: 

%SPEC%\Win32\client\client.dsw
%SPEC%\Win32\dyn_get_script\dyn_get_script.dsw
%SPEC%\Win32\Cadgen99\Cadgen99.dsw
%SPEC%\Win32\Upfgen99\Upfgen99.dsw
%SPEC%\Win32\Wafgen99\Wafgen99.dsw

Note: Executables generated from the last 4 workspaces, dyn_get_script.exe, Cadgen99.exe, Upfgen99.exe, and Wafgen99.exe are server-side utilities. It is highly unlikely that you will ever need to rebuild these.

To rebuild, open the workspace file.  Select the "Win32 Release" project under "Build/Set Active Configuration".  To build the client, select "Build/Build" or "Build/Rebuild All". Developer Studio will place the resulting executable into a subdirectory named "Release". Make sure you move the new executable to the correct place before proceeding. The kit is shipped with the executables in the following places: 

%SPEC%\client_ssl.exe
%SPEC%\server-docs\dyn_get_script.exe
%SPEC%\Cadgen99\cadgen99.exe
%SPEC%\Upfgen99\upfgen99.exe
%SPEC%\Wafgen99\wafgen99.exe

2.3 Server Setup (UNIX and Windows)

On the server, the setup involves installing the HTTPS daemon, creating and installing a certificate for SSL, creating the workload file set that will be accessed by the benchmark, and setting up the binaries for dynamic content.

The following instructions tell you how to run using the provided sample Perl implementation for the dynamic content. You may use your own API implementation in place of the sample Perl implementation. Please refer to the vendor's instructions for installation of the API binaries. The API implementation needs to conform to the specification in the "SPECweb99_SSL Run and Reporting Rules". The API implementation must be submitted with your results and will be reviewed by the SPEC web committee.

  1. Install the HTTPS daemon. Take note of the document root directory location. You will need this information for proper installation of other files on the server.
  2. Copy the server utilities to the HTTPS server's document root directory. The utilities come with the SPECweb99_SSL client kit and were either installed or built by you when you set up the clients. The following paths are with respect to the SPECweb99_SSL installation directory: 
    3. Wafgen99/wafgen99[.exe]
Cadgen99/cadgen99[.exe]
Upfgen99/upfgen99[.exe]

    NOTE: wafgen99 builds the file_set. The other two utilities are called by the dynamic code during a Reset.

  3. Create the file set. Use the wafgen99 utility provided with the SPECweb99_SSL kit to create the workload file_set on the server. 
    4. Usage: wafgen99 [options] [ [startload] targetload ]
           -C dir       Change to 'dir' before creating files
           -h           this usage message
           -n conns     Set targetload for # simultaneous_connections
           -v           Increase verbosity (currently 2 levels)
           -s           If directory exists skip it
           -t           Test Mode, no random characters in files
           -r           If directory exists remove it
           -V           validate files, -V again to validate all
           -f load      Start directory load (# simultaneous_connections)
           -F num       Start directory number

    This will create a file_set directory tree. The SPECweb99_SSL wafgen99 utility is identical to SPECweb99, so if a SPECweb99 file set exists, it is not necessary to rerun wafgen99.

  4. Configure the sample Perl implementation and place it under the document root directory The sample Perl script specweb99-cgi.pl (found in the server-docs subdirectory) implements the dynamic (GET and POST) portion of the workload.

    Compile the dynamic CGI GET (Unix only) and copy the executable under the document root

  5. Configure your HTTPS daemon to handle Perl and CGI requests  Make sure that your HTTPS daemon is configured properly to run Perl and CGI code. This may consist of associating the .pl extension with your Perl binary, enabling CGI operations, and placing the specweb99-cgi.pl and dyn_get_script in a particular place. Consult the documentation for your web server for instructions.
  6. Start the HTTPS daemon Start the HTTPS server daemon. Consult your HTTPS server documentation for instructions.
  7. Make sure your setup works using a web browser. Try some GETs to make sure that your HTTPS server is working, that your clients can access your server and that your Perl or API implementations run properly.  The following examples show some tests to ensure proper functionality.  They assume that the SPECweb99_SSL file_set created by wafgen99 resides directly under the HTTPS server document root.

    To test a static GET: From a client browser such as Netscape or Internet Explorer, specify

    https://<servername>/file_set/dir00000/class0_0

    as the URL and hit Enter (where <servername> is the name or IP address of your server).
    Note: You may get a warning about the validity of the SSL certificate; click "Yes" to accept it.

    To test a dynamic GET: From a client browser, specify

    https://<servername>/cgi-bin/specweb99-cgi.pl?/file_set/dir00000/class0_0

    as the URL and hit Enter (where <servername> is the name or IP address of your server).

    Note: for the second example to work, the following must be true: the web server must have perl installed and configured correctly; the sample Perl script must be in the cgi-bin subdirectory of the document root; the cgi-bin directory must have read and execute permissions, etc.

    In both instances you should see the class0_0 file from the web server in the browser.

    Note: This is not the complete set of operations in the test. The manager will execute one of each kind of transaction prior to launching a test. The manager will halt if it finds any errors, and you can check the appropriate .out files for error messages.

3 Running SPECweb99_SSL

SPECweb99_SSL is controlled by the manager script on the prime client. It reads a resource configuration (rc) input file. To run SPECweb99_SSL, customize the rc file to reflect your benchmark setup, then run the test from the prime client with the manager script.

3.1 Customizing the rc file

Modify the rc file on the prime client. The rc file in the SPECweb99_SSL kit is a template, containing every variable to describe a test. You may name the rc file anything you wish (and create as many as you like). The rc file has 3 main sections:

3.1.1 Changeable Benchmark parameters

SIMULTANEOUS_CONNECTIONS
Sets the number of load generators for the tests to be run. You may specify one value or a list of values. manager will run a separate test for each value in the list. During each test, N measurements will be taken, where N is the value assigned to ITERATIONS. If you use a list, each value results in a separate test with its own output files, unless ITERATIONS=1. When ITERATIONS=1, all the values in SIMULTANEOUS_CONNECTIONS get reported in a single set of output files. This is for testing and tuning purposes only. For reporting purposes ITERATIONS must be set to 3. The form <first>-<last>x<increment> can be used to run a series of tests starting at <first>, ending at <last> and incrementing by <increment>.
CLIENTS
SERVER
Clients: client-name(port)[speed]{servername}
Server: server-name as seen by the prime client
You may use multiple network adapters in a client by specifying that client multiple times with different {servername} settings and using the @@SERVER@@ form of URL_ROOT. For example:
       CLIENTS=box1{sut-fddi1} box1{sut-fddi2}
The speed setting gives relative speed of each client and the port setting is the port through which the prime client communicates with that client. For example:
      CLIENTS=client1[2] client2(2223)
client1 is twice as fast as client2, client2 will open port 2223 to receive its messages from the prime client.
URL_ROOT
URL for the directory that contains "file_set". For example, if your file_set directory is in the web server root then use "URL_ROOT=https://server1" (where server1 is the name of your web server). If you built the file_set in a directory named "specweb99_ssl" then use "URL_ROOT=https://server1/specweb99_ssl". To use multi-adapter clients use: URL_ROOT=https://@@SERVER@@/specweb99_ssl
DYNAMIC_ROOT
Or
DYN_GET_SCRIPT
DYN_CAD_SCRIPT
DYN_POST_SCRIPT
DYN_CMD_SCRIPT
The URL of the dynamic program(s) (e.g. CGI executable, ISAPI module, NSAPI module, etc). If your file_set directory is not in the web-server root then add a question mark and the URL to the directory that contains file_set.

For example, if file_set is in a subdirectory named "specweb99_ssl" then use
"DYNAMIC_ROOT=https://server1/cgi-bin/specweb99-cgi.pl?/specweb99_ssl".

You may also use separate URLs for each class of DYNAMIC operation supported: POST, standard dynamic GET, custom ad rotation (CAD) GET and Commands (fetch, reset). You may set any or all of the 4 separate variables. The unset variables will default to DYNAMIC_ROOT.

DYN_CGI_SCRIPT
You must specify the URL for DYN_CGI_SCRIPT that is a CGI-based implementation of the standard dynamic GET. This will not default to DYNAMIC_ROOT.
HTTP_PROTOCOL
Used to specify HTTP version 1.0 or 1.1. If you leave this commented out then manager makes a connection to your web server to determine what version of the protocol to use. You can set the HTTP_PROTOCOL variable in order to force SPECweb99_SSL to use a particular version of HTTP as the primary protocol. The primary protocol is used for 70% of requests (either HTTP 1.0 Keep-Alive or 1.1 Persistent connection request. The remaining 30% are HTTP 1.0 non-Keep-Alive requests. The valid settings are "HTTP/1.0" or "HTTP/1.1".
WARMUP_TIME
Amount of time in seconds to run to get to steady state prior to the 1st measurement in the test. This time is intended to warm up any caches before taking a measurement. The minimum legal value is 1200 seconds.
RAMPDOWN_TIME
Time after each iteration of the test to allow current connections and sessions to close. The minimum legal value is 300 seconds; however it may be set higher if the HTTPS server caches sessions for more than 300 seconds.
WAIT_TO_BEGIN
Time to sleep in seconds before starting each iteration. This delay is taken after the prime client tells the clients about the workload and before telling them to start working. If test startup isn't synchronizing properly, this might need to be increased.
OUTPUT_NAME
All outputs get put in ./results/<OUTPUT_NAME>.nnn.xxx where nnn is an ascending test number and xxx is the output format type (i.e. raw, ps, asc, html).
RSH,CLEANUP_WORK,CLIENT_USER,CLIENT_DIR
Optional parameters used to control removal of SPECweb99_SSL client work files from client directories before the benchmark starts. The use of this function is OPTIONAL and will not affect the validity of the run or the results. The SPECweb99_SSL client leaves work files in a work directory after each run, and these files can accumulate and interfere with normal operation. By configuring this option, you can reduce or eliminate any such problems. There are usage examples in the sample RC file

3.1.2 Configuration description parameters

The Configuration description section should contain a full description of the testbed. It should have enough information to repeat the test results, including all necessary hardware, software and tuning parameters. All parameters with non-default values must be reported in the Notes section of the description.

The configuration description has 8 categories.

Each category contains variables for describing it. For example, Hardware contains variables for CPUs, caches, controllers, etc. If a variable doesn't apply, leave it blank. If no variable exists for a part you need to describe, add some text to the notes section. The notes sections can contain multiple lines by adding a 3-digit suffix to the variable. For example, the notes_http variable can become notes_http001, notes_http002, etc.

The rc file included in the kit contains examples for filling out these fields.

3.1.3 Benchmark Constants

Any changes to the benchmark constants will invalidate the run. However, changing these constants can be useful for tuning the system.  Once you are ready for a compliant run, you can invoke manager with the -C switch; this sets the parameters to the values below to avoid setting the benchmark constants manually in the rc file.

Test timing parameters

RAMPUP_TIME
"warmup" time before 2nd through nth iteration of the test (default = 300)
RUN_TIME
measurement time for each iteration (default = 1200)
RAMPDOWN_TIME
time to wait after each iteration for connections and sessions to close (default = 300)
ITERATIONS
number of times to repeat the measurement (default = 3)

Mix parameters - control the percentage of requests of each type. Set to values between 0 and 1.

DYNAMIC_CONTENT
Percent of overall dynamic requests (default = 0.3 for 30% dynamic requests)
DYNAMIC_POST
Percent of dynamic requests that are posts (default = 0.16).
DYNAMIC_CAD_GET
Percent of dynamic requests that are Custom Ad Rotation GETS (default = .42)
DYNAMIC_CGI_GET
Percent of dynamic requests that are GETs calling the CGI code (default = .005)
Default transaction mix
Request type Percentage of mix
Static GET 70.00%
Standard Dynamic GET 12.45%
Dynamic GET with Custom Ad Rotation 12.60%
Dynamic POST 4.80%
Dynamic GET calling CGI code 0.15%

Other parameters

MAX_FILE
Number of files in each class. (default = 8)
USER_LINE_SPEED
Target line speed to emulate in bits/second (default = 400000)
USER_SPEED_LIMIT
Speed in bits/second below which a connection is deemed too slow. For a valid test, 95% of the connections must operate faster than the USER_SPEED_LIMIT (default = 320000)
REQUESTS_PER_KEEP_ALIVE
Average number of requests to do per "keep-alive" HTTPS connection. For example, if a connection randomly selects 5 requests to keep alive it will open a connection to the web server, establish an SSL session, request a page, read the server response, request a different page and read the response 4 more times before shutting down the SSL session and closing the socket connection.  Over the span of the KA connection, the client will attempt to use SSL session resumption so SSL session IDs will not have to be renegotiated.  (default = 10)
PERCENT_KEEP_ALIVE_REQUESTS
Percentage of GETs that to retrieve via "keep-alive" connections. Note that this is NOT the percentage of connections that will have the keep-alive header. (default = 0.7)
ABORTIVE_CLOSE
Set to 0 or 1 to indicate whether or not to use abortive closes. (default = 0)

3.2 Running the benchmark

Before running the benchmark check that the following are true:

Before running the benchmark, you must set up the SPEC environment by running the shrc file on the prime client. You only need to do this once per login session.

On Windows:

or

On UNIX:

From the session you set up with shrc start the benchmark by executing the manager script with specperl: 

        specperl manager [options]

manager tests all the operations on the server, tells all the clients what to run, tells them to run it, waits until the clients finish, then collects the results and creates the results summary in various output formats. 

Usage: specperl manager [options] [option=val...] [rc file...]
    -h           this help message
    -v   #       verbosity level
    -o   types   Produce output in the specified types
    -I           Do not abort on certain classes of errors
    -R           files are reformatted and not used as config files
    -l           Turn off client logging in log-client.nnn file
    -C           set all parameters need for a compliant run
    -D           turn on details in the ASCII output

The default input rc file is rc. You may specify your own rc file or files on the command line. If multiple rc files are used, they are run one after another, producing separate output files for each.

The output types available are: asc, html, ps, screen and raw. Results submitted to SPEC will appear on the SPEC website in the asc, html, ps and pdf formats. The manager script produces all output formats by default. It puts them into the results directory. In addition, the test details that appear on the screen get written to a res.<nnn> in the results directory. The 'detailed ASCII' output generated by the -D switch contains these same details. The 'detailed ASCII' format is for review, only, not publication.

4 Understanding the benchmark screen display

The manager script outputs data to the screen as it runs. This output also appears in the results/res.nnn file. The manager script writes the same data summarized on a per-client basis into the results/log-client.nnn file. The following is an annotated res file.

4.1 Test setup summary

The test preamble section gives information about the whole run, including:

The following is an example of the output from manager at the start of a run.

Opened logfile results/res.248

Opened client logfile results/log-client.248

CLIENT LIST:
  Client 1: name=localhost, port=, speed=1, host=
  Client 2: name=csg161, port=, speed=1, host=

4.2 Operational validation

Next, manager validates all the types of operations. When all the tests pass, you know that:

If any of these tests fail, manager will abort and let you correct the errors. For debugging purposes, you can set a failing operation type to 0% in the rc file to skip both the test and issuing of this operation. For example, if you haven't yet implemented the dynamic POST, set 'DYNAMIC_POST=0' in the rc file to run without it. In the following example, the CGI GET call fails because the CGI doesn't exist (HTTP 404, file not found). 

Validating all paths for rc.short

  Checking Reset command with 'https://bbb116/isapi/newz-xmit.so?command/Reset
&maxload=100&pttime=100&maxthread=100&exp=1,100&urlroot=https://bbb116/'

  Checking static GET with 'https://bbb116/file_set/dir00089/class3_6'

  Checking dynamic GET with 'https://bbb116/isapi/newz-xmit.so?/file_set/dir00089/class1_6'

  Checking dynamic CGI GET with 'https://bbb116/wrongcgi/specweb99.cgi?/file_set/dir00089/class1_6'
  **ERROR**: Can't fetch file https://bbb116/wrongcgi/specweb99.cgi?/file_set/dir00089/class1_6 with
a dynamic CGI GET request
Is DYNAMIC_ROOT (or DYN_xxx_SCRIPT) configured  correctly?
Error: 404 Not found

  Checking dynamic custom ad (CAD) GET with 'https://bbb116/isapi/newz-xmit.so?/file_set/dir00089/class1_6'

  Checking dynamic POST with 'https://bbb116/isapi/newz-xmit.so?/file_set/dir00089/class1_6'


****** FATAL ERRORS FOUND *******

4.3 Test Warnings

The manager script warns you if you have reset any of the benchmark constants. For example: 

WARNING: RUN WILL BE INVALID
Invalid test: RUN_TIME is 30, must be 1200
Invalid test: RAMPUP_TIME is 30, must be 300
Invalid test: RAMPDOWN_TIME is 10, must be 300
Invalid test: WARMUP_TIME is 30, must be greater than 1200

Tip: To have manager set all the benchmark constant to valid values automatically, append a -C immediately after manager, i.e.

specperl manager -C rc.short

4.4 Results from each iteration

The results from each iteration contains the following sections:

4.5 Screen display of ASCII output file

Lastly, you will see ASCII output format on the screen and in the res file. This is one of the formats available on the SPECweb99_SSL result pages website. See the next section on Result Pages for a description of this page.

5 SPECweb99_SSL Result Pages and Raw File

The manager script can generate the output in several formats: ASCII, HTML, and PostScript. They get written to the results directory with the names output.<nn>.[asc | html | ps]. In addition it creates an output.<nn>.raw file. Only the .raw file is needed to submit a result to SPEC, so errors in the creation of other output formats can generally be ignored.

5.1 Result pages

The result pages contain the following elements.

Hardware and Software information
The hardware vendor and model name and the HTTP Software vendor and software name/version.
SPECweb99_SSL Metric
The median value from the three iterations. The metric for the benchmark is SPECweb99_SSL. It represents the number of simultaneous connections that a server can sustain at a given bitrate.
Performance of the 3 iterations (HTML and ASCII formats only)
The result from the 3 iterations of the run.
Availability and Configuration information
All the information from the Configuration description entries in the rc file.
Notes and Tuning Information
The Notes entries from the configuration description. This section contains detailed information about every component that used to generate the results. It includes information of both the software and hardware components. All system configuration information required to duplicate the performance results must be reported. This section also includes availability dates for the different components used.
Errors (if any)
Any errors encountered in the test inputs or results. To be valid, the test must have no errors reported in this section. Any errors will cause the phrase 'Invalid Test' to appear either next to the SPECweb99_SSL result number (ASCII and HTML formats) or as a "watermark" across the page (PS format).
Performance Details of the 3 iterations (ASCII detailed output only)
manager will produce a more detailed summary of the 3 iterations if you use the -D switch. This level of detail is for review purposes only and will not appear on the output pages displayed on the SPEC web site.

Sample output pages are included with the kit ( ASCII, ASCII-detailed, HTML, PS).

5.2 Raw file

The raw file contains all the inputs from the rc file and results from the test. You can make any of the other formats from the raw file by specifying manager -o <output-formats> -R <rawfile>.

To create PS output from the output.399.raw file, use specperl manager -o ps -R results/output.399.raw
To recreate all formats from the output.250.raw file, use specperl manager -o all -R results/output.250.raw

Once a valid run is obtained and the notes are satisfactory, the raw file may then be submitted to SPEC for review (see Submitting results).

6 Troubleshooting

If problems are encountered, the following checklists may help identify the cause:

6.1 Server System

General troubleshooting:

6.2 Client system

7 Performance Tuning

The first steps to tuning any benchmark, is analyzing the workload and look for possible bottlenecks in the configuration. There are a number of factors or bottlenecks that could affect the performance of the benchmark on your system.

These tuning tips are for configuring the system to generate the best possible performance numbers for SPECweb99_SSL. Please note that these should not necessarily be used to configure and size real world systems. Configuring and sizing real world HTTP server systems is an extremely difficult exercise. SPECweb99_SSL quantifies certain important aspects of web server performance. However, other workload components may be involved in a given real world web server, e.g. directory name service, ftp, mail, news, etc. Due to the lack of accurate qualitative sizing guidelines, experience or even guesswork often becomes the determining factor in arriving at a configuration. Even experience of a similar environment can be misleading due to the huge variation in user and workload profiles across different end user sites.

Memory
Memory is an important consideration for this benchmark. Since the file set is read only (the dynamic code does not alter any of the files), there is potential for gain in performance from caching more of the file set in memory.
Network
SPECweb99_SSL is a network intensive benchmark and to ensure accurate evaluation of your server, you may want to optimize the behavior of the network. The type of network subsystem needed depends on the peak bandwidth.
TCP/IP
HTTPS is run over TCP/IP. Many TCP/IP implementations have tuning parameters for setting various buffer sizes and protocol features.
Disk I/O
The only writes that occur on the system are the ones to the HTTPS server log and the ones to the POST log. Since all accesses to the server need to be logged as defined in the run rules, there is a potential for the log disk to become the bottleneck. Consider striping the log disk (in hardware or software) if the log disk utilization appears high.
Dynamic Content Implementation
The implementation used to handle the dynamic content portion of the benchmark can have a major impact on the performance. The CGI request portion of the benchmark must use CGI to form the reply, however the other dynamic requests don't have to. Many HTTPS daemons support programming APIs that have much lower overhead for calling and for argument passing.

Note: The SPECweb99_SSL kit supplies a sample implementation that is operational and correct, but is not a good choice for high performance.  Alternative dynamic content implementations used for published SPECweb99 or SPECweb99_SSL results are available on the www.spec.org web site.

HTTPS Server Software
This is a critical component in determining the performance of the system. This will vary with the different HTTPS daemons available. Please refer to the tuning guide for your particular HTTPS daemon for guidance on tuning for the SPECweb99_SSL workload.

Many HTTPS daemons have companion products to accelerate static pages. These products typically use caching to return pages from memory, rather than disk

Client tuning
You will need to monitor the client CPU utilization to make sure that you have enough client power to saturate the server system. A rule of thumb to begin with is to have client systems operate at the 50% CPU utilization level. This will ensure that you are not bottlenecked on the client side.

8 Submitting results

Once you have a successful run, you can submit the result to the SPECweb committee for review by mailing the output.<nn>.raw file to subweb99_ssl@spec.org. When mailing the raw file, you may either attach it or inline it in plain-text format. More than one result or raw file per mail message is permitted as long as each one is included as an attachment. Note: The raw file uses the configuration and parameter information in the rc file. Please edit your rc file with the correct information prior to running the benchmark for submission.

Additional result supporting documentation (source code for the dynamic content implementation, for example) will be required by SPEC for reviewing a result. In addition, save the HTTPS daemon log file from the benchmark run as you may be asked to provide part of the log for review by the committee.

Every submission goes through a two-week review process. During the review, members of the committee may ask for further information/clarifications on the submission. Once the result has been reviewed and approved by the committee, it is displayed on the SPEC web site at www.spec.org.

For more information on submitting results to SPEC, please send mail to info@spec.org.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/).
ssldump is a SSLv3/TLS protocol analyzer written by Eric Rescorla <[email protected]> and licensed by RTFM, Inc. © 1999-2001 RTFM, Inc.

Home - Contact - Site Map - Privacy - About SPEC Facebook logo LinkedIn logo Twitter logo
webmaster@spec.org
Last updated: Mon Nov 15 16:26:12 EST 2004
Copyright 1995 - 2024 Standard Performance Evaluation Corporation
URL: http://www.spec.org/web99ssl/docs/users_guide.html