What is This Thing Called "SPECviewperf®"?

---

SPECviewperf® is a portable OpenGL performance benchmark program written in C. It was developed by IBM. Later updates and significant contributions were made by SGI, Digital (Compaq, HP), 3Dlabs (Creative Labs) and other SPECopc^SM project group members. SPECviewperf provides a vast amount of flexibility in benchmarking OpenGL performance. Currently, the program runs on most implementations of UNIX, Windows XP, Windows 2000, and Linux.

The OpenGL Performance Characterization (SPECopc) project group endorsed SPECviewperf as its first OpenGL benchmark. Performance numbers based on SPECviewperf were first published in the Q4 1994 issue of The GPC Quarterly.

SPECopc group member companies have ported the SPECviewperf code to their operating systems and window environments. The SPECopc project group maintains a single source code version of the SPECviewperf code that is available to the public.

Benchmarking with SPECviewperf

SPECviewperf parses command lines and data files, sets the rendering state, and converts data sets to a format that can be traversed using OpenGL rendering calls. It renders the data set for a pre-specified amount of time or number of frames with animation between frames. Finally, it outputs the results.

SPECviewperf reports performance in frames per second. Other information about the system under test -- all the rendering states, the time to build display lists (if applicable), and the data set used -- are also output in a standardized report.

A "benchmark" using SPECviewperf is really a single invocation of SPECviewperf with command-line options telling the SPECviewperf program which data set to read in, which texture file to use, what OpenGL primitive to use to render the data set, which attributes to apply and how frequently, whether or not to use display lists, and so on. One quickly realizes that there are an infinite number of SPECviewperf "benchmarks" (an infinite number of data sets multiplied by an almost infinite number of command-line states).

Real-World Benchmarking

SPECopc project group members recognize the importance of real-world benchmarks. From the beginning, the group has sought benchmarks representative of the OpenGL rendering portion of independent software vendor (ISV) applications. Along these lines, the project group has come up with what it calls a viewset. A viewset is a group of individual runs of SPECviewperf that attempt to characterize the graphics rendering portion of an ISV's application.

Viewsets differ from SPECapc benchmarks in that they exercise only the graphics functionality of the application. This enables direct performance comparisons of graphics hardware. Since SPECviewperf does not require an application software license to run, it is accessible to a wider range of users than SPECapc benchmarks. It is also easier to use and runs faster than SPECapc benchmarks.

Viewsets are generally not developed by the SPECopc project group; they come from the ISVs themselves. Members of the SPECopc project group often "sponsor" the ISV. Sponsorship entails helping the ISV in several areas, including how to obtain the SPECviewperf code, how to convert data sets to a SPECviewperf format, how to use SPECviewperf, how to create SPECviewperf tests to characterize the application, how to determine weights for each of the individual SPECviewperf tests based on application usage, and finally to help offer the viewset to the SPECopc project group for consideration as a standard SPECopc viewset. Any ISV wishing to develop a viewset should contact [email protected].

Currently, there are six standard SPECopc application viewsets:

• 3dsmax-02, based on SPECapc for 3ds max 3.1 configured with the Open GL driver, includes three models containing an average of 1.5 million vertices each, and tests performance of scenes with different levels of lighting.
• dx-08, based on IBM's Data Explorer application, has 10 different tests.
• drv-09, based on Intergraph's DesignReview model review package, has five different tests.
• light-06, based on Discreet's Lightscape radiosity application, has four tests.
• proe-02, based on SPECapc for Pro/ENGINEER 2001, measures two models in three modes - shaded, wireframe and hidden-line removal (HLR).
• ugs-03, based on SPECapc for Unigraphics V17, tests performance based on an engine model containing 2.1 million vertices.

These type of applications typically render large data sets. All six viewsets represent relatively high-end applications.

The input to SPECviewperf is the command line. Here are the command line options for SPECviewperf 7.1:

Program options:

SPECviewperf v7.1
 
Program options:
-polygon -pg <file>  : Viewpoint object to be used in the tests
-triangle -tr <file>  : Viewpoint object to be used in the tests
-quad -qd <file>  : Viewpoint object to be used in the tests
-mesh -mh <file>  : Mesh object to be used in the tests
-frameattr -fa <file>  : Mesh3 attributes for each frame
-rendermode -rm <mode>  : POINT, VECTOR, LINE, POLYGON, TMESH, TFAN,
  TRIANGLE, or QUAD - default LINE
-vcriteria -vcrit  : AUX Visual selection criteria - EXACT, MIN
  - default MIN
-vid <id>  : Ask AUX for visual with ID = <id>
-vaccum -vac  : Ask AUX for an accumulation buffer visual
-valpha -val  : Ask AUX for an alpha buffer visual
-vdepthbuffer -vz  : Ask AUX for a depth buffer visual
-vstencil -vst  : Ask AUX for a stencil buffer visual
-indirectrender -ir  : Render indirect - default direct
-nodither -ndi  : Disable dithering
-ortho -or  : Parallel/Orthographic projection - default Perspective
-displaylist -dl  : Render with display list mode
-vertexarray -va  : Render with vertex array mode (OpenGL1.1)
-interleaved -il  : Packing vertex data interleaved in memory
-colorper -cp <mode>  : FRAME = Color per Frame,
  PRIMITIVE = Color per Primitive,
  VERTEX = Color per Vertex - default FRAME
-texture -tx <file>  : Image for texturing
-texgen -txg <file> <mode>  : <file> is image for enviroment mapping
  <mode> is SPHERE_MAP, OBJECT_LINEAR, EYE_LINEAR
  - default EYE_LINEAR
-texobj -txo <file>  : Texture object mode <file> is list of images
-magfilter -magf <flt>  : NEAREST, LINEAR - default NEAREST
-minfilter -minf <flt>  : NEAREST, LINEAR, NEAREST_MIPMAP_NEAREST,
  LINEAR_MIPMAP_NEAREST, NEAREST_MIPMAP_LINEAR,
  LINEAR_MIPMAP_LINEAR - default NEAREST
-texenv -te <env>  : Texture enviroment, MODULATE, DECAL, BLEND
  - default DECAL
-texcomp -tc <num>  : Texture components where <num> is 1,2,3, or 4
  -default 3
-blend -bl  : Enable Blending
-srcblendfunc -sbf  : ZERO, ONE, DST_COLOR, ONE_MINUS_DST_COLOR, SRC_ALPHA,
  ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
  SRC_ALPHA_SATURATE - default SRC_ALPHA
-dstblendfunc -dbf  : ZERO, ONE, SRC_COLOR, ONE_MINUS_SRC_COLOR, SRC_ALPHA,
  ONE_MINUS_SRC_ALPHA, DST_ALPHA, ONE_MINUS_DST_ALPHA,
  - default ONE_MINUS_SRC_ALPHA
-linewidth -lw <width>  : Linewidth for wire/vector tests - default 1.0
-xwinsize -xws <side>  : Size of test windows X dimension - default 700
-ywinsize -yws <side>  : Size of test windows Y dimension - default 700
-numframes -nf <num>  : Number of frames to be rendered during measurement
  Takes priority over -mp
-numilights -nil <num>  : Turns on <num> infinite lights - default 0
-numllights -nll <num>  : Turns on <num> local lights - default 0
-colormaterial -cm <side> <mode> : <side> is FRONT, BACK, FRONT_AND_BACK 
  - default FRONT
  <mode> is AMBIENT, DIFFUSE, EMISSION, SPECULAR,
  AMBIENT_AND_DIFFUSE 
  - default AMBIENT_AND_DIFFUSE
-backface -bf  : Cull Backfacing primitives - default off
-frontface -ff  : Cull Frontfacing primitives - default off
-singlebuffer -sb  : Single buffer mode
-fog -fg  : Enable fog
-linesmooth -ls  : Enable line antialiasing
-polysmooth -ps  : Enable polygon antialiasing
-facetnormal -fn  : Use facet normals when lighting
-linestipple -lp  : Enable line stipple
-polystipple -pp   : Enable polygon stipple
-batch -bt <num>  : Batch <num> primitives together per glBegin/glEnd
  Valid with POINT, VECTOR, TRIANGLE, and QUADS
-polymodefront -pmf  : POINT, LINE, or FILL - default FILL
-polymodeback -pmb  : POINT, LINE, or FILL - default FILL
-flat -f  : Set shademodel to FLAT - default GOURAUD
-zbuffer -zb  : Enable zbuffer for tests - default off
-clip -c  : Align object on 3D clip boundary
-lighttwoside -l2s  : Light both sides of model
-localview -lv  : Define local viewer for lit tests
-normalize -nm  : Enable normal normalization
-minperiod -mp <num>  : Set minimum testing period in seconds
-mblur <num>  : Use motion blur with num being amount of decay
-aa_multi <x> <r>  : Full scene antialiasing rendered x times at an
  offset of r. r should be tuned to the viewset
-walkthru -wt  : Walkthru mode
-geomrepeat -gr  : Geometry repeat
-grab <script>   : Screen capture
-grabonly -go  : Don't run benchmark, just do screen capture
-swap <true/false>  : Force byte swapping
-notitle -nt  : Disable title screen
-startframe -sf <num>  : Set starting frame of a walkthru
-framepause -fp  : Pause after each frame waiting for enter key
-newview -nv  : Enable new view frustum calculation
-fov <angle>  : Set field of view in degrees
-spin x y z dx dy dz  : Set rotation initial values and speeds (degrees)
-clipplane  : Enables a user clip plane
-nomaterial  : Disables material changes if present
-nolight  : Disables viewperf lighting
-forcecalibrate  : Forces viewperf calibration
-tracelights -tl  : Use lights generated in scene
-norejection  : Turns off trivial rejection for walk through.
-noframecounter  : Turn off display of frame counter 
-staticFrameCount -sfc <num>  : Count of frames between data edits (default = 10)
-staticBlockCount -sbc <num>  : Number of memory blocks changed per edit (default = 3)
-staticEditSize -ses <num>  : Amount of frame data changed per edit (default = 0.001)
-staticEditsVisible -sev  : Attempt to make edits visible in the frame during execution.
-staticEditsDisabled  : Allow all data to be static, no frame edits for entire test.
 

The -threads option is applicable only if SPECviewperf has been compiled with the C Preprocessor variable MP defined. SPECviewperf should only be compiled this way if the vendor supplied makefile has defined the MP flag (-DMP).

Here is an example of the command line from test #1 of the proe-02 viewset:
viewperf -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr

This command line is providing the following directions:

• invoke the SPECviewperf program
• -mh specifies the format of the data set is mesh format (binary)
• the data set file name is WorldCar6.mh3
• -fa specifies some attributes to be processed per-frame (binary)
• the attributes are stored in gradient.mha
• -tracelights specifies that the lights contained in the command stream should be used
• -va directs SPECviewperf to render using vertex arrays
• -zb enables z-buffering
• -mp 12 will set the minimum period to 12 seconds; SPECviewperf will run this test for at least 12 seconds
• -forcecalibrate forces SPECviewperf to calibrate before running the test. This allows the frame rate to be determined so that sufficient frames are processed during the test for the defined test period.
• -xws 960 -yws 960 this specifies the X and Y dimensions of the SPECviewperf window to be 960x960 pixels.
• -grab grab.scr directs SPECviewperf to capture a section of the screen after completing the performance test.

Upon entering the command line, the system under test will run the benchmark for at least 12 seconds. This is to insure a valid test result. During the test, the SPECviewperf code renders the data set, WorldCar6.mh3, which is the WorldCar model used in the SPECapc for Pro/ENGINEER 2001™, rotating it about the center of the window with the above specified characteristics.

What follows is the output of SPECviewperf 7.1 running the first test from proe-02:

-------------------- 
Viewset: ProEngineer 
Test Number 01: WorldCar6 : Weight 13.50 
-------------------- 
 Viewperf Version 7.1
 Viewperf Arguments -mh WorldCar6 -fa gradient -tracelights -va -zb -mp 12 -forcecalibrate -xws 960 -yws 960 -grab grab.scr 
 Month 3
 Day 4
 Year 2003
 Host system HOSTNAME
 Operating System Microsoft Windows NT
 Operating System Version Version 5.1 (Build 2600: Service Pack 1)
 Host Vendor unknown
 Host Model AT/AT COMPATIBLE
 Host CPU x86 Family 15 Model 2 Stepping 7
 Host CPU Count unknown
 Host Memory Size (MB) 1023
 Host Primary Cache Size (KB) unknown
 Host Secondary Cache Size (KB) unknown
 Window System Win32
 Driver Version Version of GFX driver
 OpenGL Vendor Name of company making GFX card
 OpenGL Version Version of OpenGL supported by GFX card
 OpenGL Extensions List of all supported OpenGL Client extensions
 OpenGL Renderer Name of GFX card
 OpenGL Client Vendor GFX Card OpenGL client vendor string
 OpenGL Client Version Version of OpenGL client supported by GFX card
 OpenGL Client Extensions List of all supported OpenGLClient extensions
 GLU Version 1.2.2.0 Microsoft Corporation
 GLU Extensions List of all supported GLU extensions
 Direct Rendering False
 Double Buffer True
 Stereo False
 RGBA True
 Color Index Size -1
 Red Size 8
 Green Size 8
 Blue Size 8
 Alpha Size 0
 Accum Red Size 16
 Accum Green Size 16
 Accum Blue Size 16
 Accum Alpha Size 16
 Depth Size 24
 Stencil Size 0
 Auxiliary Buffer Count 0
 Frame BufferLevel 0
 Window Width (pixels) 960
 Window Height (pixels) 960
 Screen Width (pixels) 1280
 Screen Height (pixels) 1024
 Pixel Format ID 4
 Visual Selection Criteria MINIMUM
 Number of Execution Threads 1
 Geometry File WorldCar6
 Input Mode -mh 
 Minimum Test Period 12.000000
 Number of Frames 0
 Number of Primitives 319413
 Number of Vertices per Frame 3917495
 Number of Vertices per Primitive 12.264670
 Estimated bytes per frame 0.000000
 Static frame count 0.000000
 Model edited every 10 frames
 Amount model changed 0.001000
 Toggle Mode NONE
 Batching Count 0
 Render Mode LINE
 Color per COLOR_PER_FRAME
 Orthographic Projection FALSE
 Display List FALSE
 Vertex Array (OpenGL1.1) TRUE
 Interleaved Data FALSE
 Clip Geometry FALSE
 Walkthrough Mode FALSE
 Geom Repeat Mode FALSE
 Back Face Cull FALSE
 Front Face Cull FALSE
 Front Polygon Mode FILL
 Back Polygon Mode FILL
 Polygon Stipple Enable FALSE
 Polygon Antialiasing Enable FALSE
 Line Width 1.000000
 Line Stipple Enable FALSE
 Line Antialiasing Enable FALSE
 Number of Infinite Lights 0
 Number of Local Lights 0
 Color Material Enable FALSE
 Color Material Face FRONT
 Color Material Mode AMBIENT_AND_DIFFUSE
 Facet Normals FALSE
 Two Sided Lighting Enable FALSE
 Local Viewer Enable FALSE
 Flat Shading FALSE
 Fog Enable FALSE
 Normal Normalization FALSE
 User Clip Plane FALSE
 Material Changes Disabled FALSE
 Attribue Changes Disabled FALSE
 Trace Lights Enabled TRUE
 Texture Enable FALSE
 TexObj Enable FALSE
 Texture Generation Mode NO_TEXTURE_GENERATION
 Texture File NONE
 Texture Minification Filter NEAREST
 Texture Magnification Filter NEAREST
 Texture Environment Mode DECAL
 Texture Components 3
 Depth Test Enable TRUE
 Blend Enable FALSE
 Source Blend Function SRC_ALPHA
 Destination Blend Function ONE_MINUS_SRC_ALPHA
 Dithering Enable TRUE
 Motion Blur Amount 0.000000
 Full Scene Antialiasing Redraws 0
 Full Scene Antialiasing Jitter Amount 0.000000
 
 Number of frames run: 280, Test period: 12.187000 (sec)
 23.0 frames/sec -- -mh WorldCar6 -rm LINE -mp 12 -cp FRAME -ir -va -zb -tracelights -vz

During the calibration period SPECviewperf tests to see how many frames need to be rendered to meet the minimum time period and then runs the test.

The last line is the number of frames per second, which is the fundamental metric for each test. This is the result that is printed in the summary and individual report pages within the GPC News SPECopc section. Also printed in the summary and individual report pages is a single composite number for each viewset. This composite number is derived using a weighted geometric mean methodology (essentially the product of the results of each test raised to the power of the test weightings).

SPECviewperf 7.1

SPECviewperf 7.1 introduces new data changes to discourage optimizations that benefit the benchmark, but might not improve real-world application performance. SPECviewperf 7.1 inserts a small amount of variation at regular intervals within its viewsets. This ensures that the test system examines and processes each frame individually and is more aligned with the behavior of real-world applications. A second frame counter that moves around within the frame has also been added to the benchmark. This provides visual verification of whether frame changes are being rendered. For more information, please read the documentation provided with the SPECviewperf source code.

Rob Putney / Simon Green / Chris Watts / Bill Licea-Kane / Allen Jensen / Ian Williams