|
Results
SPECapc
Download/Order
Resources
|
| The
Application Performance Characterization Project Committee Rules
Version 2.02
Last Updated: 05/01/2006
-
Overview
-
General
Philosophy
-
All rules
declared in "The Graphics Performance
Characterization Group (SPEC/GPC): Rules For Project Groups" document
(known hereafter as the GPC Project Groups Ruleset) apply, unless
specifically overruled by a rule in this document.
-
General
Philosophy
-
Within SPEC's
Graphics Performance Characterization (GPC) Group there was a
strong belief that it is important to benchmark graphics performance
based on actual applications. Application-level benchmarks exist,
but they are not standardized and they do not cover a wide range
of application areas. Thus, the Application Performance Characterization
(SPECapcSM) Project was created within the GPC
to create a broad-ranging set of standardized benchmarks for graphics-intensive
applications.
-
The SPECapc
seeks to develop benchmarks for generating accurate application-level
graphics performance measures in an open, accessible and well-publicized
manner.
-
The SPECapc
wishes to contribute to the coherence of the field of application
performance measurement and evaluation so that vendors will be
better able to present well-defined performance measures and customers
will be better able to compare and evaluate vendors products and
environments.
-
The SPECapc
will provide formal beta benchmarks to members and final benchmark
releases to the public in a timely fashion.
-
Hardware
and software used to run the SPECapc benchmarks must provide a
suitable environment for running typical (not just benchmark)
workloads for the applications in question.
-
SPECapc
reserves the right to adapt its benchmarks as it deems necessary
to preserve its goal of fair and useful benchmarking (e.g. remove
benchmark, modify benchmark code or data, etc). If a change is
made to the suite, SPECapc will notify the appropriate parties
(i.e. SPECapc members and users of the benchmark) and SPECapc
will re-designate the benchmark by changing its name and/or version.
In the case that a benchmark is removed in whole or in part, SPECapc
reserves the right to republish in summary form "adapted" results
for previously published systems, converted to the new metric.
In the case of other changes, such a republication may necessitate
re-testing and may require support from the original test sponsor.
-
Overview
of Optimizations
-
SPECapc
is aware of the importance of optimizations in producing the best
system performance. SPECapc is also aware that it is sometimes
hard to draw an exact line between legitimate optimizations that
happen to benefit SPECapc benchmarks and optimizations that specifically
target SPECapc benchmarks. However, with the list below, SPECapc
wants to increase awareness of implementers and end-users to issues
of unwanted benchmark-specific optimizations that would be incompatible
with SPECapc's goal of fair benchmarking.
-
To ensure
that results are relevant to end-users, SPECapc expects that the
hardware and software implementations used for running SPECapc
benchmarks adhere to a set of general rules for optimizations.
-
General
Rules for Optimization
-
Optimizations
must generate correct images and results for the application under
test, for both the benchmark case and similar cases. Correct images
and results are those deemed by the majority of the SPECapc electorate,
potentially with input from the associated independent software
vendor (ISV) and/or end-users, to be sufficiently adherent to
the intent behind the application.
-
Optimizations
must improve performance for a class of workloads where the class
of workloads must be larger than a single SPECapc benchmark or
SPECapc benchmark suite.
-
For any
given optimization a system should generate correct images with
and without said optimization. An optimization should not reduce
system stability.
-
The vendor
encourages the implementation for general use (not just for running
a single SPECapc benchmark or SPECapc benchmark suite).
-
The implementation
is generally available, documented and supported by the providing
vendor.
-
In the case
where it appears that the above guidelines have not been followed,
SPECapc may investigate such a claim and request that the optimization
in question (e.g. one using SPECapc benchmark-specific pattern
matching) be removed and the results resubmitted. Or, SPECapc
may request that the vendor correct the deficiency (e.g. make
the optimization more general purpose or correct problems with
image generation) before submitting results based on the optimization.
-
It is expected
that system vendors would endorse the general use of these optimizations
by customers who seek to achieve good application performance.
-
No pre-computed
(e.g. driver-cached) images, geometric data, or state may be substituted
within an SPECapc benchmark on the basis of detecting that said
benchmark is running (e.g. pattern matching of command stream
or recognition of benchmark's name).
-
Benchmarks
-
Benchmark
Acceptance
-
Benchmark
components are defined as
-
specific
revision of an application,
-
run rules, scripts
and associated data sets.
-
New or modified
benchmark components require a 2/3-majority vote to be accepted
for publication. Selection of datecode versions of a specific
revision of an application is by majority vote.
-
A minimum
3-week review period is required for new or significantly modified
benchmark components.
-
At the end
of the review period a vote will be called to approve the proposed
changes.
-
An amendment
to a benchmark component during the review period must be unanimously
accepted. If not, the review period shall be restarted.
-
Benchmark
Code Versioning
-
Benchmarks
use the following version coding: M.m (e.g. SPECapcSM for
Pro/ENGINEER 20.0 v1.1) M is the major release number and m is
the minor release number.
-
The major
release number is only incremented when large amounts of code
are changed and the scripting language is dramatically changed
as a result -- backward compatibility is highly unlikely when
moving scripts or data sets between major releases (e.g. running
v2 scripts on a v3 executable would almost certainly fail).
-
The minor
release number is bumped if some small set of code is replaced
or removed - but the standard, unchanged scripts and data sets,
as a whole, must run on the new version (but perhaps with different
performance).
-
When there
is a new major release of a benchmark, submissions using the previous
release will be accepted for at least one submission cycle. When
a superceded benchmark is retired, the associated run-rules will
be archived in an "archived benchmark run-rules document" posted
on the Previous SPECapc Benchmarks web-page.
-
Submission,
Review and Publication
-
General
Benchmark Run Rules
-
The system
under test must correctly perform all of the operations being
requested by the application during the benchmark.
-
No changes
to any files associated with the benchmark are permitted excepted
as noted in the benchmark-specific rules (section 5 of this document).
-
The entire
display raster must be available for use by the application being
benchmarked.
-
It is not
permissible to override the intended behavior of the application
through any means including, but not limited to, registry settings
or environment variables.
-
No interaction
is allowed with the system under test during the benchmark, unless
required by the benchmark.
-
The system
under test can not skip frames during the benchmark run.
-
It is not
permissible to change the system configuration during the running
of a given benchmark. That is, one can't power off the system,
make some changes, then power back on and run the rest of the
benchmark.
-
Results
submitted must be obtained using the scripts, models, and application
revisions which are specified for that submission cycle by the
SPECapc.
-
The
color depth used must be at least 24 bits (true color), with at
least 8 bits of red, 8 bits of green and 8 bits of blue, unless
otherwise specified.
-
The
display raster resolution must be at least 1280 pixels by 1024
pixels, unless otherwise specified.
-
The
monitor refresh rate must be at least 75Hz. This requirement does
not apply to digital flat panel displays, unless otherwise specified.
-
The border width of the windows
created during the benchmark shall not exceed 10 pixels, unless
otherwise specified.
-
The monitor
used in the benchmark must support the stated resolution and refresh
rate.
-
The benchmark
must successfully obtain all requested window sizes, with no reduction
or clipping of any benchmark-related windows. Windows created
by the benchmark must not be obscured on the screen by anything
other than other elements created by the benchmark.
-
Tests may
be run with or without a desktop/window manager if the application
allows this, but must be run on some native windowing system.
-
General
Submission Content Rules
-
The submission
upload file structures are defined in the benchmark-specific section
below.
-
Submission
Process Rules
-
The submission
file names are detailed below under the benchmark-specific rules.
-
Review
Period Rules
-
Reviewers
will decide if the image quality and results of the submission
are sufficiently correct with respect to the intent of the ISV
to satisfy the intended end-users' expectations.
-
SPECapc Benchmark
Specific Rules and Procedures
-
Solid
Edge V14
-
The benchmark
must be run using Solid Edge V14.00.00.70
-
The application
can be run with Tools-Options-View Graphics Display set to Graphics
Card Driven, Software Driven or Backing Store. The choice must
be documented in the notes portion of the results.
-
Application
settings must not be changed from the defaults set by the benchmark
installation. Settings that must not be changed include,
but are not limited to:
-
Culling
-
Wireframe
display in Move Part command
-
Arc smoothness
(3)
-
The submission
must contain the file result.txt that is generated during the
benchmark run.
-
The directory
structure of the submission must be as follows:
.../company-name/system_1/SEV14/result.txt
.../company-name/system_2/SEV14/result.txt
etc...
-
The submission
file must be named company_apc_SolidEdgeV14_vN.zip
where company is the member company or organization name
in lower case and vN is the file version (e.g. hp_apc_SolidEdgeV14_v0.zip.)
The initial submission is v0. Resubmitted files must have the
version number incremented.
-
Solidworks
2005
-
The benchmark
must be run using Solidworks2005 service pack 0.
-
The application
window size must not be changed from its initial size.
-
The submission
must contain a results.txt file generated by running the benchmark.
Its contents are extracted from the file apcresultsN.txt,
generated by benchmark GUI , where N is the number
of the benchmark test run.
-
The submission
will be derived from the best composite generated by the default
5 benchmark runs, as controlled and reported by the benchmark
GUI.
-
The appearance
of the the application's Quick Tips / Dynamic Help box, or other
pop-ups that do not dismiss themselves, will cause the benchmark
run to be invalid. The benchmark FAQ has information on
how to prevent this.
-
The directory
structure of the submission must be as follows:
.../company-name/system_1/sw2005/results.txt
.../company-name/system_2/sw2005/results.txt
etc...
-
The submission
file must be named company_apc_sw2005_vN.zip where company is
the member company or organization name in lower case and
vN is the file version (e.g. ibm_apc_sw2005_v0.zip.) The
initial submission is v0. Resubmitted files must have the version
number incremented.
-
3dsmax7
-
The benchmark
must be run using 3ds max version 7, service pack 0 (i.e. unpatched).
-
The application
windows will be visible as specified:
-
The "3ds
max 7" window is maximized and fills the screen (The window
is not occluded by any other windows.) Note: Task-bar auto-hide
should be enabled.
-
Drivers
and custom drivers must be configured (explicitly or implicitly)
to use:
-
GL_TEXTURE_MIN_FILTER
= GL_LINEAR_MIPMAP_LINEAR
-
GL_TEXTURE_MAG_FILTER
= GL_LINEAR
-
Textures
cannot be modified, i.e. reduced in size or depth.
-
Wireframe
objects must NOT be drawn using triangle strips.
-
The following
Viewport parameters must be checked (i.e. enabled) in the Viewports
Tab found in the 3ds max 7 Customize->Preferences menu (Viewports
tab):
-
Backface
cull on object creation
-
Mask viewport
to safe region
-
Update
background while playing
-
Display
world axis
-
The following
Viewport parameters must NOT be checked in the 3ds max 7 Customize->Preferences
menu (Viewports tab):
-
Attenuate
lights
-
Filter
environment backgrounds
-
The benchmark
can be run using either OpenGL or Direct 3D.
-
For
OpenGL submissions, an application graphics driver or plug-in
may be used, but the images captured from the benchmark must
not differ from the supplied 3ds max OpenGL driver's images by
more than 5%. If the application driver or plug-in’s images
do not match, the submitter may apply for a waiver based on documented
errors in the 3ds max OpenGL driver. The availability and support
of an application driver or plug-in shall not affect the submission's “single
supplier” or “parts built” status.
-
The submission
must contain the files result.txt and result.xls that are generated
during the benchmark run.
-
The directory
structure of the submission must be as follows:
.../company-name/system_1/3dsmax7/result.txt
.../company-name/system_1/3dsmax7/result.xls
.../company-name/system_2/3dsmax7/result.txt
etc...
-
The submission
file must be named company_apc_3dsmax7_vN.zip where company is
the member company or organization name in lower case and vN is
the file version (e.g. hp_apc_3dsmax7_v0.zip.) The initial submission
is v0. Resubmitted files must have the version number incremented.
-
Submitters
are not required to include pixel-comparison images.
-
A committee
member raising a challenge to a submission can employ pixel-comparison
images to support the challenge. However, reference and grabbed
images must be generated on a system with identical GPU, graphics
and custom drivers and settings to the submitted configuration
being challenged.
-
Maya
6.5
-
The benchmark
must be run using Maya 6.5
-
The benchmark
script must be run with the command 'mayaTest(3)' where '3' is
the number of runs.
-
The submission
must contain the results.txt submission file as well as the scoring
spreadsheet used to calculate the result.
-
The directory
structure of the submission must be as follows:
.../company-name/system_1/maya65/results.txt
.../company-name/system_1/maya65/MayaResults.xls
etc...
-
The submission file must be
named company_apc_maya65_vN.zip where company is the member company or
organization name in lower case and vN is the file version
(e.g. ibm_apc_maya65_v0.zip.) The initial submission is v0. Resubmitted
files must have the version number incremented.
-
Pro/ENGINEER
Wildfire 2.0
-
The
benchmark must be run using build “M160” of Pro/ENGINEER
Wildfire 2.0. Submissions may be made with 32-bit or 64-bit versions
of M160. However, submissions made with 32-bit Pro/ENGINEER Wildfire
2.0 on Windows XP 64-bit Edition are prohibited.
-
The
config.pro file must be used as-is and may not be modified or
overridden.
-
The
script files “utilities\runbench.bat” or “utilities/runbench.csh” may
be modified as necessary to enable execution of the benchmark
on the system being tested. If modified, the modified version
must be included in the benchmark submission.
-
The
submission must contain the “proe_result.txt” file
(as generated by the “proescore” program) as well
as the corresponding “trail.txt” file generated from
running the benchmark. Both of these files may be found in the "results" directory
after a successful run of the benchmark. The first section of
the “proe_result.txt” file must be edited to reflect
the system configuration.
-
There
must be no license-related warnings in the “trail.txt” file.
-
The
directory structure of the submission must be as follows:
.../company-name/system_1/proewildfire2/proe_result.txt
.../company-name/system_1/proewildfire2/trail.txt
(may be compressed)
.../company-name/system_1/proewildfire2/runbench.bat
(or runbench.csh, if modified, as required by rule c above)
.../company-name/system_2/proewildfire2/proe_result.txt
.../company-name/system_2/proewildfire2/trail.txt
(may be compressed)
.../company-name/system_2/proewildfire2/runbench.bat
(or runbench.csh, if modified, as required by rule c above)
etc...
-
The
submission file must be named company_apc_proewildfire2_vN.zip
or company_apc_proewildfire2_vN.tar.z where company is
the member company or organization name in lower case and vN is
the file version (e.g. “sgi_apc_proewildfire2_v0.tar.z” and “intel_apc_proewildfire2_v0.zip”.)
The initial submission is v0. Resubmitted files must have the
version number incremented.
-
UGS
NX3
-
The benchmark must be run using build “3.0.3.2” of
Unigraphics NX3. Submissions are only accepted on Windows XP Professional
SP2 and later.
-
The script file “%ProgramFiles%\SPECapc\nx3bench\nx3runbench.bat” may
be modified to change the directory where the benchmark files
are located on the system being tested. If modified, the modified
version must be included in the benchmark submission.
-
The application window must be maximized and all toolbars and
the resource bar must be turned off.
-
NX3's Fixed Frame Rate feature must be disabled.
-
The system's booted configuration as specified in the "boot.ini" file
must contain the "/3GB" flag.
-
The chosen setting for View Frustum Culling must be documented
in the Comments field of the "results.txt" file.
-
The "Disable Line Smooth" option under Preferences>Visualization
Performance>General Settings must be unchecked.
-
All other application settings should be default.
-
The submission must contain the “nx3result.txt” file
as well as the corresponding “nx3result.xls” file
generated from running the benchmark. Both of these files may
be found in the "results" directory after a successful
run of the benchmark. The first section of the “nx3bench.xls” file
must be edited to reflect the system configuration.
-
The directory structure of the submission must be as follows:
.../company-name/system_1/nx3bench/nx3result.txt
.../company-name/system_1/nx3bench/nx3bench.xls
.../company-name/system_1/nx3bench/result.txt
.../company-name/system_1/nx3bench/nx3bench.log
.../company-name/system_1/nx3bench/nx3runbench.bat,if
modified, as required by rule c above)
.../company-name/system_2/nx3bench/nx3result.txt
.../company-name/system_2/nx3bench/nx3bench.xls
.../company-name/system_2/nx3bench/result.txt
.../company-name/system_2/nx3bench/nx3bench.log
.../company-name/system_2/nx3bench/nx3runbench.bat,
if modified, as required by rule c above)
etc...
-
The submission file must be named company_apc_nx3bench_vN.zip
or company_apc_nx3bench_vN.tar.z where company is the member company
or organization name in lower case and vN is the file version
(e.g. “sgi_apc_nx3bench_v0.tar.z” and “intel_apc_nx3bench_v0.zip”.)
The initial submission is v0. Resubmitted files must have the
version number incremented.
|
|
