The
OpenGL Performance Characterization Project Rules
Version
1.6
Last
Updated: 1/23/2003
- Overview
- General Philosophy
- The OpenGL Performance Characterization Project of
SPEC/GPC (henceforth abbreviated as SPECopcSM) believes the user
community will benefit from an objective series of tests, which can serve
as common reference and be considered as part of an evaluation
process.
- The SPECopc seeks to develop benchmarks for
generating accurate OpenGL performance measures in an open, accessible and
well-publicized manner.
- The SPECopc wishes to contribute to the coherence
of the field of OpenGL performance measurement and evaluation so that
vendors will be better able to present well-defined performance measures;
and customers will be better able to compare and evaluate vendors'
products and environments.
- The SPECopc will provide formal beta software to
members and final software releases to the public in a timely
fashion.
- Hardware and software used to run the SPECopc
benchmarks must provide a suitable environment for running typical OpenGL
programs.
- SPECopc reserves the right to adapt its benchmarks
as it deems necessary to preserve its goal of fair and useful benchmarking
(e.g. remove benchmark, modify benchmark code or data, etc). If a change
is made to the suite, SPECopc will notify the appropriate parties (i.e.
SPECopc members and users of the benchmark) and SPECopc will re-designate
the metrics (e.g. changing the metric from DRV-04 composite to DRV-05
composite). In the case that a benchmark is removed in whole or in part,
SPECopc reserves the right to republish in summary form "adapted" results
for previously published systems, converted to the new metric. In the case
of other changes, such a republication may necessitate re-testing and may
require support from the original test sponsor.
- Overview of Optimizations
- SPECopc is aware of the importance of
optimizations in producing the best system performance. SPECopc is also
aware that it is sometimes hard to draw an exact line between legitimate
optimizations that happen to benefit SPECopc benchmarks and optimizations
that specifically target SPECopc benchmarks. However, with the list below,
SPECopc wants to increase awareness of implementers and end-users to
issues of unwanted benchmark-specific optimizations that would be
incompatible with OPC's goal of fair
benchmarking.
- To ensure that results are relevant to end-users,
SPECopc expects that the hardware and software implementations used for
running SPECopc benchmarks adhere to a set of general rules for
optimizations.
- General Rules for Optimization
- Optimizations must generate correct images for a
class of programs, where the class of programs must be larger than a
single SPECopc benchmark or SPECopc benchmark suite. Correct images are
those deemed by the majority of the SPECopc electorate to be sufficiently
adherent to the OpenGL specification for the targeted end-user community
(e.g. users of OpenGL on PDAs would have lower
quality expectations than those using high-end workstations).
- Optimizations must improve performance for a class
of programs where the class of programs must be larger than a single
SPECopc benchmark or SPECopc benchmark suite and applicable to at least
one end user application. For any given optimization a system must
generate correct images with and without said optimization. An
optimization must not reduce system stability.
- The vendor encourages the implementation for
general use (not just for running a single SPECopc benchmark or SPECopc
benchmark suite). As an indicator that the implementation is suitable for
general use, graphics configurations submitted for the SPECopc benchmark
suite must be able to run the corresponding SPECapc application benchmarks
if applicable.
- The implementation is generally available,
documented and supported by the providing vendor.
- It is expected that vendors would endorse the
general use of these optimizations by customers who seek to achieve good
application performance.
- No pre-computed (e.g. driver cached) images,
geometric data, or OpenGL state may be substituted within an SPECopc
benchmark on the basis of detecting that said benchmark is running (e.g.
pattern matching of command stream or recognition of benchmark's
name).
- Every OpenGL implementation in both immediate and
display list mode must fully process every GL element presented to
it that will impact the frame buffer and GL state.
- Differences to the frame buffer between immediate
and display list modes must not exceed 0.01% of the number of pixels in
the window.
- In the case where it appears the guidelines in
this document have not been followed, SPECopc may investigate such a claim
and request that the optimization in question (e.g. one using SPECopc
benchmark-specific pattern matching) be removed and the results
resubmitted. Or, SPECopc may request that the vendor correct the
deficiency (e.g. make the optimization more
general purpose or correct problems with image generation) before
submitting results based on the optimization.
- Membership
- Membership
- Membership in the SPECopc is open to any
organization that has a direct and/or material interest in OpenGL graphics
performance benchmarking.
- Members are expected but not required to be active
participants developing and improving SPECopc benchmarks.
- Members are entitled to secure access to
development code.
- Members are entitled to unlimited publication
rights.
- New members become eligible for voting on the
2nd consecutive qualified meeting. The first qualified meeting
may have been attended prior to becoming a member. Qualified meetings are
defined in Section 2.04(b).
- A member maintains voting rights by attending 1
out of the last 3 qualified meetings. A member loses their voting rights
upon missing 3 consecutive qualified meetings.
- A member regains voting rights on attending a
second consecutive qualified meeting.
- Associate Status
- Associate status is available to non-profit
organizations.
- All SPECopc, GPC and SPEC Rules and Rights apply
to Associates unless specifically stated otherwise.
- Associates are entitled to secure access to
development code.
- Associates do not have voting rights.
- Officers and Elections
- On an annual basis the SPECopc will elect from its
membership the following officers:
- Chairperson
- Vice Chairperson
- Secretary-Treasurer
- The Chairperson's responsibilities are to
- conduct meetings,
- send out the agenda on time,
- conduct votes on time,
- deal with outside organizations such as the
press,
- police the submission, review and appeal
processes.
- The Vice-Chairperson's responsibility is to do the
chairman's job when the chairman is not available.
- The Secretary-Treasurer responsibilities are
to:
- record minutes,
- maintain the rules document,
- keeps a history of email,
- track finances and interact with the GPC and SPEC
Board in that regard.
- Meetings
- The SPECopc has three types of meetings (not
including sub-committee meetings)
- Regular quarterly face to face meetings
- Special SPECopc face to face meetings for the
full membership
- Conference Call meetings
- SPECopc meetings
which qualify for attendance are limited to:
- face to face meetings scheduled one month in
advance and
- conference calls scheduled two weeks in advance.
- Membership Dues and Billing
- Dues for the SPECopc will be set annually by the
SPEC Board of Directors with input from the SPECopc. Once set, the dues
amount will be recorded in the SPEC minutes and communicated to the
SPECopc by the SPEC office.
- Dues payment, purchase order, or letter of intent
must be received at the SPEC office in time for the January annual
meeting. Dues must be paid by the end of February. Failure to meet these
deadlines will result in loss of membership and voting rights which will
be reinstated when full payment is received at the SPEC office.
- Non-Member Publication
- The SPECopc will accept submissions from
non-members for review and publication on the SPEC public website.
- Non-member submissions must follow the same
procedures as member submissions.
- Non-members are not eligible to participate in
reviewing results.
- Non-members will be charged per system
configuration for their submissions. Any change in hardware or software
constitutes a new configuration.
- On an annual basis the SPECopc will establish the
pricing for non-member publication. The amounts will be recorded in the
SPECopc minutes.
- A configuration will be published on-line for one
year, unless the submitter notifies the publisher that it should be
removed.
- After one year, the configuration will be removed
automatically, unless the submitter notifies the publisher that it should
remain on-line.
- There are no additional non-member fees for
extending on-line publication beyond one year.
- The SPECopc project group may remove published
results due to benchmark revision. In this case, the submitter will be
given notice by the project group and may, at no charge, resubmit the
identical configuration for the revised benchmark.
- Benchmarks
- Benchmark Acceptance
- Benchmark components are defined as
- code sets (e.g. SPECviewperf�, SPECglperf�),
- run rules, scripts and associated data sets (e.g.
viewsets or SPECglperf script).
- New or modified benchmark components require a
2/3-majority vote of the SPECopc electorate to be accepted for
publication.
- A minimum 3-week review period is required for new
or significantly modified benchmark components.
- At the end of the review period a vote will be
called to approve the proposed changes.
- An amendment to a benchmark component during the
review period must be unanimously accepted. If not, the review period
shall be restarted.
- It is the option of any future SPECviewperf Viewset
author(s) to require passing of selected conformance tests prior to
submission of results for that viewset.
- Benchmark Code Versioning
- Benchmark code is defined as the set of source
code required to build and run a benchmark executable (e.g. SPECviewperf and SPECglperf).
- SPECglperf Benchmark code uses the following version coding:
M.m.p (e.g. 3.1.2) M is the major release
number, m is the minor release number and p is the patch level.
- The major release number is only incremented
when large amounts of code are changed and the scripting language is
dramatically changed as a result -- backward compatibility is highly
unlikely when moving scripts or data sets between major releases (e.g.
running v2 scripts on a v3 executable would almost certainly
fail).
- The minor release number is bumped if some small
set of code is replaced or removed - but the standard, unchanged scripts
and data sets, as a whole, must run on the new version (but perhaps with
different performance).
- Patch releases can contain additions of new
properties and additions of new attributes to existing properties, but
cannot change or remove any existing properties, attributes or
functionality. These are typically used for bug fixes, small
enhancements and so forth.
- SPECviewperf Viewset
Versioning
- The version of a SPECviewperf viewset should
be incremented if:
- changes to SPECviewperf affect the performance of the viewset,
- or changes to the Viewset script affect performance,
- or if the viewset data
changes,
- or if rule changes affect the acceptance
criteria.
- New results for the previous version of a Viewset will no longer be published.
- SPECglperf Script Versioning
- The version of a SPECglperf script should be incremented if:
- changes to SPECglperf
affect the performance of the script,
- or changes to the SPECglperf script can affect performance,
- or if rule changes affect the acceptance
criteria.
- Benchmark Run Rules
- Benchmark Run Rules
- The system under test must perform all of the
OpenGL functionality requested by the benchmark with the exception that
the system does not have to support dithering.
- The systems under test must be OpenGL Conformant
for the pixel format or visual used by the benchmark.
- Settings for environment variables, registry
variables and hints must not disable compliant behavior.
- No interaction is allowed with the system under
test during the benchmark, unless required by the benchmark.
- The system under test can not skip frames during
the benchmark run.
- It is not permissible to change the system
configuration during the running of a given benchmark. For example, one
can not power off the system, make some changes, then power back on and
run the rest of the benchmark.
- Screen grabs for SPECviewperf will be full window size.
- The monitor must support the stated resolution and
refresh rate and must fully display all of the benchmark tests being
submitted.
- Results to be made public must be run by official
scripts that may not be changed, with the following exceptions (which must
be documented if not the default):
- In SPECviewperf:
- specific selection of visual/pixel format on a
per-test basis
- the multithreading flag (-th) on approved multi-threading viewsets
- Visual/pixel format required:
- May be selected on a per-test basis by
submission of the viewset script.
- If RGB visual/pixel format is requested, it
must have at least eight bits of red, eight bits of green and eight
bits of blue.
- If destination alpha is requested, it must
have at least 1 bit.
- If depth buffer is requested, it must have at
least 16 bits of resolution.
- Screen resolution must be large enough to run the
individual tests at their requested window size, with no reduction or
clipping of test window.
- Tests may be run with or without a desktop/window
manager, but must be run on some native windowing system.
- Submission and Review
Rules
- Submission Preparation Rules
- The rules for the submission and review cycle to
be used are those posted on the SPECopc web site two weeks prior to the
submission deadline.
- The benchmark versions to be used for a submission
are those posted on the SPECopc web site two weeks prior to the submission
deadline.
- All benchmark sources for a submission must be the
same as that posted on the SPECopc web site two weeks prior to the
submission deadline.
- Members who wish not to review the submission of
other specific members due to conflict of interest must submit that list
to the SPEC office prior to the submission deadline. The SPEC office will
hold the list in confidence from other members.
- Submission Content Rules
- The information supplied must reflect the system
as tested.
- All fields in the configuration description file
must be supplied.
- A date must be supplied for 'General Availability'
that is accurate for the entire system - hardware, software, O/S, drivers,
etc.
- If the system as tested is not available from the
submitter through the normal ordering process, the "Comments" area of the
results page must describe how the system may be acquired.
- Date fields must always contain a valid date.
"Now" is not valid in a date field.
- A SPECviewperf
submission can be for one or more viewsets per
configuration.
- Price includes system and monitor as
tested.
- The color depth used must be at least 24 bits
(true color).
- The display raster resolution must be at least
1280 pixels by 1024 pixels.
- The monitor refresh rate must be at least 75Hz.
This requirement does not apply to digital flat panel displays.
- Alternate currency from the US dollar can be
submitted as price and the submission will be sorted separately on the
summary pages for Price and Price/Performance.
- The submitter is required to declare sufficient
information to reproduce the performance claimed. This includes but is not
limited to:
- non-default environment variables,
- non-default registry variables,
- hints,
- compiler name and version,
- compiler command line,
- changes to the standard makefiles.
- Any information required to be reported such as
non-default environment variables, registry variables or hints, that does
not have a predefined field must be documented in the "Comments" area of
the results page.
- Valid submissions must include screen captures if
required by the benchmark.
- The SPECviewperf binary
must be submitted if it is not one of the standard binaries.
- Results previously published for a system can be
resubmitted.
- Previously published results being re-submitted
can only have price changes.
- The SPECviewperf
submission upload file must have the structure defined in Figure
4.03-1.
Figure 4.03-1
- Each member company must ensure that the upload
file contains data for all the new configurations and existing published
configurations they wish to continue publishing.
- Standardized cache nomenclature are as
follows:
- (D+I) is a Unified instruction and data
cache
- (D/I) is a for separate instruction and data
caches
- A number followed by KB or MB can be used to
describe the size of the cache.
- Caches dedicated to a processor are listed as
per processor cache size.
- Caches shared by multiple processors are listed
by total size.
- Each component of the submitted software
configuration (including the graphics driver) shall be:
- uniquely identified,
- available to SPECopc members, upon demand, by
the submission deadline and for the duration of the review
process,
- available to the public by the publication date, with
continued availability at least through the next submission
deadline.
- Subsequent to publication, replacing elements of a
submitted configuration must not result in more than a
5% performance degradation in any of the submitted benchmark
results. The submitted results for this configuration will be removed from
the SPEC public website, if this requirement is not met.
- On or before the date of publication, the
submitted configuration shall be available for purchase by the public, for
the specified price or less, with a firm delivery date of 60 days or
less.
- Submission Process
Rules
- Each benchmark (SPECviewperf, etc.) is considered a separate
submission.
- Submissions of each benchmark (SPECviewperf, etc.) must be in separate tar/zip files.
- The submission file names must contain opc_v for SPECviewperf and
opc_g for SPECglperf,
contain all lower case letters and not contain '.' except prior to the zip
or tar file extension (e.g. intel_opc_v_jun99_v0.zip). The file version is
denoted prior to the file extension. The initial file version is v0.
Resubmitted files must increment the version number.
- A submitter of SPECopc benchmark results must
upload their submission to the proper server location by the submission
deadline date. The submitter must not create any new directories on the
server when uploading their submission.
- The submitter must notify SPEC Office after a
submission is uploaded to the server prior to the submission deadline with
contact information for questions about the submission.
- The submitter must contact the SPEC office if they
have attempted to upload their submission and were not successful.
- The SPEC office will
not disclose who has submitted results until the submission
deadline has passed.
- Submissions will not be accepted after the
submission deadline.
- The upload directory will be set to write-only
until the submission deadline has passed. Then it is set to read-write
(not modify) after the submission deadline.
- If a submitter is notified that their submission
format is incorrect, they must re-send their submission in proper format
within 3 business days of notification.
- Abuse of the resubmission allowance is grounds for
rejection of a submission.
- Review Period
Rules
- SPECopc members must keep other members' submitted
results SPECopc-confidential until they are publicly available.
- SPEC Office pairs reviewers to submitters.
- SPECviewperf review pools will be independent of each other.
The SPEC office will send the list of contact information for the
submissions under review.
- All members will have access to all benchmark
submissions once the review period begins.
- The review period shall be 10 calendar
days.
- Submissions cannot be withdrawn during the review
process.
- If a primary reviewer has a question with a
submission they must pose the question to the submitter first.
- Any reviewer who has questions with a submission
must :
- Pose these questions to the submitter and cc the
primary reviewer OR,
- Pose these questions to the primary reviewer.
The primary reviewer must then pose these questions to the submitter
OR,
- Pose these
questions to an officer of the SPECopc. The officer of the
SPECopc must then pose these questions to the submitter and cc the
primary reviewer.
- The submitter can request that their submission be
rejected on stated technical grounds.
- With public permission of the primary reviewer, a
submitter may resubmit their submission. The submitter must notify the
gpcopc mailing list with the date and version of
the resubmitted file.
- The submitter must provide the primary reviewer
access to the system under test at the submitter's facilities if requested
by the reviewer during the review period. The reviewer must state prior to
the visit what part of the submission is going to be verified. Travel
expenses are the responsibility of the reviewer.
- Previously published results being re-submitted
can only be reviewed for consistency with the previous submission, and
price changes.
- Price can be challenged. If so, the submitter must
provide documentation that the system can be purchased for the price
quoted. Price must be valid for 90 days from date of publication. Quantity
1 pricing must be used.
- Reviewers will decide if the image quality of the
submission is sufficiently adherent to the OpenGL
specification to satisfy the intended end user's expectations. If a
reviewer rejects the quality of an image for a stated reason, the
submitter can ask for a vote of the full SPECopc electorate. In case of a
tie the submission is rejected.
- System configurations submitted for the SPECopc
benchmark suite must be able to run the corresponding SPECapc application
benchmarks if applicable. If this criteria is not
met the submission will be rejected.
- By the end of the review period, the primary
reviewer of a submission must approve it without comment, approve it with
comment, or reject it with comment. The submitter may appeal a rejection
as described in Section 5.05.
- Comments for rejection of a submission received
after the end of the review period will not delay publication of the
submission.
- Review Appeal
Rules
- The appeal period shall be 2 weeks, and
immediately follow the review period.
- Any submitter of a rejected submission can make
their case on the gpcopc email alias during the
appeal period.
- At the end of the appeal period, if there is no
resolution, the Chair shall call a vote to approve or reject the
submission.
- The whole SPECopc electorate votes on approval or
rejection of an appealed submission. A simple majority of the SPECopc
electorate is required to approve or reject the appeal. In case of a tie
the submission is rejected.
- Challenging Approved
Results
- Any member may challenge approved results at any
time. This includes:
- archived results,
- currently published results.
- The burden of proof that the result should be
modified is on the member who is
challenging the result.
- The challenge must be ratified by a majority vote
of the committee.
- The Chair will call a special review cycle in the
event that there is a ratified challenge to currently published results .
- A ratified challenge to archived results can only
result in annotation, not removal or modification. The annotation will be
determined by the majority of the committee.
- Publication Rules
- SPECopc Publication
- Benchmark results for publication by the SPECopc
must adhere to Articles I, IV and V.
- Unofficial Publication
- Benchmark results for publication elsewhere (e.g.
industry journals, vendor web sites, analyst reports) must adhere to
Articles I and IV.
- The SPECopc or any SPECopc member reserves the
right to request and receive evidence that the published results have been
achieved in accordance with the rules and that published information is
accurate.
- SPECopc metrics may be estimated. Metrics shall
not be estimated for configurations that are capable of running the
benchmark. All estimated metrics must be clearly identified as estimated.
Licensees are encouraged to publish actual SPECopc metrics as soon as
possible. Proper trademark usage for estimated results would be in the
following forms:
- SPECviewperf Awadvs-04 estimated score of 30 fps
-
SPECviewperf |
Awadvs-04 |
30 |
est. |
SPECviewperf |
Light-04 |
122 |
est. |
- Adoption
Adopted by the SPECopc on January 23,
2003.
Changes for version
1.1 adopted June 10, 1999.
Changes for version 1.2 adopted January 12,
2000.
V1.4 changes -- 5.02 (d) added
V1.5 changes --
4.01.i.2(2), 4.01.i.2(4), 5.02.w
V1.6 changes -- 1.03.c,
5.04.o