Graphics & Workstation Performance Group Charter
Revised March 26, 2009
The Graphics & Workstation Performance Group, commonly referred to as the GWPG, is the representative body of
the GWPG member organizations. The GWPG is an approved committee of the Standard Performance Evaluation
Corporation (SPEC), and as such, operates under the bylaws and guidelines of SPEC.
1.0 Objectives
To permit standardized graphics & workstation performance measurement and evaluation by creating unambiguous
vendor neutral measures for product evaluation and comparison.
To develop specifications, programs, and benchmarks for generating accurate performance measures in an open,
accessible and well-publicized manner.
To provide tools, if applicable, which are available as examples for vendors and customers to use for their own
performance measurement and evaluation needs.
To provide software releases to the public in a timely fashion.
To validate the specification, programs, and benchmarks by the development of a publicly available implementation.
To contribute to the coherence of the field of graphics & workstation performance measurement and evaluation so that
customers will be better able to compare and evaluate vendor's products and environments.
To refrain from performing actual performance measures or evaluations.
1.1. SPEC General Policies and Principles
The intent of this section is to provide an overview of the general policies and principles of the SPEC organization, as
context for the charter of the Graphics and Workstation Performance Group. The controlling documents for
governance of SPEC are the Articles of Incorporation and Bylaws, filed with the State of California, and available
from the SPEC Administrator.
In the event of disagreement between this document and SPEC's Bylaws, or disagreement between this document and
decisions by the SPEC Board of Directors, said Bylaws or decisions shall take precedence over this document.
2.0 Activities
The GWPG, through Approved Projects, is developing a suite of standardized specifications that provide a
standardized method of measuring graphics & workstation related performance for different hardware platforms. The
Approved Projects are the responsibility of standing Subcommittees of the GWPG.
The current Approved Project Groups are the Graphics Performance Characterization (GPC) and the Application
Performance Characterization (APC) Groups.
2.1 The Graphics Performance Characterization Group (SPECgpc) is establishing graphics performance benchmarks
for systems running under the OpenGL application programming interface (API) or other graphics APIs.
2.2 The Application Performance Characterization Group (SPECapc) is establishing a broad-ranging set of
standardized benchmarks for graphics & workstation related applications.
2.3 Other Approved Project Subcommittee Upon approval of a new project, which meets the criteria
outlined in the Objectives listed above, a new corresponding standing Subcommittee may be added by a majority vote
of the GWPG.
3.0 Responsibilities
3.1 GWPG
The GWPG is the administrative body for the operations of the GWPG organization and its standing Subcommittees.
The GWPG reviews and monitors all activities for compliance with these procedures and for compliance with the
bylaws of SPEC.
The GWPG provides administrative management for the activities of its Subcommittees and is responsible for
reviewing and approving all new project applications.
3.2 GWPG Approved Projects
The GWPG Approved Projects are assigned to standing Subcommittees which are the operational bodies responsible
for the development, production and management of specifications, programs, and benchmarks for performance
measurement. Projects are approved by a two-thirds (2/3) vote of the members of the GWPG. All final releases of
rules and specifications applicable across multiple projects, programs, and benchmarks shall be approved by the
GWPG prior to release.
The Subcommittees will be responsible for the work to develop, evaluate and approve the specifications, programs and
benchmarks. All Subcommittee working documents and records of the Subcommittee's actions, methods and
procedures shall be retained for review and inspection. The specifications, programs, and benchmarks described
herein will be made available to all interested parties at the earliest possible release dates.
The specifications, programs, and benchmarks will list the names and affiliations of individuals participating in the
development effort.
4.0 Structure
The cost of the GWPG and its Subcommittees will be paid via membership fees or dues. Any organization can
become a sustaining member by payment of the membership fee or annual dues.
The GWPG will put in place and maintain an appeals process through which any interested or affected party may
register and have heard a complaint concerning GWPG or Subcommittee action or inaction on any Approved Project.
Meeting locations will rotate as appropriate. Potential locations will be provided to the SPEC office and final
decisions will be approved by the GWPG.
Face-to-Face Meetings of the GWPG and Subcommittees will be held at least four (4) times a year. Additional
meetings will be called as appropriate with sufficient advance notice. Any such meetings may be convened in person
or by electronic means. Agendas will be prepared and circulated to all members in advance of meetings.
Meeting minutes will be kept by the elected secretary and issues raised at meetings or through electronic means, will
be recorded in the manner of standards discussion. An electronic mail archive will be kept.
GWPG and Subcommittees will adhere at all times to the principles of due process and antitrust compliance as set
forth in detail in the Guidelines for Handling SPEC Information.
Exceptions to policy: For due cause, the GWPG may vote to approve exceptions to this policy or may adopt
resolutions that may conflict with policies expressed in this document; in such cases an amendment to this document
will be brought forth within a reasonable time period afterwards.
Conflict with Bylaws or Board: In the event of disagreement between GWPG resolutions vs. SPEC's Articles of
Incorporation and Bylaws or decisions by the SPEC Board of Directors, the GWPG shall yield.
Election timing: To maintain continuity and consistency of representation on the GWPG, elections are held at the
annual meeting.
Affiliation: Entering into a relationship of affiliation (as defined in the SPEC Bylaws, Section 2.1) does not alone
constitute the creation of a vacancy. (For example, Company B is not a member of the GWPG or one of its groups. It
buys all the assets of, and takes on all the responsibilities of, Company A, including Company A's OSSC seat.) If,
however, such affiliation would cause or appear to cause a member and its affiliate(s) to have two or more votes on the
GWPG, then, as described in the Bylaws, a single voting representative shall be designated and one or more vacancies
shall be declared.
Voting representative: Each GWPG member institution shall designate a voting representative. If the designated
voting representative is absent from a GWPG meeting, another person employed by the member institution (or its
affiliates) may temporarily represent the member. In the event of a lack of clarity as to who has the vote, the GWPG
Chair may refuse to accept a vote until the designated voting representative appoints a substitute in writing.
Proxies: The GWPG does not allow use of proxy voting (that is, the authorization of an individual who is not an
employee of a member institution to vote on behalf of that member), because GWPG members are expected to
participate, to be actively involved in discussion, and to evaluate the evidence that may be presented. All GWPG votes
shall be open and public votes, except election of officers, which is by secret ballot. Exceptions to the secret ballot
may be made if there is only one candidate per office.
It should be noted that the lack of passage of a motion by the GWPG does NOT imply approval of the CONVERSE of
the motion.
Advisers: GWPG can, after notification in writing to the Board of Directors, enlist the aid of advisers and consultants
from the industry and academe.
Coordination with Board: Certain organizational activities require board participation and approval. For example,
the board must approve press releases, setting prices for benchmark licenses, and any major resource, reallocations of
SPEC administrative staff and Public Relations to assist with the non-technical aspects of releasing a new benchmark
suite. This work may be facilitated by a joint committee of the board and the GWPG (e.g. Planning Committee) or
proposals may be brought directly to the board by the Group or Project Group Chair.
5.0 Membership
Terminology note about SPEC "Membership": SPEC is a California nonprofit mutual benefit corporation.
governing law has certain formal terminology regarding membership (see, for example, California Corporations Code,
sections 5056 and 7331). SPEC's Bylaws authorize only one class of membership in the corporation. It is understood
that one sometimes sees the term "member" used in a looser sense, and indeed the law appears to recognize that the
term "member" may be used in a less formal sense (7333). However, this section 5.2 follows the formal usage of the
Bylaws, which clearly say that SPEC has only one class of "Membership".
Organizations are the sustaining members of the GWPG and its subcommittees. Organizational membership is attained
by submission of the designated membership fees or annual dues of any of the GWPG subcommittees. The
membership fees and annual dues are set by the Subcommittee and approved by a majority vote of the GWPG.
5.1 Participation in GWPG
Participation in GWPG is open to any organization that has a direct and/or material interest in workstation or graphics
performance benchmarking. Members of the GWPG are active participants in one or more Subcommittees.
All members are eligible to nominate candidates for offices and the SPEC Board of Directors, and to champion
proposed benchmarks. All members have the opportunity to review and comment on all benchmarks developed by
GWPG. This can be done through active participation in the subcommittee developing the suite or during the
benchmark's final call for comments.
5.1.1 Participation in Subcommittees
Participation in Subcommittees is open to any organization that has a direct and/or material interest in scope of
performance benchmarking for the Subcommittee. In addition, a supporting role is open to any individual who has a
direct and/or material interest in the scope of activity and is qualified as a technical expert in this scope of activity.
Supporting individuals or organizations participate only in the technical meetings of the Subcommittee.
5.2 Participation Levels
5.2.1 Sustaining Membership
Sustaining membership is offered to any organization that remits the annual membership fee or annual dues. This
entitles the organization to full voting membership in organization meetings of the GWPG and Subcommittees.
Universities and non-profit institutions are eligible to become sustaining members, but at their option may prefer to
join as "Non-profit Associates", as described in section 5.2.2.2.
Sustaining members may become active participants in any working group or subcommittee of GWPG.
5.2.2 Non-Membership Roles
SPEC does make an effort to accommodate the participation of academic institutions and other non-profit
organizations that might be unable to meet the full requirements for Sustaining Membership. Those applying for Non-
Membership roles must be willing to abide by SPEC's Bylaws, the Fair Use guidelines, the Guidelines for handling
SPEC-confidential material, and GWPG's Charter. Accommodations are made by special arrangement of the GWPG
and can include reduced (or waived) dues and other arrangements. These accommodations generally fall into three
categories, none of which have voting rights:
- Ex Officio, may be granted to any organization or individual that can contribute to the overall success of the
program,
- SPEC Associates, those which pay a reduced fee and receive full benefits, and
- SPEC Supporting Contributor, whose contributions and access is restricted to the one specified benchmark
development process.
5.2.2.1 Ex Officio
Ex Officio may be granted to any organization or individual that can contribute to the overall success of the program.
Ex Officio is reviewed annually and requires a two-thirds (2/3) vote of the members of the GWPG or respective
Subcommittee. Ex Officio do not have voting rights.
5.2.2.2 Associates
To encourage participation from universities and other non-profit institutions interested in its work, SPEC maintains a
SPEC Associate role that enables non-profit organizations to share in the SPEC process, but (since they are not full
members) does not allow for voting on benchmarks.
Associates pay an initiation fee and annual dues as determined by the Board of Directors.
Associates have the opportunity to review and comment on all benchmarks developed by the GWPG. This can be done
through active participation in the subcommittee developing the suite or during the benchmark's final call for
comments.
5.2.2.3 Supporting Contributors
The role of Supporting Contributor is available to any commercial organization, academic institution, or individual by
invitation only. A subcommittee may issue an invitation to an interested party to apply for this role. The application
must be approved by the GWPG and the Board.
Supporting contributors may participate in subcommittees and have no voting rights and pay no dues. Supporting may
help develop new benchmark suites, provide expertise to the GWPG or help provide other support within the GWPG.
5.2.2.2 Table of Benefits
|
SUSTAINING |
ASSOCIATE |
EX
OFFICIO/SUPPORTING |
| DUES |
FULL |
DISCOUNT |
NONE |
| GENERAL MEMBERSHIP VOTING |
YES |
NO |
NO |
| PARTICIPATE IN FACE-TO-FACE
MEETING, CONFERENCE CALLS,
BENCHMARK DEVELOPMENT |
YES |
YES |
YES |
| GWPG WORK IN PROGRESS |
YES |
YES |
YES - if providing support
for code;
otherwise NO. |
| ELIGIBLE TO ELECT GWPG
OFFICERS |
YES |
NO |
NO |
| ELIGIBLE TO VOTE FOR BOARD
MEMBERS |
YES |
NO |
NO |
| ELIGIBLE TO PARTICIPATE IN
ANY GWPG SUBCOMMITTEES |
ANY/ALL |
ANY/ALL |
ANY/ALL IF SUPPORT
IS PROVIDED |
| COPIES OF GWPG SUITES |
ALL |
ALL |
ALL |
| SPEC/GWPG EMAIL ALIAS |
ANY/ALL |
ANY/ALL |
Any Supporting |
| ACCESS TO PRO.SPEC.ORG |
YES |
YES |
LIMITED - Controlled by
Chair to selected pages
only. |
| RESULT PUBLICATION ON
WWW.SPEC.ORG |
UNLIMITED |
UNLIMITED |
Dependent on new suite
support; otherwise, non-
member price set by
GWPG. |
| PARTICIPATION PERIOD |
Dues Paid and
meeting other membership Requirements |
Dues Paid and meeting other participation
Requirements |
ONE YEAR,
RENEWABLE.
GWPG may terminate at
any time. |
| RESULTS REVIEW |
YES |
YES |
May participate in the
results review session for
which results were
submitted. |
| ATTEND ANNUAL MEETING |
YES |
YES |
YES |
| ELIGIBLE FOR GWPG OFFICE |
YES |
YES |
NO |
5.3 Officers of Committees and Subcommittees
The officers for the GWPG and each Subcommittee shall be the Chairperson, Vice Chairperson and the
Secretary and such subordinate officers as may be appointed by the Chairperson.
The terms of the Chairperson, Vice Chairperson and Secretary shall be one year and each officer shall be
elected by a simple majority of the voting members of the respective committee or subcommittee. The
number of terms of any officer shall be unlimited.
The Chairperson shall preside at all meetings. The Chairperson may designate the Vice Chairperson or, in
the latter's absence, another alternate to serve in his or her place.
The Vice Chairperson shall assist the Chairperson in all duties of the office.
The Secretary shall assure that all activities are accurately recorded. The secretary shall also assure that all
records of membership and voting status are maintained.
The Secretary is responsible for preparing and distributing the minutes of all official meetings.
These responsibilities may be carried out with the assistance of SPEC.
5.4 Voting in Organization Sessions of GWPG and Subcommittees
Organization Sessions of GWPG and Subcommittees are those meetings that are convened to address issues
that are deemed necessary to conduct the business of the GWPG or Subcommittee. These issues are left to
the specific agenda of the GWPG or Subcommittee, but are subject to appeal under the appeal process
outlined in the Guidelines Section of these procedures.
Organizations are the voting members in Organization Sessions of the GWPG, and Subcommittees. Each
member organization is allowed one vote on any issue. Organizations may send as many representatives as
necessary to represent their interests. (This number may be limited if meeting facilities or SPEC is
constrained.)
Voting status is achieved by in-person attendance at the second consecutive official (qualified) meeting of
the GWPG or Subcommittee.
Voting status is lost if the organization fails to remit payment for membership fees or annual dues. Voting
status is restored by payment of these fees or dues.
Voting status is also lost if the organization is not represented in person during at least two consecutive
official meetings of the GWPG or subcommittee. Voting status is restored at the second consecutive official
meeting after attendance has been resumed.
Issues may be designated for resolution by ballot by voting members of the GWPG or Subcommittee. Ballot
may be by standard mail, by electronic means, conference call voice ballot or a combination of any of the
three. A ballot is deemed valid if a quorum of eligible voting organizations returns ballots. Voting is
Approved, Approved with Comment, Disapproved with Comment, or Abstained with Comment.
Disapproval and Abstained votes require comment on the nature of the vote.
5.5 Voting in Technical Sessions of Subcommittees
Technical Sessions of Subcommittees are those meetings in which discussions are based on the technical
merit of the issues being addressed. These issues are left to the specific agenda of the Subcommittee, but are
subject to appeal under the appeal process outlined in the Guidelines Section of these procedures.
The Graphics and Workstation Performance Group (SPEC/GWPG):
Rules For Project Groups
Version 1.10
Last Updated: 09/13/2007
- Overview
- Project Groups and Scope of Rules
- Two project groups exist under the umbrella
of SPEC/GWPG:
- The Graphics Performance Characterization
Project Group (henceforth abbreviated
as SPECgpcSM)
- The Application Performance Characterization
Project Group (henceforth abbreviated
as SPECapcSM)
- The rules contained in this document
shall apply to both project groups (SPECgpc
and SPECapc).
- Each project group shall maintain its
own project group rules document, which
shall apply in conjunction with this
document. - SPECgpc
Project Group Rules - SPECapc
Project Group Rules
- Where a project group's rule overrides
a rule in this document, this will be
explicitly indicated in that project
group's rule document.
- Should a new project group be approved
or an existing one dissolved, this document
shall be updated accordingly.
- Membership
- Membership
- Membership in the SPEC/GWPG and its project
groups
is open to any organization that has
a direct and/or material interest in
graphics or workstation-related application
performance benchmarking.
- Membership
in one or more SPEC/GWPG project
groups leads to membership of SPEC/GWPG.
- Members
are expected but not required to
be active participants developing
and improving the respective project
group's benchmarks.
- Members
are entitled to secure access to
development code.
- Members
are entitled to unlimited publication
rights.
- New
members become eligible for voting
on the 2nd consecutive qualified
meeting. The first qualified meeting
may have been attended prior to becoming
a member. Qualified meetings are
defined in Section II.4.b.
- A
member maintains voting rights by
attending 1 out of the last 3 qualified
meetings. A member loses their voting
rights upon missing 2 consecutive
qualified meetings.
- A
member regains voting rights on attending
a second consecutive qualified meeting.
- For
a qualified meeting for which attendance
in person is expected, attending
remotely (e.g. by telephone) does
not count as qualified attendance.
- Voting
status is lost if the organization
fails to remit payment for membership
fees or annual dues. Voting status
is restored by payment of these fees
or dues.
- Associate
Status
- Associate
status is available to non-profit
organizations.
- All rights and rules of the respective
project group, GWPG and SPEC apply to
Associates unless specifically stated
otherwise.
- Associates are entitled to secure access
to development code.
- Associates do not have voting rights.
- Officers and Elections
- On an annual basis the project groups
will elect from their eligible voting
memberships the following officers:
- Chairperson
- Vice Chairperson
- Secretary
- The Chairperson's responsibilities are
to
- conduct meetings,
- send out the agenda on time,
- conduct votes on time,
- deal with outside organizations
such as the press,
- represent and respond on behalf
of the group to external questions
and queries,
- interact with the SPEC/GWPG committee,
and
- police the submission, review
and appeal process.
- The Vice-Chairperson's responsibility
is to do the chairperson's job when the
chairperson is not available,
or if the chairperson is subject to a
conflict of interest.
- The Secretary's responsibilities are
to:
- record minutes,
- maintain the rules document,
- keep a history of email.
- If
an officer is subject to a conflict
of interest in pursuance of his or
her duties and if any defined succession
of responsibility would fail to resolve
the conflict of interest, the committee
may appoint any committee member
to fulfill that officer's duties
for the scope of the discussion in
which the conflict of interest exists.
- Meetings
- SPEC/GWPG project groups have three types
of meetings (not including ad-hoc working-group
meetings)
- Regular quarterly face-to-face
meetings
- Special face-to-face meetings
for the full membership
- Conference-call meetings
- Meetings
which qualify for attendance are
limited to:
- face-to-face
meetings scheduled at least
one month in advance and
- conference
calls scheduled at least
two weeks in advance and
which are explicitly indicated
as qualified at least two
weeks in advance.
- Voting
- Issues
may be designated for resolution
by ballot by voting members of the
GWPG or Subcommittee. Ballot may
be by standard mail, by electronic
means, conference call voice ballot
or a combination of any of the three.
A ballot is deemed valid if a quorum
of eligible voting organizations
returns ballots. Voting is Approved,
Approved with Comment, Disapproved
with Comment, or Abstained with Comment.
Disapproval and Abstained votes require
comment on the nature of the vote.
- A
valid vote requires a quorum. A quorum
is met if at least 66% of eligible
voting members respond.
- Membership Dues and Billing
- Dues are assessed on the basis of membership
of SPEC/GWPG's project groups.
- Dues for the SPEC/GWPG project groups
will be set annually by the SPEC Board
of Directors with input from the SPEC/GWPG.
Once set, the dues amount will be recorded
in the SPEC minutes and communicated
to the SPEC/GWPG by the SPEC office.
- Dues payment, purchase order or letter
of intent to pay for a given calendar
year must be received at the SPEC office
by March 1st of that year. Alternatively,
a letter of intent to join the respective
project group must be received by the
SPEC office by March 1st of that year
with a subsequent dues payment by May
1st of that year. Failure to meet these
deadlines will result in loss of membership
and voting rights. Membership will be
reinstated when full payment is received
at the SPEC office. Voting rights will
be reinstated according to the attendance
rules in section II.1.g and II.1.h.
- Non-Member Publication
- The SPEC/GWPG project groups will accept
submissions from non-members for review
and publication on the SPEC public website.
- Non-member submissions must follow the
same rules and procedures as member submissions.
- Non-members are not eligible to participate
in reviewing results.
- Non-members will be charged for their
submissions according to an approved
fee structure. Any change in hardware
or software constitutes a new configuration.
- On an annual basis the SPEC/GWPG will
establish the pricing and periods for
non-member publication. These will be
recorded in the SPEC/GWPG minutes and
published on the GWPG web-site.
- Following acceptance by the assigned
reviewers, a non-member's submission
will not be published until the SPEC
office has received the submission fee
in full.
- The SPEC office will not deposit funds
provided by the non-member submitter
until the submission has been accepted
by the assigned reviewers.
- A configuration will be published on-line
for six months, unless the submitter
notifies the publisher that it should
be removed.
- After six months, the configuration will
be removed automatically, unless the
submitter notifies the publisher that
it should remain on-line.
- There are no additional non-member fees
for extending on-line publication beyond
six months.
- Each SPEC/GWPG project group may remove
published results from its web pages
due to benchmark revision. In this case,
the submitter will be given notice by
the project group and may, at no charge,
resubmit the identical configuration
for the revised benchmark.
- Benchmarks
- Each project group shall document all benchmark-related
rules in its respective project group rules document.
- Submission and Review Rules
- Submission Preparation Rules
- The rules for the submission and review
cycle to be used are those approved by
the respective project group's committee
prior to the submission deadline. The
approved rules must be posted to the
respective project group's web-site by
the first publication date for the benchmark.
- Version compliance: The benchmark and
(where applicable) application versions
to be used are those approved by the
respective project group's committee
prior to the submission deadline. The
approved benchmark (and application)
versions must be posted to the respective
project group's web-site by the first
publication date for the benchmark.
- All benchmark sources for a submission
must be the same as that approved by
the respective project group's committee
prior to the submission deadline. The
approved benchmark sources must be posted
to the respective project group's web-site
by the first publication date for the
benchmark.
- Submission Content Rules
- The information supplied must reflect
the system as tested.
- Configuration description: All fields
in a submission's results file must be
supplied, unless the field names are
marked "opt.", indicating an
optional field.
- Submitters must specify a date for 'General
Availability' that is accurate for the
entire system - hardware, software, O/S,
drivers, etc.
- The "Comments" area of the
results page must describe how the system
may be acquired.
- Date fields must always contain a valid
date. "Now" is not valid in
a date field.
- Price includes system and monitor as
tested.
- Price may be submitted with an alterate
currency from the US dollar. The submission
will be sorted separately on the summary
pages for Price and Price/Performance.
- The submitter is required to declare
sufficient information to reproduce the
performance claimed. This includes but
is not limited to:
- non-default environment variables,
- non-default registry variables,
- hints,
- compiler name and version,
- compiler command line,
- changes to the standard makefiles.
- Any information required to be reported
such as non-default environment variables,
registry variables or hints, that does
not have a predefined field must be documented
in the "Comments" area of the
results page.
- Valid submissions must include screen
captures if required by the benchmark.
- Results previously published for a system
can be resubmitted. Resubmissions do
not require the inclusion of screen capture
images.
- Previously published results being re-submitted
can only have price changes.
- Each member company must ensure that
the upload file contains data for all
the new configurations and existing published
configurations they wish to continue
publishing.
- Standardized CPU nomenclature is as follows:
- CPU / Processor: a physical package
containing one or more cores.
- Socket – Receptacle or
physical connection between processors
and the system.
- Core: set of execution units
which completely implement the
instruction set of a processor
architecture and are capable
of running one or more threads.
- Thread: Processor-directed sequence
of instructions
- All processors in the system,
the number of their cores and
the number of threads (if more
than one) a core can execute
must be disclosed in the system
description whether or not they
are directly enabled by system
software or application software.
If different from the number
physically present, the number
of processors, cores and threads
enabled must also be disclosed.
- Standardized CPU cache nomenclature is
as follows:
- (D+I) designates a unified instruction
and data cache
- (D/I) designates separate instruction
and data caches
- A number followed by KB or MB
can be used to describe the size
of the cache.
- Caches dedicated to a processor
are listed as per processor cache
size.
- Caches shared by multiple processors
are listed by total size
- Each component of the submitted configuration
(including the graphics driver) shall
be:
- uniquely identified,
- available to members of the respective
project group, upon demand, by
the submission deadline and for
the duration of the review process,
- verifiably available to the public
by the publication date, with
continued availability at least
for the life of the submission,
with sufficient information in
the comment field to enable users
to directly obtain this component.
- Price must be valid for two submission
periods from date of publication. Quantity
1 pricing must be used.
- Subsequent to publication, any change
to or replacement of elements for a submitted
configuration must not result in more
than a 5% performance degradation in
the submitted benchmark results. Upon
demonstration of such a degradation,
the submitted results for this configuration
will be removed from the SPEC public
website.
- On or before the date of publication,
the submitted configuration shall be
available for purchase by the public,
for the specified price or less, with
a firm delivery date of 60 days or less.
Submissions will be categorized as either “Single
Supplier” or “Parts Built”,
where “Single Supplier” is
defined as a configuration where
all hardware, OS, and drivers are sold
and supported by the submitting supplier. “Supported” is
defined as providing hardware, drivers
and associated technical support, and
that the drivers are available from the
system supplier. “Parts Built” is
defined as a configuration built and
supported by multiple suppliers.
- “Parts built” system pricing
must include enough detail to reproduce
all aspects of the submission, including
performance and price, and include all
hardware and O/S costs necessary to run
benchmark.
- Submission Process Rules
- Each
benchmark is considered a separate
submission.
- Submissions
of each benchmark's results (e.g.
Maya6.5™, Solidworks 2007™,
SPECviewperf, etc.) must be in separate
tar/zip files.
- A
submitter of benchmark results must
upload his or her submission to the
proper server location by the submission
deadline date and time. The submitter
must not create any new directories
on the server when uploading the submission.
- The submitter must notify SPEC Office
after a submission is uploaded to the
server prior to the submission deadline
with contact information for questions
about the submission.
- The submitter must contact the SPEC office
if they have attempted to upload their
submission and were not successful.
- The SPEC office will not disclose who
has submitted results until the submission
deadline has passed.
- Submissions will not be accepted after
the submission deadline.
- The upload directory will be set to write-only
until the submission deadline has passed.
Then it is set to read-write (not modify)
after the submission deadline.
- If a submitter is notified that their
submission format is incorrect, they
must re-send their submission in proper
format within 3 business days of notification.
- Review Period Rules
- SPEC/GWPG project group members shall
keep all submitted results confidential
to the respective project group until
those results appear on the public SPEC
web site. The exception to this rule
is that members are free to make their
own submitted results public at any time.
- SPEC Office assigns reviewers to submitters.
- Members who wish not to review the submission
of other specific members due to conflict
of interest must submit that list to
the SPEC office prior to the submission
deadline. The SPEC office will hold the
list in confidence from other members.
- The various SPECapc and SPECgpc pools
of eligible reviewers will be independent
of each other. The SPEC office will send
the list of contact information for the
submissions under review.
- All members will have access to all benchmark
submissions once the review period begins.
- The review period shall be 5 calendar
days.
- Submissions cannot be withdrawn during
the review period without cause and without
prior approval of the primary reviewer.
A submitter who is granted permission
to withdraw a submission must inform
the committee by email of the reason
for withdrawal.
- If a primary reviewer has a question
with a submission they must pose the
question to the submitter first. The
primary reviewer may also pose questions
to the respective project group's officers
or SPEC/GWPG Chair for clarification
of rules if needed.
- Any reviewer who has one or more questions
relating to a submission must:
- Pose the question(s) to the submitter
and cc the primary reviewer,
OR
- Pose the question(s) to the primary
reviewer. The primary reviewer
must then pose the question(s)
to the submitter, OR
- Pose the question(s) to an officer
of the respective project group.
The officer must then pose the
question(s) to the submitter
and cc the primary reviewer
- With permission of the primary reviewer,
as communicated through the respective
project group's email alias, the submitter
can request that their submission be
rejected on stated technical grounds.
- With permission of the primary reviewer,
as communicated through the respective
project group's email alias, a submitter
may resubmit a submission to resolve
issues found during the review process.
The submitter must notify the respective
project group's mailing list with the
date and version of the resubmitted file(s).
- The submitter must provide the primary
reviewer access to the system under test
at the submitter's facilities if requested
by the reviewer during the review period.
The reviewer must state prior to the
visit what part of the submission is
going to be verified. Travel expenses
are the responsibility of the reviewer.
- Previously published results being re-submitted
can only be reviewed for consistency
with the previous submission, and price
changes.
- If the reviewer disputes the stated pricing,
the submitter must provide documentation
that the system can be purchased for
the price quoted.
- By the end of the review period, the
primary reviewer of a submission must
designate the status of the submission
one of: “accepted without comment”, "accepted
with comment", “pending with
comment”, or “rejected with
comment”. The submitter may appeal
a rejection as described in "Review
Appeal Rules" below.
- Any comments for rejection of a submission
received after the end of the review
period will not delay publication of
the submission.
- A submission designated “pending
with comment” will not
go public and will remain pending
until the submitter addresses all
comments. Once the comments are addressed
the web master will post to the public
site. Any member who feels comments
are not satisfactorily addressed
may challenge the submission according
to Section IV.6 for challenging approved
results.
- If a submitter repeatedly makes submissions
that are non-compliant or which do not
address concerns identified in the previously-assigned
reviewers' rejection comments, the reviewer
may engage the committee to solicit appropriate
action, which may be up to and including
an embargo on submissions from that submitter
for a period of time.
- Review
Appeal Rules
- The appeal period shall have the same
duration as one submission cycle, and
shall immediately follow the review period.
- Any submitter of a rejected submission
can make their case to the respective
project group's email alias during the
appeal period.
- At the end of the appeal period, if there
is no resolution, the project group Chair
shall call a vote to accept or reject
the submission.
- The project group electorate votes on
accepting or rejecting an appealed submission.
A simple majority is required to accept
or reject the appeal. In case of a tie
the submission is rejected.
- Challenging Accepted Results
- Any member may challenge accepted results
at any time. This includes:
- archived results,
- currently published results,
and
- resubmitted results not subject
to the regular submission review
process.
- The burden of proof that the result should
be modified is on the member who is challenging
the result.
- The challenge must be ratified by a majority
vote of the project group's electorate.
- The project group Chair will call a special
review cycle for a resubmission in the
event that there is a ratified challenge
to currently published results.
- A ratified challenge to archived results
can only result in annotation, not removal
or modification. The annotation will
be determined by the majority of the
electorate. It is the responsibility
of the challenger to verify that the
results have been annotated correctly
on the public website within two working
days from the ratification of the challenge.
- Publication Rules
- Official Publication
- Benchmark results for publication by
the SPECgpc or the SPECapc must adhere
to Articles concerning "Overview", "Benchmark
Run Rules" and "Submission
and Review Rules" as presented in
this document AND the respective project
group's rules document.
- Unofficial Publication
- Benchmark results for publication elsewhere
(e.g. industry journals, vendor web sites,
analyst reports) must adhere to Articles
concerning "Overview" and "Benchmark
Run Rules" as presented in this
document AND the respective project group's
rules document.
- The respective project group or any member
thereof reserve the right to request
and receive evidence that the published
results have been achieved in accordance
with the rules and that published information
is accurate.
- SPECgpc or SPECapc metrics may be estimated.
Metrics shall not be estimated for configurations
that are capable of running the benchmark.
All estimated metrics must be clearly
identified as estimated. Licensees are
encouraged to publish actual SPECgpc
or SPECapc metrics as soon as possible.
Adoption
V1.10 adopted on 09/13/2007
(reflects transition from GPC to GWPG)
V1.04 adopted on 10/20/2006
V1.03 adopted on 08/04/2006
V1.02 adopted on 04/27/2006
V1.01 updated on 02/09/2006 to align wording with SPEC
policy
V1.00 adopted on 01/25/2006
