SPEC/GPC Project Group Rules

The Graphics Performance Characterization Group (SPEC/GPC):
Rules For Project Groups

Version 1.01

Last Updated: 02/09/2006

Overview
1. Project Groups and Scope of Rules
  1. Two project groups exist under the umbrella of SPEC/GPC:
    - The OpenGL Performance Characterization Project Group (henceforth abbreviated as SPECopc^SM)
    - The Application Performance Characterization Project Group (henceforth abbreviated as SPECapc^SM)
  2. The rules contained in this document shall apply to both project groups (SPECopc and SPECapc).
  3. Each project group shall maintain its own project group rules document, which shall apply in conjunction with this document. - SPECopc project rules - SPECapc project rules
  4. Where a project group's rule overrides a rule in this document, this will be explicitly indicated in that project group's rule document.
  5. Should a new project group be approved or an existing one dissolved, this document shall be updated accordingly.

Membership
1. Membership
  1. Membership in the SPEC/GPC and its project groups is open to any organization that has a direct and/or material interest in graphics or graphics-related application performance benchmarking.
  2. Membership in one or more SPEC/GPC project groups leads to membership of SPEC/GPC.
  3. Members are expected but not required to be active participants developing and improving the respective project group's benchmarks.
  4. Members are entitled to secure access to development code.
  5. Members are entitled to unlimited publication rights.
  6. New members become eligible for voting on the 2nd consecutive qualified meeting. The first qualified meeting may have been attended prior to becoming a member. Qualified meetings are defined in Section II.4.b.
  7. A member maintains voting rights by attending 1 out of the last 3 qualified meetings. A member loses their voting rights upon missing 3 consecutive qualified meetings.
  8. A member regains voting rights on attending a second consecutive qualified meeting.
2. Associate Status
  1. Associate status is available to non-profit organizations.
  2. All rights and rules of the respective project group, GPC and SPEC apply to Associates unless specifically stated otherwise.
  3. Associates are entitled to secure access to development code.
  4. Associates do not have voting rights.
3. Officers and Elections
  1. On an annual basis the project groups will elect from their eligible voting memberships the following officers:
    1. Chairperson
    2. Vice Chairperson
    3. Secretary-Treasurer
  2. The Chairperson's responsibilities are to
    1. conduct meetings,
    2. send out the agenda on time,
    3. conduct votes on time,
    4. deal with outside organizations such as the press,
    5. represent and respond on behalf of the group to external questions and queries,
    6. interact with the SPEC/GPC committee, and
    7. police the submission, review and appeal process.
  3. The Vice-Chairperson's responsibility is to do the chairperson's job when the chairperson is not available, or if the chairperson is subject to a conflict of interest.
  4. The Secretary-Treasurer's responsibilities are to:
    1. record minutes,
    2. maintain the rules document,
    3. keep a history of email,
    4. track finances and interact with the GPC and SPEC Board in that regard.
4. Meetings
  1. SPEC/GPC project groups have three types of meetings (not including ad-hoc working-group meetings)
    1. Regular quarterly face-to-face meetings
    2. Special face-to-face meetings for the full membership
    3. Conference-call meetings
  2. Meetings which qualify for attendance are limited to:
    1. face-to-face meetings scheduled at least one month in advance and
    2. conference calls scheduled at least two weeks in advance and which are explicitly indicated as qualified at least two weeks in advance.
5. Voting
  1. A quorum is defined as the presence at a qualified meeting of at least 66% of the set of members who hold voting rights. If too few voting members are present for a quorum, voting cannot take place.
  2. The electorate is defined as the set of members who hold voting rights and who are present at the time of a vote, provided that a quorum is satisfied.
6. Membership Dues and Billing
  1. Dues are assessed on the basis of membership of SPEC/GPC's project groups.
  2. Dues for the SPEC/GPC project groups will be set annually by the SPEC Board of Directors with input from the SPEC/GPC. Once set, the dues amount will be recorded in the SPEC minutes and communicated to the SPEC/GPC by the SPEC office.
  3. Dues payment, purchase order or letter of intent to pay for a given calendar year must be received at the SPEC office by March 1st of that year. Alternatively, a letter of intent to join the respective project group must be received by the SPEC office by March 1st of that year with a subsequent dues payment by May 1st of that year. Failure to meet these deadlines will result in loss of membership and voting rights. Membership will be reinstated when full payment is received at the SPEC office. Voting rights will be reinstated according to the attendance rules in section II.1.g and II.1.h.
7. Non-Member Publication
  1. The SPEC/GPC project groups will accept submissions from non-members for review and publication on the SPEC public website.
  2. Non-member submissions must follow the same rules and procedures as member submissions.
  3. Non-members are not eligible to participate in reviewing results.
  4. Non-members will be charged for their submissions according to an approved fee structure. Any change in hardware or software constitutes a new configuration.
  5. On an annual basis the SPEC/GPC will establish the pricing and periods for non-member publication. These will be recorded in the SPEC/GPC minutes and published on the GPC web-site.
  6. A configuration will be published on-line for six months, unless the submitter notifies the publisher that it should be removed.
  7. After six months, the configuration will be removed automatically, unless the submitter notifies the publisher that it should remain on-line.
  8. There are no additional non-member fees for extending on-line publication beyond six months.
  9. Each SPEC/GPC project group may remove published results from its web pages due to benchmark revision. In this case, the submitter will be given notice by the project group and may, at no charge, resubmit the identical configuration for the revised benchmark.
Benchmarks
1. Each project group shall document all benchmark-related rules in its respective project group rules document.
Submission and Review Rules
1. Submission Preparation Rules
  1. The rules for the submission and review cycle to be used are those approved by the respective project group's committee prior to the submission deadline. The approved rules must be posted to the respective project group's web-site by the first publication date for the benchmark.
  2. Version compliance: The benchmark and (where applicable) application versions to be used are those approved by the respective project group's committee prior to the submission deadline. The approved benchmark (and application) versions must be posted to the respective project group's web-site by the first publication date for the benchmark.
  3. All benchmark sources for a submission must be the same as that approved by the respective project group's committee prior to the submission deadline. The approved benchmark sources must be posted to the respective project group's web-site by the first publication date for the benchmark.
2. Submission Content Rules
  1. The information supplied must reflect the system as tested.
  2. Configuration description: All fields in a submission's results file must be supplied, unless the field names are marked "opt.", indicating an optional field.
  3. Submitters must specify a date for 'General Availability' that is accurate for the entire system - hardware, software, O/S, drivers, etc.
  4. The "Comments" area of the results page must describe how the system may be acquired.
  5. Date fields must always contain a valid date. "Now" is not valid in a date field.
  6. Price includes system and monitor as tested.
  7. Price may be submitted with an alterate currency from the US dollar. The submission will be sorted separately on the summary pages for Price and Price/Performance.
  8. The submitter is required to declare sufficient information to reproduce the performance claimed. This includes but is not limited to:
    1. non-default environment variables,
    2. non-default registry variables,
    3. hints,
    4. compiler name and version,
    5. compiler command line,
    6. changes to the standard makefiles.
  9. Any information required to be reported such as non-default environment variables, registry variables or hints, that does not have a predefined field must be documented in the "Comments" area of the results page.
  10. Valid submissions must include screen captures if required by the benchmark.
  11. Results previously published for a system can be resubmitted. Resubmissions do not require the inclusion of screen capture images.
  12. Previously published results being re-submitted can only have price changes.
  13. Each member company must ensure that the upload file contains data for all the new configurations and existing published configurations they wish to continue publishing.
  14. Standardized CPU cache nomenclature is as follows:
    1. (D+I) designates a unified instruction and data cache
    2. (D/I) designates separate instruction and data caches
    3. A number followed by KB or MB can be used to describe the size of the cache.
    4. Caches dedicated to a processor are listed as per processor cache size.
    5. Caches shared by multiple processors are listed by total size
  15. Each component of the submitted configuration (including the graphics driver) shall be:
    1. uniquely identified,
    2. available to members of the respective project group, upon demand, by the submission deadline and for the duration of the review process,
    3. verifiably available to the public by the publication date, with continued availability at least for the life of the submission, with sufficient information in the comment field to enable users to directly obtain this component.
  16. Price must be valid for two submission periods from date of publication. Quantity 1 pricing must be used.
  17. Subsequent to publication, any change to or replacement of elements for a submitted configuration must not result in more than a 5% performance degradation in the submitted benchmark results. Upon demonstration of such a degradation, the submitted results for this configuration will be removed from the SPEC public website.
  18. On or before the date of publication, the submitted configuration shall be available for purchase by the public, for the specified price or less, with a firm delivery date of 60 days or less. Submissions will be categorized as either “Single Supplier” or “Parts Built”, where “Single Supplier” is defined as a configuration where all hardware and drivers are sold and supported by a single supplier. “Supported” is defined as providing hardware, drivers and associated technical support, and that the drivers are available from the system supplier. “Parts Built” is defined as a configuration built and supported by multiple suppliers.
  19. “Parts built” system pricing must include enough detail to reproduce all aspects of the submission, including performance and price, and include all hardware and O/S costs necessary to run benchmark.
3. Submission Process Rules
  1. Each benchmark is considered a separate submission.
  2. Submissions of each benchmark's results (e.g. Maya6.5™, Solidworks 2005™, SPECviewperf, etc.) must be in separate tar/zip files.
  3. A submitter of benchmark results must upload his or her submission to the proper server location by the submission deadline date and time. The submitter must not create any new directories on the server when uploading the submission.
  4. The submitter must notify SPEC Office after a submission is uploaded to the server prior to the submission deadline with contact information for questions about the submission.
  5. The submitter must contact the SPEC office if they have attempted to upload their submission and were not successful.
  6. The SPEC office will not disclose who has submitted results until the submission deadline has passed.
  7. Submissions will not be accepted after the submission deadline.
  8. The upload directory will be set to write-only until the submission deadline has passed. Then it is set to read-write (not modify) after the submission deadline.
  9. If a submitter is notified that their submission format is incorrect, they must re-send their submission in proper format within 3 business days of notification.
4. Review Period Rules
  1. SPEC/GPC project group members shall keep all submitted results confidential to the respective project group until those results appear on the public SPEC web site. The exception to this rule is that members are free to make their own submitted results public at any time.
  2. SPEC Office assigns reviewers to submitters.
  3. Members who wish not to review the submission of other specific members due to conflict of interest must submit that list to the SPEC office prior to the submission deadline. The SPEC office will hold the list in confidence from other members.
  4. The various SPECapc and SPECopc pools of eligible reviewers will be independent of each other. The SPEC office will send the list of contact information for the submissions under review.
  5. All members will have access to all benchmark submissions once the review period begins.
  6. The review period shall be 5 calendar days.
  7. Submissions cannot be withdrawn during the review period without cause and without prior approval of the primary reviewer. A submitter who is granted permission to withdraw a submission must inform the committee by email of the reason for withdrawal.
  8. If a primary reviewer has a question with a submission they must pose the question to the submitter first. The primary reviewer may also pose questions to the respective project group's officers or SPEC/GPC Chair for clarification of rules if needed.
  9. Any reviewer who has one or more questions relating to a submission must:
    1. Pose the question(s) to the submitter and cc the primary reviewer, OR
    2. Pose the question(s) to the primary reviewer. The primary reviewer must then pose the question(s) to the submitter, OR
    3. Pose the question(s) to an officer of the respective project group. The officer must then pose the question(s) to the submitter and cc the primary reviewer
  10. With permission of the primary reviewer, as communicated through the respective project group's email alias, the submitter can request that their submission be rejected on stated technical grounds.
  11. With permission of the primary reviewer, as communicated through the respective project group's email alias, a submitter may resubmit a submission to resolve issues found during the review process. The submitter must notify the respective project group's mailing list with the date and version of the resubmitted file(s).
  12. The submitter must provide the primary reviewer access to the system under test at the submitter's facilities if requested by the reviewer during the review period. The reviewer must state prior to the visit what part of the submission is going to be verified. Travel expenses are the responsibility of the reviewer.
  13. Previously published results being re-submitted can only be reviewed for consistency with the previous submission, and price changes.
  14. If the reviewer disputes the stated pricing, the submitter must provide documentation that the system can be purchased for the price quoted.
  15. By the end of the review period, the primary reviewer of a submission must designate the status of the submission one of: “accepted without comment”, "accepted with comment", “pending with comment”, or “rejected with comment”. The submitter may appeal a rejection as described in "Review Appeal Rules" below.
  16. Any comments for rejection of a submission received after the end of the review period will not delay publication of the submission.
  17. A submission designated “pending with comment” will not go public and will remain pending until the submitter addresses all comments. Once the comments are addressed the web master will post to the public site. Any member who feels comments are not satisfactorily addressed may challenge the submission according to Section IV.6 for challenging accepted results.
5. Review Appeal Rules
  1. The appeal period shall have the same duration as one submission cycle, and shall immediately follow the review period.
  2. Any submitter of a rejected submission can make their case to the respective project group's email alias during the appeal period.
  3. At the end of the appeal period, if there is no resolution, the project group Chair shall call a vote to accept or reject the submission.
  4. The project group electorate votes on acceptance or rejection of an appealed submission. A simple majority is required to approve or reject the appeal. In case of a tie the submission is rejected.
6. Challenging Accepted Results
  1. Any member may challenge accepted results at any time. This includes:
    1. archived results,
    2. currently published results, and
    3. resubmitted results not subject to the regular submission review process.
  2. The burden of proof that the result should be modified is on the member who is challenging the result.
  3. The challenge must be ratified by a majority vote of the project group's electorate.
  4. The project group Chair will call a special review cycle for a resubmission in the event that there is a ratified challenge to currently published results.
  5. A ratified challenge to archived results can only result in annotation, not removal or modification. The annotation will be determined by the majority of the electorate. It is the responsibility of the challenger to verify that the results have been annotated correctly on the public website within two working days from the ratification of the challenge.

Publication Rules

Official Publication
1. Benchmark results for publication by the SPECopc or the SPECapc must adhere to Articles concerning "Overview", "Benchmark Run Rules" and "Submission and Review Rules" as presented in this document AND the respective project group's rules document.
Unofficial Publication
1. Benchmark results for publication elsewhere (e.g. industry journals, vendor web sites, analyst reports) must adhere to Articles concerning "Overview" and "Benchmark Run Rules" as presented in this document AND the respective project group's rules document.
2. The respective project group or any member thereof reserve the right to request and receive evidence that the published results have been achieved in accordance with the rules and that published information is accurate.

SPECopc metrics may be estimated. Metrics shall not be estimated for configurations that are capable of running the benchmark. All estimated metrics must be clearly identified as estimated. Licensees are encouraged to publish actual SPECopc metrics as soon as possible. Proper trademark usage for estimated results would be in the following forms:

SPECviewperf Awadvs-04 estimated score of 30 fps

SPECviewperf	Awadvs-04	30	est.
SPECviewperf	Light-04	122	est.

Adoption

V1.00 adopted on 01/25/2006
v1.01 updated on 02/09/2006 to align wording with SPEC policy

Standard Performance Evaluation Corporation