First SPECmedia Release to Use Kernels to Simulate Real Application

Multimedia has always been a target that is a bit out of focus and moving too fast for benchmark developers. A project group under the umbrella of the non-profit Standard Performance Evaluation Corp. (SPEC) has taken on the challenge, however, and plans its first benchmark for early 2000.

SPECmedia, formerly called the Multimedia Benchmark Committee (MBC), was formed to develop a wide range of standardized benchmark suites that will help vendors and users compare and improve multimedia system performance. The group operates under SPEC's Graphics Performance Characterization (GPC) Group.

SPECmedia's first benchmark will measure multimedia performance through a set of kernel functions that are essential in video, audio and speech codes. The project group will focus on temporal, spatial and statistical domains of video compression. For audio and speech cases, transforms and analysis/synthesis routines are included. Within each domain, several representative functions are being selected. The execution time from each category will be used to form a composite score for the benchmark.

SPECmedia has licensed software from MIT to be used as part of its quality measurement system. The video fidelity metric under development by SPECmedia compares a reference sequence with a sequence decoded by the system under test. It accounts for display device calibration, color appearance, and spatio-temporal sensitivity. The process starts when both video signals are gamma-corrected and mapped into a perceptual color space. Then, the difference between the two signals is computed. The next step is to assess the visibility of the difference signal. The signal is split into two paths, one called excitatory and the other inhibitory. In each path, the signal is filtered by a temporal and spatial filter. Each filter corresponds to a sensitivity mechanism. The difference between the two paths -- the visible signal -- is computed and pooled spatially and temporally to compute a distortion figure for the sequence.

Content for testing is being collected from actual video broadcast material. Three hours of content has been licensed to SPECmedia from CNN to use for testing sequences. Encoding is being done on a volunteer basis by SPECmedia member companies.

The term multimedia -- always ill-defined within the marketplace -- hints at the complex challenges that face SPECmedia. Multimedia merges video, graphics, processing and sound, making it much more challenging for benchmark developers than other technology areas.

Adding to the complexity is the SPECmedia approach to benchmarking. The group is dedicated to developing a systems-level, multi-platform benchmark, one that will allow a wide range of participants. It will also be designed to accommodate vendors with different quality. Within this realm, functionality might be measured by "good enough for the money," "better-quality picture without dropping frames," "how many frames a machine can pump through without dropping a frame," or any number of other measurements being decided upon by SPECmedia members.

"We are doing everything possible to achieve consensus so that this benchmark will be accepted throughout the industry," says Ahmad Zandi of Sun Microsystems, the SPECmedia chair. "This and future SPECmedia benchmarks will measure both performance and quality across a wide range of platform and architectures. These benchmarks will benefit the industry by being repeatable, scalable, open and fair to everyone."

Faced with its sizable challenge, SPECmedia has decided that its first benchmark will focus on MPEG-2 video only. The benchmark will concentrate primarily on Main Profile at Main Level (MP@ML) attributes that support up to 15 Mbps and will support Low profiles as well. There will be a maximum of 720 horizontal pixels for MP@ML. In the computer space, SPECmedia is examining 256, 65.5K and 16-million colors, and MPEG-1 and MPEG-2 resolutions for NTSC and PAL. In the consumer space, the group is looking at 12 bits/pixel (4:2:0) color at the same resolutions as those under consideration for computers.

SPECmedia membership information is available by e-mailing [email protected]. A SPECmedia FAQs document is available on this web site.