Building a Better (Linux) GPU Benchmark

Introduction

For those who follow our Linux reviews, we have made a lot of headway in the last two months. Our benchmarking has improved, our graph engine is top notch and we are working closely with all the major manufacturers to bring a definitive resource for Linux hardware to our readers. Today, we want to introduce everyone to our GPU Linux benchmarks and how we will run them in the future. This isn't a comparative analysis yet, but we won't keep you waiting long for that.

The adherent flaw with any benchmark is that you, the reader, only receives a sampling of data - engineers and statisticians alike call this "data compression". When we sample data from a timedemo and format it into an average frames per second, we lose all sort of valuable data, such as what the lowest frames per second was, what the highest was, when the largest dip in FPS had occured, what the image looked like, and the list goes on. There have been a few attempts to convey more than just an average FPS in video benchmarks, most notably with FRAPS. FRAPS does not entirely address the issue of reproducibility and FRAPS runs on Windows only.

Fortunately, we have been graced with some very talented programmers who worked with us to build a benchmarking utility similar to FRAPS (on Linux) that we may eventually port over to Windows as well. Consider this to be our experiment in advancing our benchmarking methods while using Linux as our guinea pig. Eventually, we anticipate releasing the benchmark complete with source to the public.

Here is how our utility works, as explained by the lead developer, Wiktor Kopec.

"The program computes frames per second for an application that uses OpenGL or SDL. It also takes screenshots periodically, and creates an overlay to display the current FPS/time.

"This is accomplished by defining a custom SwapBuffers function. For executables that are linked to GL at compile time, the LD_PRELOAD environment variable is used to invoke the custom SwapBuffers function. For executables that use run-time linking - which seems to be the case for most games - a copy of the binary is made, and all references to libGL and the original glXSwapBuffers function are replaced by references to our library and the custom SwapBuffers function. A similar procedure is done for SDL. We can then do all calculations on the frame buffer or simply dump the frame at will."

You can read more about SDL and OpenGL. SDL is a "newer" library bundled with most recent Linux games (Medal of Honor: AA, Unreal Tournament 2004). In many ways, SDL behaves very similarly to DirectX for Linux, but utilizes OpenGL for 3D acceleration.