To get the full lowdown on 3DMark03, Atomic went to the source and ask Tero Sarkkinen (T.S), Executive Vice President of Sales and Marketing, Futuremark Corporation, President of Futuremark’s US operation and Patric Ojala (P.O), Senior Manager, Benchmark Development of Futuremark Corporation for a full rundown.
- How long has 3DMark®03 been in development and how does this compare with the development time of previous versions?
T.S.We have developed 3DMark®03 full-time for about one year (we launched 3DMark®2001 SE in January 2002). However, we started initial research and development already long before that, so all in all the development time has been fairly long – longer than for the development time for 3DMark®2001 for example.
- Is the engine still using MAX-FX? How much left remains in this engine and how easy is it to efficiently add DX9 features to it?
T.S.This is an important change what we have done in 3DMark®03. Our previous products have all used MAX-FX as the 3D engine. With the introduction of vertex and pixel shaders in DirectX 8, massive middleware became needless in 3D games for hardware with full support for DX8 or later. Therefore 3DMark®03 is basically directly written onto DirectX. We used a small library of helper functions in order to avoid unnecessary rewriting for each test.
All vertex processing is handled by vertex shaders and all pixel processing (except for game test 1 and the fill rate tests) is done with pixel shaders. This makes 3DMark®03 a very forward-looking benchmark, and it might scale somewhat differently than many games available today, which still mainly rely on fixed function vertex and pixel processing. Then again, the mission of 3DMark® is to give an insight into how a given hardware performs with the next generation of 3D games. This is also a key difference between 3DMark® and a game used as a benchmark. 3DMark® is able to tell you the objective performance of your hardware, while a specific game will tell how your hardware performs on that one specific game.
- In your opinion, what are the most exciting features of 3DMark®03?
T.S.Well, aside from the tests being an incredible eye-candy, it has to be the totally revamped Pro ORB, i.e. Professional Online Result Browser. This is enabled in the Pro version only, and allows you do even more extensive searches and compare against multiple other results at the same time. Since we have an active user base of over one million, the Pro ORB is just one unbelievably cool tool and resource.
As far as individual tests go, I am particularly excited about the totally new 3D sound performance tests, CPU tests, and the image quality tools. We enabled image quality tests to be more accurate and reliable by letting the user to run any scene with so-called frame-based rendering, i.e. dictating the test to draw all screens that would appear if FPS was kept at the specified constant level. This enables you to produce exactly the same screens in different systems in different benchmark runs. Then, you can compare the images and spot possible differences.
- How closely do you work with both hardware and software developers during the development of a new 3DMark benchmark? Do trends in game development influence the features used in the benchmark?
T.S.The very core of our approach to benchmark development is to work in close cooperation with industry's key players. We have found it to be the only reliable way to ensure that we have the latest knowledge about where the industry is heading. This information is crucial in designing the workloads and tests. We spend several man-years in cooperating with the companies in our BETA program as we produce our software.
The benchmark development process starts with us first conducting an extensive study using internal and external sources. This is where input from e.g. game developers come into the picture. It enables us to sketch a general roadmap as regards what game developers believe will be important and relevant new technologies. We then proceed and start working with our BETA program companies and thus are able to get an industry-wide view of the most important technologies from hardware developers' point of view.
From these ingredients we then make basic decisions for the general direction of the benchmark. The end result of this phase is a specification for the benchmark, which has been reviewed by the BETA members.
The rest is then implementing the specification and working with the BETA program companies as we send out milestones and collect feedback from them. These phases are called prototypes, alphas, betas, release candidates and finally the gold version of the software.
We use this process, since it has proven to result in a neutral and balanced benchmark, which stays relevant during a substantially long period of time. This approach also enables all participating companies to have a complete visibility on our development and they are able to give feedback on various stages.
- When do you think we will see the widespread use of DirectX 9 features in games?
T.S.I am confident that DirectX 9 features adoption in games will be faster than that of DirectX 8 features. This is mainly due to the fact that many ordinary folks will just be amazed about how big a difference the new technology makes on their screens.
However, game publishers obviously need to carefully make decisions on where to set games' requirements bar. Hence, the more low price point DX9 compatible hardware there is, the faster the new features will be adopted in games.
Speaking of which, we have been able to accelerate this process via couple of ways. In addition to the 3DMark® demonstrating what is possible, we have successfully provided data of the installed hardware base from our massive databases. For instance, our data proved to a game developer that the true installed base their game is more high-end than they had thought. They then made appropriate decisions and chose more demanding hardware requirements for their next games.
- What lifespan will 3DMark®03 have? Especially in the light of the fact that Microsoft are extending the duration of each Direct X version, and that it is unlikely we will see a new version of Direct X until the launch of Longhorn.
T.S.Well, we still see 3DMark®2000 being used quite a bit :) Quite frankly, we expect the lifespan to be several years. You must remember that 3DMark® is the single one most demanding benchmark for a PC, and only the real high-end gaming systems will be able to score high in the beginning. It will take several years for the benchmark to become obsolete to the mainstream PCs.
- 3DMark® has the strange honour of being a benchmark with a loyal fanbase. How do things like competitive benchmarking for ORB honours influence the way Futuremark works?
T.S.The ORB has been just a tremendous service that has let our users to show off what their systems really can do. It is grown to become the most popular section of our web site and it is largely due to the constant flow of feedback from the users and our talented web team that has implemented many features based on that feedback.
The ORB contains over 5 million real-life benchmark results, so it is a goldmine for anyone interested in really digging deeper into PC performance. We know users who keep their whole PC 'performance history' over there: There, they can easily see what a difference e.g. a driver update or graphics card upgrade did for their PC's performance and so forth.
We have realized that we have built a service that is really useful and valuable to our user base and we want to be able to keep on developing it further. After all, our mission is to help consumers to make informed purchasing decisions.So big thanks to everyone that have submitted their scores to ORB, keep'em coming!!!
- What's your view on the new HLSLs: CG, DX9 and
Issue: 137 | June, 2012