"Nearly 18 months ago Futuremark released 3DMark03 - the latest in the line of 3D performance benchmarks and their first to make use of DirectX9 technologies. While, internally, they probably realised that not all was happy in in the Beta members (the group of members that steer the benchmark - which includes the IHV's who's products will be tested by the benchmark) camp, its doubtful that anyone would have expected the rough ride that was to ensue over disagreements with what the benchmark tests and overzealous optimisations coded into drivers.

After some time all parties patched up the majority of their differences and the work on the next release of 3DMark was allowed to progress relatively unfettered. Over the past few months Futuremark have been busy putting together the 3DMark04 and they are giving us some small insights as to what we may see when the benchmark is ultimately released.

We've fired a bunch of questions at Futuremark and Nicklas Renqvist, Patric Ojala, and Tero Sarkkinen gives us some answers...

3DMark is billed as a gamers benchmark, and is designed to offer a look into the future of the types of 3D hardware feature use and applications of those features. With 3DMark2001 and the use of the Max Payne engine it was easy for people to see a correlation between the benchmark and an actual application, however with 3DMark03 this was obfuscated without any direct links to engines. How do you decided what features to use and how they should be applied - is this all done in-house, or through canvassing of other developers? What specific research has been taken for 3DMark04?

Nick: Before we start to develop a new version of 3DMark, we have done a lot of research in advance. Our research consists of talking to other developers, IHVs (our Benchmark Development Program members), studying available tech-papers, looking & testing new and innovative tech-demos and so on. All this combined gives us a pretty good picture of where the real-time 3D is heading, and what features are a “must” in the next generation engines. This is exactly we did in 3DMark03. Our research told that the stencil-shadow technique was going to be used in future games, so we implemented that feature into 3DMark03. So far, there are only a handful of games using stencil-shadows, but there will be more and big titles coming out using about the same shadow technique.

When we are satisfied with the new ideas we would like to use, we start planning the game-tests and which features we could use in the various tests. Some features never make it into the engine, but they aren’t forgotten. If they are good enough, they will find their way into the next version of the benchmark. There is only so much we can squeeze into the benchmark while keeping an acceptable development time.

The research for the next 3DMark has been in the works ever since 3DMark03 was released in February 2003. We are always open for discussion with game developers on future innovations and what techniques will be used in next generation game-engines. Of course our BDP members play a very big role when we plan our next 3DMark versions, as do various tech-papers and forward looking tech-demos. BDP members, for example, share their technology roadmaps with us and that helps us determine which types of tests are the most valid during the benchmark’s life.

Patric: Commenting on the ‘gamers benchmark’ slogan is really a marketing issue, and maybe Tero could add something about that. From maybe a less PR perspective, I think the slogan still fits very nicely. We do indeed aim to measure game performance, but our focus is on the 3D performance of games. This is an informed decision, since graphics performance tends to increase faster than CPU performance, and all benchmarks in time become system (meaning usually CPU) bound anyway. Game benchmarks are in general heavily system bound and/or present current or past generation graphics technology. So there clearly is a gap in the range of available benchmarks. No other benchmark measures 3D performance of future games, and that’s where 3DMark is aimed.

As for the engine choice, MAX-FX was an excellent engine as long as 3DMark just measured fixed function multi-texturing performance (’99, 2000 and mostly 2001). The introduction of shader technology made such massive middleware blocks unneeded from 3D performance measurement point of view. The middleware blocks are needed only for legacy hardware code paths. Since 3DMark is only forward looking without legacy code paths, we had no need for MAX-FX type technology anymore in 3DMark03. Game test 1 indeed still does multi-texturing, but only two code paths were needed because of the content – dual texturing and quad texturing paths. So there was no reason to add a whole engine just to get two multi-texturing code paths.

The new 3DMark engine is more of a ‘game engine’ than the one in 3DMark03. That engine was a first generation shader engine for a smaller variety of shaders. It just loaded pre-compiled shaders in shader language and the artwork. The new 3DMark engine is a next generation shader engine, with a wide variety of shaders dynamically built and runtime compiled. This is how we believe future game engines will work, and some developers have already confirmed it.

Tero: The Gamers’ Benchmark term has been associated with 3DMark from the very beginning and also continues to be so. The reason has not changed over the time: 3DMark is a tool we make for gamers with their needs on our mind. It helps gamers to objectively assess today how today’s technology will perform in games of tomorrow, i.e. to get important data points of how ‘future proof’ their investment is.

3DMark03 was also advertised as a DirectX9 benchmark, and yet only one of the four game tests required DirectX9, with two using DirectX8.1 and one DirectX7 features. Will 3DMark04 remove the legacy tests and target DirectX9 only? If it is DirectX9 only will you be requiring Shader Model 3.0 in any of the game tests, given the previous sentiment has been that you'll require the newer features when at least two vendors are supporting it, or have signalled intent to within close proximity of release?

Nick: The next 3DMark will require a fully PS2.0 capable graphics card for all game tests. None of the game tests will use shaders below version 2.0, and the benchmark is really mainly targeted at the second generation of DX9 compliant cards. Of course users with older/first generation DX9 hardware can run the benchmark and get reliable results, but the tests loads aren’t really designed for that generation. I want to remind all readers that 3DMark is a benchmark, and is NOT optimized in the same fashion as games are. There are no lighter code paths for value and legacy hardware, all systems produce the same rendering. Also, 15 fps for example is a too low frame rate for enjoyable game play, but it is quite enough for producing comparable 3D performance results. 3DMark thereby measures quite well also below playable frame rates. When game developers are developing a game, their main focus (performance-wise) is to offer enjoyable game play for all gamers. Our focus is to stress the hardware, and not to make the benchmark user necessarily experience a smooth run. Also a point to remember is that I doubt any game so far released, or to be released this year, uses similar shaders & amount of lights & shadows to the same extent as the next 3DMark does. The amount of polygons on screen and use of dynamic lights & shadows is on a totally different level than what gamers have seen so far.

None of the tests in the next 3DMark will actually require SM3.0, but if the user has a graphics card capable of doing SM3.0, it will be used by default. There is actually a new option in the settings, from where the user can choose which shader model will be used (2_0, 2_a, 2_b or 3_0). I think that widens the use of the next 3DMark even more as you can for example compare your graphics cards SM2.0 and SM3.0 performance.

We cannot reveal any features of upcoming hardware since that information is all under NDA.

It been mentioned that in order to support multiple shader models (PS_2_0, PS_2_a, PS_2_b, PS_3_0) you will be utilizing longer shaders in some of the game tests, but then compiling these to the optimal HLSL shader profile available for the particular board that its operating on. Is this the case and if so how is this achieved? If this is the case, presumably all the shaders will be written via HLSL rather than hand written as in the 3DMark03?

Nick: The next 3DMark will use a completely new 3D engine, which dynamically builds HLSL shaders. The shaders are dynamically built and runtime compiled using the most optimal target for the hardware in use. We strongly believe that this is a structure future game-engines will be using because of its flexibility. It goes without saying that the compilations produce the exact same rendering, no matter which target. This is an absolute must for enabling objective performance comparisons.

Patric: Hand written shaders have their benefits. They can be highly optimized for some certain hardware. Then again, a game using shaders for all or most materials will probably have quite a bunch of them. Manually typing hundreds of different shaders optimized for different hardware makes no sense for a developer. Therefore the HLSL model is a great thing. Write a shader that is easy to understand and modify, which is then automatically optimized for the installed hardware. Or better yet, use some dynamic shader generation implementation and create the needed huge amount of shaders even more easily.

In 3DMark03 Games tests 2 (Battle for Proxycon) and 3 (Trolls Lair) were designed to mimic the Doom3 rendering model in DirectX, with an initial Z-Only pass and then a shadow pass for each dynamic light. Will any of the game tests be following a similar rendering model as this, or is it "all change"?

Nick: The next 3DMark is completely different from what 3DMark03 is. The engine is new, shaders are new (dynamically runtime built HLSL shaders), the shadow technique is new. It is an all new 3DMark version!

Patric: I think we worked enough with stencil shadows last time. If you want to measure that, why not use 3DMark03? The early footage of Doom3 did indeed inspire us, but our goal was really not to try to ‘measure Doom3 performance’ or ‘mimic Doom3 in DX’. Also, stencil shadows are a better choice in darker indoor scenes with less space and less edges throwing shadow volumes. Our new shadow model is based on perspective shadow maps (PSM). There are no shadow volumes adding TONS of fill rate and vertex load like in stencil shadows, but it is still a global solution with self-shadowing. Who knows if some even better dynamic shadow implementation is invented for the 3DMark version after this next one.

In these two game tests the shadowing obviously makes heavy use of stencil shadows, which aren't necessarily the best for producing soft shadowing techniques. Are you looking at any alternative shadowing methods in the next 3DMark?

Patric: These days projected shadow maps seem like the best choice if you want soft dynamic shadows, and I think most games out today with soft shadows use that. There are extensions to perspective shadow maps offering soft shadows, but those tend to be very heavy on the hardware and more suitable for small tech demos than full scale games. For example the Smoothie trick requires the identification of the edges throwing the shadows, which was one of the burdens of stencil shadows. For soft shadows we would still probably choose projected shadow maps, but we did that already in 3DMark2001. Then again, back in the ‘2001 days sharp edged projected shadows were quite enough for the hardware.

3DMark03 obviously caused a lot of contention in some quarters, due to the performance with the PS2.0 tests. Part of the reason for this was that you chose to utilize full precision float shaders and NVIDIA's hardware wasn't optimal in these conditions, despite it being the DirectX default. Would you like to expand on the reasons for this choice? Will 3DMark04 be utilising a mix of full and partial precision shaders in the game tests, dependant on whether the quality allows it? If so, will you be offering options to test partial, mixed and full precision modes such that performances and qualities can be compared?

Patric: Full precision is indeed the DX default and the very few places we used float values in 3DMark03 full precision was needed to keep the image quality. Then again, most materials in ’03 used 1.x shaders that are all fixed point. Now all shaders are SM 2 or 3 and many materials look just the same in full and partial precision. We therefore have a precision mix as default. Materials that get rendered the same in half precision are allowed to use that if the hardware supports it. Then again, if some material gets reduced image quality in half precision, full precision is forced. We plan on adding a switch to force full precision in all shaders for comparison, but that’s only an optional setting.

DirectX9 brought various forms of branching in the shader pipeline - with VS2.0's basic static constant branching, VS2.x's dynamic branching, PS2.x's conditional write masks and Shader Model 3.0's full dynamic branching. 3DMark03 made no use of any branching code, either for Vertex or Pixel Shaders - will this change in 3DMark04 at all?

Patric: We are currently testing branching, so it’s a bit early to say how much of it will be included in the final product.

One feature that has seen a lot of publicity over the past year is High Dynamic Range rendering. With graphics hardware that have good support for float textures and PS2.0, HDR effects can be achieved reasonably effectively, and more recently GeForce 6800 has introduced floating point filtering and blending which should enable a simpler method to high quality HDR. What are your thoughts on these techniques and will you be featuring HDR in the next benchmark?

Patric: HDR is a great thing! We would like to use HDR in all game tests, but the lack of fully supporting hardware prevents us from doing that in this next 3DMark. All (or at least most) DX9 hardware have floating point support, but in order to really utilize HDR, you also need full support for blending and filtering of float textures. Those are still rarely supported features.

Although Normal mapping, for generation models that appear higher detailed than they actually are, has been with us for some time, the technique appears to be only recently becoming more ubiquitous. ATI has introduced 3Dc normal map compression which should allow for high detail normal maps (hence higher detail appears on the rendered output) at a lower performance overhead - there is also a fallback method that requires a tweak to the DXT5 compressor. Will you be featuring tests with normal map compression?

Patric: DXT5 is commonly supported by today’s DX9 hardware. We will therefore most likely be using DXT5 compression for normal maps. This is still work in progress, so we’ll see what we end up doing in the final product. 3Dc is still ATI specific AFAIK, so that is not so interesting for us regarding 3DMark.

PCI Express is a new technology that has arisen over the past few months, and while not being a core 3D feature it does affect graphics boards by giving a much larger downstream and upstream bandwidth for the graphics interface to the host. Will you be taking advantage of this increased bandwidth at all? Will there even be specific tests to see the transfer rates afforded by different implementations?

Patric: This is also still work in progress. PCI-Express is indeed one of the latest hypes in PC hardware, so it would be nice to utilize that somehow.

In previous version of 3DMark we saw numerous feature tests that had lots of specific feature tests, such as EMBM or Dot3, or Pixel Shaders, however this was cut back in 3DMark03. Will we see more feature tests in the next version?

Patric: 3DMark is a very ambitious project for a small company like ours. There is so much that could yet be added when it’s time to launch. We have a mile long list on what cool feature test ideas, but we usually have so little time to actually implement some of them. By far most our efforts go into making the game tests. They are the most important part of 3DMark and the tests that offer the best measurements. When there is time, we try to add as many feature tests as possible, but once the game tests are done, there usually is way too little time left.

Nick: We dropped the sound tests for the next 3DMark simply because there isn’t such a big demand for a new sound test right now. If people want to benchmark their soundcards we suggest to use the sound tests found in 3DMark03. Of course if the demand for new sound tests grows substantially, we might add sound tests to the next 3DMark via a patch or an update, but the benchmark won’t ship with sound tests.

Due to some of the issues that arose with 3DMark03 and some disagreements with the spirit of your benchmarking guidelines, you introduced a set of optimization guidelines and an approved driver scheme which you tested to validate whether the driver stuck to you guidelines. Will you optimization policy change at all with 3DMark04? Will you still be monitoring things in a similar fashion as you are at the moment? Is there even items you can code within the benchmark itself to assist in user spotting potential issues?

Nick: Our benchmarking guidelines and optimization guidelines will stay as they are. The same rules will apply for the next 3DMark as they do for 3DMark03. We will keep on monitoring new driver releases in order to ensure reliable results, but I am not yet sure if we will have some monitoring capabilities in the benchmark itself. As most of you know, it is very difficult (if not next to impossible) to create a detection mechanism for detecting all application specific optimizations, and any detection system built-in into 3DMark isn’t really a viable option because it would be outdated in an instant. Any such mechanism could get detected in the driver, re-routed and thus making it pointless. We will however continue to research different approaches on how to “fool” the drivers in case of any application specific optimizations, and if we come up with a working solution, it will be implemented into the benchmark.

The gap between the high end graphics boards and the low end appears to be growing ever bigger, and enthusiasts are using an ever greater range of image quality features by default on their applications. Are we likely to see a change in the default settings, such as higher resolutions with FSAA and/or Anisotropic Filtering, or does the differing IHV implementations make this difficult? Will we see a return to high end low detail benchmarks for the high end and low end boards?

Nick: As much as we would like to have Anisotropic filtering and Anti-Aliasing as default, it most probably won’t happen because it would jeopardize apples-to-apples comparison. The problem is that even though 3DMark would ask for Anti-Aliasing or Anisotropic filtering, the driver has the final say in what will be rendered. If the driver tells the application “yeah, AA is on”, it could still render the frame without any AA. That is a problem, and a problem that is very difficult to solve. This doesn’t mean that we wouldn’t trust what drivers do, but the risk is too high that some driver version would have broken AA / AF, and thus making the result not comparable. We would love to have 4xAA and 8xAF on as default, but until we come up with a 100% way to ensure that they really are on during the default run, they may remain in the benchmark only as options. The default resolution is still not quite decided, so that’s something I can’t comment on.

The next 3DMark will continue the same route as we took with 3DMark03. It won’t have any low & high detail scenes; only one single detail level.

Tests proved that 3DMark03's game tests, bar game test 1, proved to be fairly resilient to CPU performance, instead squarely putting the onus on the graphics performance. We've been talking to Tim Sweeney regarding the UnrealEngine3 and he's often remarked how graphics bound the engine is - do you believe there to be a trend in this direction, or do you feel that your onus only on graphics performance is not inline with games and it might need to be changed in the future? (Note: for Beyond3D's purposes we like the fact that it is remarkably graphics bound as we want to be testing the graphics card performance)

Nick: 3DMark mainly measures the graphics performance of the PC. For full system benchmarking we strongly suggest using PCMark04. In 3DMark2001 SE the game tests were actually GPU/VPU bound when it was released, but by time it has become more and more bottlenecked by the rest of the system (CPU, memory speeds etc). 3DMark03 has also come to a point where the CPU has more impact on the result. The latest generation of graphics cards are incredibly fast, and thus making the GPU/VPU wait for the CPU to catch up in many cases.

Patric: We always design a new 3DMark version to scale with the graphics hardware, so that it can efficiently compare the performance of the very latest and greatest graphics hardware at least at launch time. In time each 3DMark version becomes more or less CPU bound anyway, as graphics hardware performance grows faster than CPU performance. So if we launch an already CPU bound benchmark, it will become only more so over time and we never really would measure the graphics performance. Many game benchmarks out there were CPU bound already at launch, become even more so over time, and therefore offer less interesting graphics performance measurements.

Mr. Sweeney may be right. Shader technology allows the use of DX-wrapper type light graphics engines, like the one we were using in 3DMark03, and also the one we’ll use in the next 3DMark. These need less CPU power to get something drawn on screen, which leaves more system resources for other tasks. You can of course program your engine to use the CPU also for some rendering, but graphics hardware of today is so powerful that only few drawing optimizations on the CPU end up boosting the performance in a reasonable way. Most of these have some even more efficient implementation on the GPU/VPU. I guess CPU optimizations for rendering make more sense for code paths intended for legacy or value hardware.

With the numerous changes to the graphics pipeline we've seen over the past few years what elements have most interested you and what do you believe will have the most impact in games in the future? What changes would you like to see in the future?

Nick: Well I think the shaders are, and will be, the biggest step forward in real-time 3D. In other words, more programmability and flexibility. When the shaders were introduced in DirectX8.0, it was already clear that they will have a great impact on future games and vastly improve the visual quality of special effects (such as water etc) and lighting. When we released 3DMark2001 introducing the Nature game test, many were awed by the water effect which was something completely new in real-time 3D. The shader effect didn’t get unnoticed, and pretty soon games started to have similar looking water effects. When we designed 3DMark03 we took the water-effect one step further, and since the SM2.0 gives a much better result than SM1.x, the water shader we created for 3DMark03 was again awed by many. Shaders have made it possible to create very real-life looking materials / effects, and in the future it will only get better! I am very excited to see what other developers will be releasing within the next couple of years. The early screenshots and tech demos I have seen on future game-engines (for example UE3) are simply amazing! They are still work-in-progress, so I am sure that the final result will be even more dazzling!

Patric: I’d just like to add that shaders are not just for special effects like a water surface. This seems to be a common belief, since also games still use shaders mostly only for some special materials. I believe, and I’m sure the forward looking game developers agree, that shaders are here to replace the old fixed function vertex processing and multi-texturing. In the future all games will use shaders for all vertex and pixel processing. There may be fallbacks for pre-shader cards still doing multi-texturing etc, but the day will come when the first PC game is released that _requires_ powerful shader hardware and offers no fixed function fallback.

Finally, will there be any cows featured in the latest 3DMark?

Za informaciu spasibo saytu http://www.beyond3d.com/interviews/fm04/index.php?p=1

I tak obsugdaem....

Ты тока щас, очнулся? Мог бы и скрин выложить, он же там есть!

Quick XP эээ... вообщем выпей его, пока не поздно:) Мог бы ссылку дать

Drager
[Overclocker]

На дату-то гляньте...

офигеть...

[Overclocker]
Ты паходу его уже перепил

И охота ж в старых темах копаться