On the Search for the Truth — Games Benchmark Suite Instead of 3D Winbench 98
You may have wondered why I never used 3D Winbench 98 for my 3D accelerator reviews. I found out pretty soon after 3D Winbench 98 was released, that the results scored in this benchmark did not show any conformity with frame rates scored in any real world 3D game. It also seemed pretty strange to me that some graphics card manufacturers were keen on equipping me with new drivers right after 3D Winbench 98 was available, sometimes even with the words ‘these drivers will perform better in 3D Winbench 98’. Comments like that don’t really enforce your believe in that benchmark, it rather sounds pretty much as if these new drivers were simply improved for this particular benchmark. Now whilst suspecting something like that I couldn’t really be bothered doing any research about this unpleasant issue. Instead of this I started concentrating on finding new and good 3D games that included some decent frame counters. ‘Turok’ was one of the first, now the awesome game ‘Incoming’ and the latest playable demo version of the impressively looking ‘Forsaken’ as well as ‘Quake II’ complete my 3D benchmark suite for the time being. It was actually pretty annoying that I often had to explain to graphics card manufacturers why I wouldn’t use 3D Winbench, because some were particularly keen on me using this benchmark. Others suggested Final Reality and although I really liked that benchmark when it was new, I really went off it latest since I saw the amusing posting of Mercury, where a Voodoo Rush card scored higher than a Voodoo2 card in Final Reality. I’d rather call that ‘Final Irreality’.
I740 Faster than Voodoo2 or RIVA 128 ? Not really !
It would have stayed like this and I wouldn’t have cared about 3D Winbench 98, if there wouldn’t have been all that noise about the yesterday released Intel 740 2D/3D chip, coming up with rumors that said the i740 would be a particular fast 3D chip because it would score higher than nVidia’s RIVA in 3D Winbench 98. I was lucky enough that I just had received a Real3D Starfighter board, which is based on the i740 and so I could check out how well it performed straight away. Now the Starfighter isn’t performing bad at all, but in the above mentioned ‘games benchmark suite’ it’s clearly performing worse than Voodoo2 and RIVA. It puzzled me why that had to be the case, and I wondered if it was all hype that was being published all over the internet yesterday. I had to find out if I was just getting strange results, or if this was really the truth. Hence I ran 3D Winbench 98. I was shocked when I saw the result, it was higher than any result I had seen before and a quick test run of my Monster 3D II showed that Voodoo2 did only just reach that result as well. Now please understand this situation. Voodoo2 is scoring twice or even three times the frame rates of the i740 in any of the games I ran, but 3D Winbench 98 claims that their 3D performance is equal!!!
ATI Rage Pro Turbo Driver — Faster in 3D Winbench 98, Equal or Slower in 3D Games
This was the very moment when I decided that it’s about time to put an end to the spreading of hype by misleading 3D performance results achieved with 3D Winbench 98. I carried on testing other cards with 3D Winbench 98 and found out an even more ridiculous situation in case of the ATI Rage Pro Turbo drivers. Can you remember when last week ATI made a big PR effort about the so called Rage Pro Turbo, which is supposed to increase 3D performance by 40% only due to a revolutionizing new driver? Well, testing with this new drivers produced indeed a considerably higher result in 3D Winbench 98, however the frame rates of my games benchmark suite were either untouched or they even went down by 25% in the upcoming and great game ‘Incoming’ . What is this supposed to mean? The new driver scores better in 3D Winbench 98, but in real world applications it’s either the same or even worse? Is that what YOU expect from a driver that is announced big time?
The Industry is Aware of the Problem with 3D Winbench 98
I thought it was time to hear what the 3D chip manufacturers had to say to this issue. I contacted 3Dfx, nVidia and Real3D. Unfortunately it was already too late for contacting ATI, so didn’t have a chance of getting their statement about these miraculous Rage Pro Turbo driver. Real3D took my request very seriously. They are not used to the tough PC 3D card market yet and I guess that’s one of the reasons why they were most straight forward in helping to solve the puzzle with these high numbers the i740 is scoring in 3D Winbench 98. Although I may have thought that the i740 could have been optimized for 3D Winbench 98 in the beginning, I’m very sure that this is not the case. Real3D was taking this issue very seriously and they did not hesitate a second with a statement addressing the problem.
«Creating a level playing field to benchmark 3D graphics chips is a difficult and almost impossible assignment. Ziff Davis’ intent to measure the performance and quality of graphics chips with the 3D Winbench tests is the most thorough attempt to date. However, Real 3D does not feel the entire purchase decision should be based on the results of 3D Winbench. We are a strong believer in evaluating how a graphics board performs with actual applications such as games, and just as important, taking a measure of image quality. It is this combination of performance and quality that gives you a true measure of how good a graphics board is. One without the other doesn’t produce a compelling or exciting visual experience. With the StarFighter and Intel740, we believe what you get is a 2D/3D/video board offering the industry’s best combination of both — price/performance and image quality.»
Chris Stellwag, Real3D
I also send them a little interview, which reads like this:
1) You Real3D guys are pretty new in the PC graphics card business. What did Real3D do before?
It is true Real 3D is new to the mainstream PC graphics board business, but we are certainly not new to the 3D graphics business. In fact, Real 3D and its heritage companies practically invented 3D graphics and computer image generation (that goes back to General Electric Aerospace, Martin Marietta, and then Lockheed Martin). We started with 3D graphics back in the 1960’s with the Apollo space program when we designed Lunar Docking Simulators. In addition to our military heritage, you will also know Real 3D as the company who has provided Sega with arcade chips and boards since 1993, for both Sega Model 2 and Model 3 hardware. We continue to be Sega’s arcade hardware provider and have shipped more than 200,000 systems since 1993. In mid-1996, Real 3D started co-developing a mainstream PC graphics chip, which was announced yesterday as the Intel740. Real 3D also makes high-end image generation systems (our PRO-1000 image generators) and a 3D scanning device called RealScan 3D.
2) How close are you working with Intel? Are you developing drivers together?
We work very closely with Intel as a development partner on both the chip design and the drivers. We co-developed the reference drivers together and these became part of Intel’s baseline drivers several months ago. Real 3D has continued to optimize and enhance the reference drivers for the StarFighter.
3) Was the i740 completely developed by Real3D or was it a joint venture between you and Intel?
The 3D core of the Intel740 is primarily Real 3D’s architecture; however, the entire development of the Intel740 was definitely a joint effort between Real 3D, Intel, and Chips & Technologies.
4) Is is correct that you have better chances of optimizing drivers, because you guys developed the chip and know it therefore inside out, or does this not come into account?
We don’t want to give away too many secrets, but because of our intimate knowledge of the 3D pipeline and the architecture of the chip, we certainly believe we have a better chance of optimizing drivers and getting the most out of the hardware. In fact, as you see in the cover letter I had sent you with the board, we don’t believe any other 740-based board is going to perform at the level of the StarFighter.
5) How important is 3D Winbench for Real3D and how important do you think it is for the 3D card industry?
This was addressed in the statement above.
6) Is it correct that drivers can be easily optimized for one application, e.g. 3D Winbench 98?
I don’t know if it can be characterized as «easy», in fact, it’s extremely difficult. When you have highly skilled engineers like those we have at Real 3D and other board/chip companies have, though, if they put their mind and skills into optimizing for a particular game or application, the answer is usually yes.
7) Has Real3D optimized the driver of the Starfighter for 3D Winbench98?
NO. NO. NO. The drivers for StarFighter are optimized for the very best performance without compromising image quality, which we think will become more and more important to gamers and serve as more of a discriminator in the future. Keep in mind where Real 3D’s heritage comes from – the military where visual quality and fidelity means everything (it’s not a stretch to say it means lives).
8) If you were a normal home or office user, would you base your buying decision on the 3D Winbench scores of a 3D accelerator?
This was addressed in the statement above.
9) If ‘no’, what would you base your buying decisions on?
A combination of performance and quality, of which Winbench serves as one of the benchmarks and would be a part, but certainly not the only thing. You also need to look at reputation, experience, and word-of-mouth (which is something a web site like yours delivers – good or bad. That’s why your opinions are important to us). I know we don’t have much of a reputation in the mainstream graphics board business, but people can certainly extrapolate and assume that if we do first-rate graphics for the military and Sega, we most likely know what we’re doing with regards to 3D graphics.
10) Could you imagine that graphic card manufacturers would try and influence the buying decisions of users by optimizing their drivers particularly for 3DWinbench 98?
Yes, we believe this could happen.
11) What are the strengths of the Starfighter in your opinion?
Unmatched image quality at its price point, and certainly unmatched in a 2D/3D/video graphics board. The image quality combined with tremendous performance makes the StarFighter a board to seriously consider.
12) What are the weaknesses of the Starfighter in your opinion?
You can always ask for more polys or more fill, etc., but we think right now the StarFighter has all the performance and features it needs to be a leader in this industry.
13) How’s the weather in Orlando currently?
Sunny and 75 through the weekend.
Thanks to Chris Stellwag, Real3D
3Dfx Interview
3Dfx, the technology leader in the 3D graphic chip market, did also not hesitate in filling out this little interview:
1) Everyone should know it by now, you are producing awesome 3D gaming hardware. Is there any company you would see as a dangerous competitor right now?
We don’t see anyone as a dangerous competitor now. 3Dfx is uniquely focused on developing 3D technology as a 3D game platform as the new console. We also have a significant technological lead where it counts—in real game performance. We will maintain this lead.
2) Do you think that the i740 is going to be a big player in the 2D/3D card market soon?
It’s hard to say at this point. Intel is moving to a place where it is unproven.
3) How much do you guys believe in benchmark results published in magazines or on the web?
It depends on the benchmark—ones that assess real game play are the ones we take most seriously. Artificial benchmarks are part of marketing hype.
5) How important is 3D Winbench for 3Dfx and how important do you think it is for the 3D card industry?
3D Winbench does not translate well into how a card will perform on real games. It’s a boasting right but has very little to do with actual game play. A serious gamer would be crazy to buy a product just based on 3D Winbench figures.
6) Is it correct that drivers can be easily optimized for one application, e.g. 3D Winbench 98?
Yes. It’s part of the benchmarking «game.»
7) Has 3Dfx optimized the driver of the Voodoo^2 for 3D Winbench98?
No, we’ve optimized for the highest game performance. Some of those optimizations may help in 3D Winbench runs, but the focus is on game performance.
8) If you were a normal home or office user, would you base your buying decision on the 3D Winbench scores of a 3D accelerator?
(see above)
9) If ‘no’, what would you base your buying decisions on?
Game performance—how cutting edge games look and play is the real issue and should be the basis for evaluation.
10) Could you imagine that graphic card manufacturers would try and influence the buying decisions of users by optimizing their drivers particularly for 3DWinbench 98?
Yes. It turns out that there are many ways to cheat on 3D Winbench. Optimized drivers made only for running the benchmark is only one trick that gets played.
11) What are the strengths of the Voodoo^2 in your opinion?
Voodoo2 is completely software compatible with Voodoo Graphics. We preserve consumer’s software investments. We build on the strengths of Voodoo Graphics. The speed and quality provided by this next-generation are unequalled. We’re only just now getting a hint of want can be generated on a Voodoo2 platform.
12) What are the weaknesses of the Voodoo^2 in your opinion?
I suppose you could say lack of 2D support, but the focus of Voodoo2 is really on 3D graphics performance—period. It is meant to be the very best 3D graphics for gaming on a PC.
13) I know it’s annoying, but are there any news about Banshee?
Speaking of 2D and 3D! The world will see Banshee later this year.
Steve Schick, 3Dfx
nVidia Interview
These are the answers of nVidia, the company that currently sells the most 3D accelerator chips.
1) I know that you guys at nVidia are working like crazy, I once spoke to Dwight Diercks at 3 a.m. PST. Did you try getting your new beta OpenGL ICD out before the i740 launch on purpose or didn’t you care about Feb 12 ?
We work hard because we set the difficult goal of delivering the best products in the world. When we specified our OpenGL implementation we wanted it to be fantastic. We could have cheesed a miniport driver but we wanted to deliver a full ICD. A full ICD is a big job. It’s taken us a little longer than we had hoped but we put out the betas as soon as we had something we were proud of.
2) What are nVidia’s thoughts about the i740? Do you think it’s gonna be a tough competitor?
The i740 is slower than the Riva 128 and almost a year later. It will probably be a tough competitor for the companies that don’t have a great 3D product and who are trying to sell on price.
3) Who does nVidia see as most important competitors anyway?
We try to be our own most important competitor. Our goal is to obsolete our products before someone else does. In the process of doing that we will deliver truly state of the art products for our customers. We’ve set the goal of delivering an entirely new generation of graphics capability and performance every 12 months. We have no fewer than 3 separate development teams operating concurrently to obsolete each other. Imagine 3 internal competitors that know exactly what each other are doing! That’s the toughest competition anywhere!
4) How important is 3D Winbench for nVidia and how important do you think it is for the 3D card industry?
Every developing marketplace needs some way of measuring the goodness of a product. 3D Winbench is a good start at a metric for measuring goodness of 3D cards and systems. Is it perfect? No. Is it important that the industry support such a metric? Yes. Otherwise, it’s very hard to get a quick understanding of a product. But it’s just that — a *quick* understanding, not a *complete* understanding. You wouldn’t buy an automobile based on the horsepower of the engine. Yet knowing the horsepower of the engine does give you a quick understanding of the auto. And that’s the value of 3D Winbench.
5) Is it correct that drivers can be easily optimized for one application, e.g. 3D Winbench 98?
Yes, drivers can be tuned specifically for a benchmark. But trying to fool customers is a terrible practice and just won’t work for long for any company that tries it. Real customers run real applications. 3D Winbench is just an approximation of a whole bunch of applications. It’s a good first step but by no means as good as testing a bunch of real applications.
6) Has nVidia optimized the drivers of the Riva 128 for 3D Winbench98?
Yes! But that’s the wrong question. What you should have asked is whether we have optimized our drivers to exaggerate our performance on 3D Winbench98 relative to Riva 128 performance on real 3D applications. The answer to that is an absolute *NO*. We have certainly used 3D Winbench as one of the tools for improving our drivers. 3D Winbench does things with D3D that no existing applications mimic. Any company that has not extensively studied 3D Winbench is arrogant and has likely produced a set of D3D drivers that are far less robust than they should be. Similarly, any company that has high 3D Winbench scores with disproportionately less performance on real applications should be viewed with _great_ suspicion. NVIDIA is extremely self-conscious about Riva 128 performance and quality on real applications and has made a huge investment in testing and quality assurance. Looking great on benchmarks while looking slow on real apps is a slimy way to build a business. We won’t do it.
7) If you were a normal home or office user, would you base your buying decision on the 3D Winbench scores of a 3D accelerator?
No. 3D Winbench would be just one data point that would help me make a decision.
8) If ‘no’, what would you base your buying decisions on?
For 3D applications, I’d also like to understand the performance on a mix of real world applications that seem similar to what I will use. I’d also want to know how state of the art my card was because that would give me some idea of how long it would be before I would have to replace it. So I’d try to understand the basic technology beyond 3D processor speed: AGP, amount of memory, graphics memory bus width, and support for high resolutions. I’d also want to know it’s 2D performance and whether the technology was coming from a reputable company that would service their products. I’d also want to know the price and what software came bundled with the card.
9) Could you imagine that graphic card manufacturers would try and influence the buying decisions of users by optimizing their drivers particularly for 3DWinbench 98?
Of course. Look at what happened with Dhrystone. That’s why real applications performance must be measured in addition to 3DWinbench to get a complete picture.
10) What are the strengths of the RIVA 128 in your opinion?
Great 2D and 3D performance in a single card. Extraordinary performance upside due to the 128-bit bus. Riva 128 based cards will run new high resolution titles well at 800×600 and beyond. Plus, the true power of the Riva 128 is just beginning to be tapped. Our next software release will *dramatically* improve performance on both D3D and OGL. Riva 128 based cards scale in performance well up to 400 MHz on P2 systems. Throw in a world class OpenGL implementation (full ICD!) and then be amazed at how little it all costs.
11) What are the weaknesses of the RIVA 128 in your opinion?
For a few people, lack of 8M memory support for 1600×1200 true color. For most, es macht nichts. (Wow, Kevin shows his German roots!!! ‘es macht nichts’ means ‘it doesn’t matter’)
12) What’s the next chip of nVidia gonna be? When will it be launched?
Even better than the first…but that’s all I can say for now 🙂
13) When will the final OpenGL ICD be finished?
The OpenGL ICD is in testing now and will ship within 60 days.
14) Do you guys ever sleep?
Generally very little. Our CEO never sleeps and we all aspire to match his abilities.
Thanks a bunch Kevin for these detailed and very honest answers!
Kevin Schuh, nVidia
3D Winbench 98 Scores and Real World Benchmark Results — Not Exactly Equivalent
All benchmarks were run under the following configuration:
- Abit LX6 motherboard
- 2×32 MB SDRAM DIMMs
- Pentium 2 300 CPU
- Quantum Fireball ST 3.2 HDD
- Windows 95, OSR 2.1, DirectX 5, build 155
- 3Dfx Voodoo2 driver 4.10.01.0031-beta
- nVidia RIVA 128 driver 4.10.01.230
- Real3D Starfighter driver 4.10.01.0132
- ATI Rage Pro driver 4.10.2278
- ATI Rage Pro Turbo driver 4.10.2294
- The Voodoo2 is leading over the i740 by 2 points. Does that mean that i740 is as fast as Voodoo2? Certainly not!
- ATI Rage Pro Turbo driver is a lot faster than the previous Rage Pro driver? Only as long as you play 3D Winbench 98.
- i740 is faster than the RIVA? The games tell a different story.
Incoming from Rage is one of the most awesome games this year and can not possibly be ignored. Here the world’s in order again. The Voodoo2 shows what it means to be the technology leader and ATI’s Rage Pro Turbo driver shows that ATI didn’t mean 40% more performance in the press release, they wanted to say 25% less performance.
Voodoo2 takes it over the top, RIVA looks great too, Forsaken will be a game that looks great and plays well even on slower systems.
Please note that these are frame rate results of the downloadable demo version of Turok. The retail version on CD show completely different and much higher frame rates. Results scored with the retail version of Turok cannot be compared to any Turok TMARK results published on this website.
Good old Turok also agrees with Voodoo2 being the No.1 and RIVA being the No.2. Note the difference between the Rage Pro Turbo and the Rage Pro driver.
The i740 will soon come with an ICD for OpenGL which will improve the performance in Quake II considerably. The OpenGL wrapper that’s used here can only offer 37% of the Voodoo2 performance. ATI’s OpenGL driver is supposed to be released in the first half of March 1998.
So What Are We Going to Do Now ?
Well, let’s take the advice from the industry. Do not base your buying decisions on 3D Winbench 98. Never mind about awards given due to results of this benchmark. Intend on real world results and let nobody make you believe that an artificial benchmark is able to tell you any truth about the 3D gaming experience.
Is this the place for blaming ZDBOp? Certainly not! They had an extremely hard job in putting a benchmark together for Direct3D which is pretty comprehensible, particularly on the side of the quality checks. So what is wrong? If we want to point the finger then we could do this at editors who make their job easy by just running this one benchmark and hence by not doing the research that it takes for this job. In the first place we have to point the finger at ourselves though. Everybody is asking for an easy solution and be honest to yourself, don’t you want to believe in just one benchmark telling it all? If editors are making their job easy by simply relying on 3D Winbench 98 then it’s up to the readers telling these guys off. But as I said, we all want an easy solution and wouldn’t it have been nice if this one benchmark would have told us all? ZDBOp had the courage of trying to live up to this wish. They didn’t succeed, but when you think about it … they couldn’t, could they?
A wise man once said that the people want to be cheated at. I disagree with this in terms of computer hardware performance. We want the truth, we are sick of falsely inflated benchmark results, but we have to face the fact that this takes more work than just reading one magical number. That means that we also have to be able to ask questions, be critical, disbelieve the PR hype. And the companies who are trying to fool us … ? Well, we’ll see if these companies will be able to carry on with their way of doing business. This includes publications that don’t really care about authenticy of their results. It’s up to you … take the chance, get informed, don’t let them fool you!
