Did Internet Explorer 9 Cheat In The SunSpider Benchmark?
While Mozilla engineer Rob Sayre was benchmarking Firefox 4 with different browsers, he noticed something with Internet Explorer 9 - Internet Explorer 9 was around 10 times faster than the other browsers in a particular test (math-cordic) in the SunSpider benchmark. While Chrome and Opera scored took around 10ms in that test, Internet Explorer 9 finished it it in about 1ms.
Update: We have changed the title of the story as some commentators have rightly pointed out the previous one was too biased.
Sayre investigated further by modifying the code for that test used in the SunSpider Benchmark a bit. He made two variations of the test little bits of codes to the original – one by adding a “true” and another by adding a “return”. As these "true” and “return” in this context does not do anything there should be minimal impact.
The result though is startling – Internet Explorer 9 took around 20 times longer with the two new tests compared with the original. While the original took only 1ms in my laptop, the other two took around 20ms each.
This is quite a startling result. The result for cordic (which is the original) should not differ much from the other two.
Here are Google Chrome and Opera respectively with the same test:
As you can see very clearly the addition of “true” and “return” has minimal impact on the result with Opera and Chrome. The result in all three tests stays almost the same at around 9.6ms in Chrome and 7.7ms in Opera. With Internet Explorer 9, though, the addition of “true” and “return” is enough to send up the 1ms it got in the original test to 20ms.
There are three possible explanation for this weird result from Internet Explorer:
Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation. [Update: This analysis from Hacker News seems to support this - http://news.ycombinator.com/item?id=1913315]
- A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.
What do you think?
Click here if you want to run the test yourself.
The diff for the modified tests are available here and here.
1. Reporting a bug on a fragile analysis