A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer might be cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers.

While Mozilla engineer Rob Sayre was benchmarking Firefox 4 with different browsers, he noticed something with Internet Explorer 9 - Internet Explorer 9 was around 10 times faster than the other browsers in a particular test (math-cordic) in the SunSpider benchmark. While Chrome and Opera scored took around 10ms in that test, Internet Explorer 9 finished it it in about 1ms.

Update: We have changed the title of the story as some commentators have rightly pointed out the previous one was too biased.

Update 2: To make user experience better, the IE team has been tuning the JavaScript engine for real world sites over the last few weeks. Based on this, an updated platform preview build (Version 7) for IE9 has been released for you to try.

Sayre investigated further by modifying the code for that test used in the SunSpider Benchmark a bit. He made two variations of the test little bits of codes to the original – one by adding a “true” and another by adding a “return”. As these "true” and “return” in this context does not do anything there should be minimal impact.

The result though is startling – Internet Explorer 9 took around 20 times longer with the two new tests compared with the original. While the original took only 1ms in my laptop, the other two took around 20ms each.

This is quite a startling result. The result for cordic (which is the original) should not differ much from the other two.

Here are Google Chrome and Opera respectively with the same test:

As you can see very clearly the addition of “true” and “return” has minimal impact on the result with Opera and Chrome. The result in all three tests stays almost the same at around 9.6ms in Chrome and 7.7ms in Opera. With Internet Explorer 9, though, the addition of “true” and “return” is enough to send up the 1ms it got in the original test to 20ms.

There are three possible explanation for this weird result from Internet Explorer:

1. Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation. [Update: This analysis from Hacker News seems to support this - http://news.ycombinator.com/item?id=1913315]
2. Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine solely for the SunSpider Benchmark. This seems very unlikely to me.
3. A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

What do you think?

Click here if you want to run the test yourself.

The diff for the modified tests are available here and here.

Further Readings:

1. Reporting a bug on a fragile analysis

2. JS Benchmarks: Closing In