Did Internet Explorer 9 Cheat In The SunSpider Benchmark?

By Ricky on November 17th, 2010

A Mozilla engineer has uncovered something embarrassing for Microsoft – Internet Explorer might be cheating in the SunSpider Benchmark. The SunSpider, although developed by Apple, has nowadays become a very popular choice of benchmark for the JavaScript engines of browsers.

While Mozilla engineer Rob Sayre was benchmarking Firefox 4 with different browsers, he noticed something with Internet Explorer 9 - Internet Explorer 9 was around 10 times faster than the other browsers in a particular test (math-cordic) in the SunSpider benchmark. While Chrome and Opera scored took around 10ms in that test, Internet Explorer 9 finished it it in about 1ms.

Update: We have changed the title of the story as some commentators have rightly pointed out the previous one was too biased.

Update 2: To make user experience better, the IE team has been tuning the JavaScript engine for real world sites over the last few weeks. Based on this, an updated platform preview build (Version 7) for IE9 has been released for you to try.

Sayre investigated further by modifying the code for that test used in the SunSpider Benchmark a bit. He made two variations of the test little bits of codes to the original – one by adding a “true” and another by adding a “return”. As these "true” and “return” in this context does not do anything there should be minimal impact.

The result though is startling – Internet Explorer 9 took around 20 times longer with the two new tests compared with the original. While the original took only 1ms in my laptop, the other two took around 20ms each.

This is quite a startling result. The result for cordic (which is the original) should not differ much from the other two.

Here are Google Chrome and Opera respectively with the same test:

As you can see very clearly the addition of “true” and “return” has minimal impact on the result with Opera and Chrome. The result in all three tests stays almost the same at around 9.6ms in Chrome and 7.7ms in Opera. With Internet Explorer 9, though, the addition of “true” and “return” is enough to send up the 1ms it got in the original test to 20ms.

There are three possible explanation for this weird result from Internet Explorer:

Microsoft cheated by optimizing Internet Explorer 9 solely to ace the SunSpider Bechmark. To me, this seems like the best explanation. [Update: This analysis from Hacker News seems to support this - http://news.ycombinator.com/item?id=1913315]
Microsoft engineers working on Internet Explorer 9 could have been using the SunSpider Benchmark and unintentionally over-optimized the JavaScript engine solely for the SunSpider Benchmark. This seems very unlikely to me.
A third option (suggested in Hacker News) might be that this is an actual bug and adding these trivial codes disaligns cache tables and such throwing off the performance entirely. If this is the reason, it raises a serious question about the robustness of the engine.

Update: Microsoft has updated their blog post addressing this issue. (Read here.) They attribute this to dead code elimination. They did not include any explanation as to why the dead code elimination fails miserably on adding "true" or "return", which changes nothing in the actual function, or even why it fails when the for statement is replaced by while as demonstrated in Hacker News. It could be a bug or them over-training the dead code elimination method for SunSpider though.

With this revelation, I cannot help but wonder if Microsoft did something like this in their HTML5 Demos where Internet Explorer 9 beats Google Chrome hands down.

What do you think?

Click here if you want to run the test yourself.

The diff for the modified tests are available here and here.

Further Readings:

1. Reporting a bug on a fragile analysis

2. JS Benchmarks: Closing In

« Previous Post

Related Posts by Tags: benchmark, cheat, chrome, ie9, internet explorer, internet explorer 9, javascript, Microsoft, opera, sunspider

Pingback: Tweets that mention Internet Explorer 9 Caught Cheating In SunSpider Benchmark -- Topsy.com
Pingback: Lastest Internet Marketing News | Affiliate Business Marketing and Hot Internet Marketing
foo bar.

Nice of you to decide what the results mean for us. No one would ever guess that you hate Microsoft and are a completely biased *censored* who doesn’t know journalism and wouldn’t know it if it reached up and bit him on the ass.
- Pathogen
  
  umad?
- MaGaO
  
  Erm… I find a question mark at the end, so they aren’t deciding anything for you. What’s even more, the article states three hypothesis, only one of which would require bad faith. Nobody would ever guess that you love Microsoft and are a completely biased *censored* who doesn’t know how to read critically and wouldn’t know it if it reached up and bit you in the ass (not “on”, mind you).
  - A B
    
    Please read the headline of the article and the first line. If that’s not making a conclusion for us, then what is?
    - Fredmastro
      
      I agree. I was under the impression it was obvious cheating, with no other conclusion. After reading, I’m like, it’s a possibility but there are two other options. So the title is very misleading.
  - http://pulse.yahoo.com/_OLWZEF6PGQOKJQY6B7S2ZRZNFA Ron Johnson, Jr.
    
    Adding a “?” to the end of a sensationalist headline is an incredibly old technique, used only by ethically-challenged rumor-mongers.
- js
  
  @foo bar: How do you explain it then?
- Km_2_go
  
  The article lists several explanations and their feelings on the likelihood of each. If you have an informed opinion that differs, why not voice it rather than trotting out the tired old derp accusation of “biased journalism”?
Guest

microsoft? cheating?? who would have thought???
http://www.facebook.com/paul.hutch Paul Hutchinson

This shouts bug at me.
Ice_shack

I find it rather odd that it was Firefox being tested and Chrome and Opera results were being displayed. Maybe spend more time with your own product and less time hating Mic$oft and your level of journalism will rise above the gutter.
It’s obvious you don’t like IE, hell anybody developing for the web don’t like IE. At least I don’t waste my day worrying about it.
- Boris
  
  Rob Sayre (a Mozilla developer) was testing Firefox. He also put the test code up on a website so anyone can run it. He also didn’t say anyone was cheating or anything; he just presented the data he found. You can read his actual blog posts linked under “Further Readings” on this article.
  
  Ricky (the author of this article) chose to run it in Opera and Chrome and put the resulting screenshots in his article. Why he did that is unclear; perhaps he doesn’t actually use Firefox but does those two browsers?
  - http://digitizor.com/ Ricky Laishram
    
    The reason I did not run this in Firefox was because I do not use Firefox. All that I wanted to demonstrate was how the result should be. Both Chrome and Opera performed as expected and so I did not see any reason to do the test again with Firefox.
- Zarmanto
  
  I suspect that since it was a Mozilla engineer conducting the tests, he simply didn’t want people thinking that he was tooting his own horn by trotting out the Mozilla test results as his comparison. Thus, he used two other (competitor’s) browsers for this discussion, both of which illustrate his point quite effectively. The bottom line is that something is clearly not operating as it should in the IE9 code, and with results as dramatic as described herein, I would have to agree with the author that the most likely explanation is indeed foul play.
  - http://digitizor.com/ Ricky Laishram
    
    He did test Firefox. The choice of Chrome and Opera was mine like I have said earlier.
Toji

@foo bar: His conclusions are perfectly reasonable given the evidence. It’s like finding out that your game starts performing significantly worse when you rename the executable from “quake3.exe” to “quack3.exe”. Obviously there’s something fishy going on.

When it comes to benchmarks, which people are so fond of tossing around as “proof” of one products superiority over another, fishy numbers almost always indicate someone is trying to game the system. This is no different, and I think that it’s safe to assume that Microsoft is trying to pad their numbers unless they can prove otherwise. Remember, the most likely explanation is usually the right one.
- http://www.youtube.com/watch?v=jCJLK3KgzyQ mfranke
  
  Toji, excellent points.
  
  I remember other “benchmark cheating” by Intel, AMD, ATI and Nvidia.
  
  The Quake 3 example you gave.
  One of the 3dmark suites was cheated so that all geometry was ignored beyond the “on rails” camera view.
  Half Life 2 was having its textures reduced in quality to perform better.
  The list goes on.
skp

so you changed the test to meet your needs. Shouldnt the headline say ie 9 tem times faster then firefox unless you change test.
- Ricky
  
  No the test is not changed to meet anyone’s need. In case yo have not noticed I did not even test with Firefox – just Chrome and Opera.
  
  The point is if making these trivial changes in the code impacts the performance that much, there is something fishy going on.
  - Jose_X
    
    You are very likely correct, but Microsoft can set the record straight by simply showing us the responsible code.
    
    They can optimize so that their javascript engine is not a javascript engine at all when it comes to the benchmark. If the utility of this is hardly generalizable, then they are clearly gaming the benchmark.
    
    We’ll see how Microsoft responds. If it’s an honest mistake, they should probably have the bug fixed in no time and their new code should be much more robust to different but similar benchmarks.
    
    More importantly, how well do they perform on typical standards based webpages?
    
    You know, there was a time when there was a standard for coming up with scientific results and that involved openness. I don’t think closed source software developers really understand this fundamental point.. or they do but have trouble dealing with the potential implications to their take-home pay.
    - Anonymous
      
      How about you find out yourself Jose_X, download it. IE 9 beta is public for you to download.Why would Microsoft bother to try and cheat on this? Besides millions of users of IE 9, not to mention thousands of web developers have been hammering away at IE 9 for several months now. If Microsoft was trying to cheat, then it would have already shown itself by now if not when it was first released several months ago.The truth of the matter is that IE 9 most likely really is that fast overall considering that is what the public beta testers along with web developers have been saying for several months now on.
Jata

Microsoft have tried since the launch of the IE9 platform previews to discredit the SunSpider benchmark in favor of Hardware Acceleration as the true speed. Why they would want to cheat in a benchmark they consider microbenchmarking?
I guess the Mozilla engineer should invest more time in trying to figure out why the Firefox hardware acceleration works so bad compared to IE9 instead of trying to throw paronoid hypothesis
- Jose_X
  
  Why they might want to cheat? For marketing purposes.
  
  If the hardware acceleration is honest and generalizes, then that is widely usable and should count. If you tune to a very narrow type of code that will not be found frequently in the real world, then that is not that useful.
  
  What are the specs to the test they want people to perform?
jc

Mozilla FF 4b7:
============================================
RESULTS (means and 95% confidence intervals)
——————————————–
Total: 17.1ms +/- 3.1%
——————————————–

math: 17.1ms +/- 3.1%
cordic: 5.8ms +/- 5.2%
cordic-with-return: 5.7ms +/- 6.1%
cordic-with-true: 5.6ms +/- 6.6%
Chris

This article is terrible and an exaggeration. Way to be a dramawhore for internet traffic.
Martin

Actually there is another possibility:
I had a look at the code for the cordic test. And a modern compiler of a static programming language had probably resulted in 0ms. The reason is simple, every function that is called in that context is a “function” in the mathematical sense – thus it has no side effects. The function has no result, and thus the result isn’t used.
If the compiler is intelligent he will notice that the whole script can be replaced by “return;”, because it doesn’t do anything. In a dynamic language you can’t know of course that some of the stuff that I just said doesn’t change at any point, so you can’t use the same optimizations, but if you do the optimizations only for the final loop, I can’t see IE shouldn’t just skip the whole (25.000 times repeated) loop…
So – I don’t say that IS what happens – but it is absolutely possible that the IE JS compiler is simply slightly more inteligent.
- Ricky
  
  Yes an intelligent JS Engine should do that but as someone has found out in hacker News, replacing the “for” with “while” also impacts the performance.
  
  http://news.ycombinator.com/item?id=1913315
- Jose_X
  
  And it’s also possible that the “smarter” compiler has been “tuned” by a human for this particular benchmark. That would be called gaming.
  - Anonymous
    
    Which has essentially been disproven Jose_X.
MozSux

Lame…
http://psoug.org/ Oracle Tech

Microsoft caught cheating AGAIN? Wait a second, let me put on my “shocked” face.
http://jorendorff.blogspot.com/ Jason Orendorff

I work on Mozilla’s JavaScript VM, so take this for what it’s worth:

Making JavaScript fast only on one meaningless little script (a single SunSpider benchmark) in a way that won’t speed up any real Web page, ever, is a waste of time. Whether you want to call it “cheating” or not is beside the point. The point is, you can’t take IE9′s SunSpider scores as a measure of how fast its JS engine is on real web pages.
- to Jason Orendorff
  
  I agree. I also believe that benchmarks help sell products even if the benchmark is not real world.
  
  I feel the issue at hand is how likely is it that a person/team spent time purposefully attempting to create buzz through artificial/fraudulent means.
  In this comments section I have read one post which addresses this question by providing a possible option as to why this is happening. Beyond that post, there have been no counter arguments.
  I like your post, I believe it applies however to a different larger picture than the one the author is referring to.
  In addition, if they are found to be ‘cheating’ here..are they doing it with others? If so, to what degree?
http://heydojo.co.cc Tony

Cheats never prosper.
Schizoid

All this bs from both sides. I really don’t care how it or any other browser performs in a benchmark and whehter or not it’s cheating. Forget testing the code from the back-end…I want to know which loads actual websites/pages faster? If you can’t tell because they’re too close in ms then speed is irrelevant and it’s all about features, etc.
- st33d
  
  Most websites load their pages WITH Javascript you muppet.
  
  A crap Javascript engine will load pages slower.
anon

Javascript is a security risk, and shouldn’t be used by anyone.

The only reason JS and JS speed is so hyped by the media and browser makers is because massive amounts of the IT industry loves the idea of web based applications.

Proprietary software, like MS office, is not good for the user because the user is reliant on a 3rd party to keep supplying software for the user’s data. At least the user can, if they really want to, keep an old PC maintained with copies of the proprietary application.

Web applications are much more risky from the user’s point of view. The application is really running on someone else’s computer, and frequently they even hold onto the user’s documents too. If the kind of control proprietary applications on a desktop PC made the world’s richest man, imagine how industry is foaming over the profit prospects from the control web applications potentially gives them?

Once you realise web applications are not in your interest (FOSS is most likely to be so), you will realise that JS is only used on line for annoyances, and so stepping away from the massive security risk it is is easier.
- monkey
  
  I can’t believe that you think JS is a security risk. Do you live in the dark ages?
  It runs in a closed environment, the browser.
  It is only annoying when badly written.
  - nightwalk
    
    Actually, javascript can bootstrap java & provide the real IP of the system to the remote website. That *definitely* qualifies as a security risk for those living under oppressive regimes who depend on technologies like tor & ssl to keep them safe. Not that it matters much, though. Flash can supposedly do even worse
    - Walter
      
      Javascript doesn’t *have* to bootstrap Java. The one can be enabled while the other is disabled.
- Lakota
  
  JS runs in your browser, not the server. You have no idea what the fuck you’re talking about.
  - Jose_X
    
    JS can run on the server, but that is not even what parent was saying. They were saying that the real heart of the application (or at least important parts wrt privacy) can be run on the server.
    
    A simple web app, a wiki, uses the browser as well as the server.
    
    Or do you think all browser applications seize their contact with the server forever once the main page is loaded in the browser?
  - Michael Westen
    
    buddy, U definitely need an update: http://en.wikipedia.org/wiki/Server-side_JavaScript
SickPuP

I am confused as to how this is cheating? The engine runs faster than all other browsers and performs gets the correct result. I call that efficiency and not cheating. The author uses the term over-optimized. That is one of the dumbest phrases ever. The point of optimization is to be the best, you can’t go over it. It sounds to me like the JScript engine in IE9 is better than all other browsers and people are jealous. I saw a test of the top 4 browsers which compared over 200+ tests, only one which was the one mentioned in this article. In that IE9 outperformed in almost every test. So did they “cheat” on those too? If it works and its better people will try to devalue it with pointless articles.
- hrimhari
  
  I’m sorry that you missed the point. Let me try to help you here.
  
  The author’s proposal is that optimizing the code in a way that only produces optimization in a Benchmark is cheating. It is so because the optimization only applies when comparing browsers, not in real day-to-day use.
  
  The proof would be that adding insingnificant changes to the test code will make the result jump from 1ms to 20ms, only in IE. While other browsers are consistent, with a result way better in the modified test (9ms and 6ms).
  
  It would mean that IE9 is superfast in this benchmark, but is superslow in day-to-day use. While the other browsers would be fast in both benchmark and day-to-day use.
  - monkey
    
    And after all, consistency is key. If it isn’t consistent, whether maliciously so or accidentally, it is a FAIL!
  - guest
    
    Did it not occur to anyone that Sunspider is not the only JS benchmarking tool and IE9 seems to score well in other tools too?
    
    IE9 is in a class of its own compared to IE8 and is surely comparable in speed to Chrome which I used exclusively. I9 is fast in real life performance, no developer finding a single line of suspect code in respect to synthetic JS bencmarks is going to change that.
    - haydoni
      
      Yes, which begs the question: have they “cheated” these tests also (the author asks this at the end of their post).
      
      @hrimhari mentions above that a small deviation in the test code makes only a small difference (as one would expect) in other browsers’ benchmarks but IE9 is significant slower…
      Clearly this suggests some parts of IE9 have been coded specifically to beat the benchmarks, be these Sunspider or otherwise…
      This work does not improve IE9 in real life performance.
      
      “IE9 is in a class of its own [with Chrome (!)] compared to IE8″ this made me laugh.
      
      No one is saying IE9 isn’t fast, what they are saying is that it’s nowhere near as fast as some benchmarks suggest.
  - Me
    
    But doesn’t that imply that the benchmark itself is flawed? If a company can manipulate the results like that, then what good is a benchmark anyway? Maybe the other browsers are exploiting other flaws as well to increase their speeds by only 5%? I think that this article should focus on how bad the benchmark product is rather than IE9 exploiting it.
    - the_madman
      
      No. The benchmark isn’t flawed because you don’t manipulate the results, you manipulate the benchmark. When you manipulate the benchmark, then test the manipulated benchmark for all browsers, all of them will have to do the rounds of the manipulated benchmark. In the case of Opera, Google Chrome and Firefox, the results change very little in respect to the other browsers. However, in Internet Explorer, the results fluctuate dramatically, even for changes considered insignificant to the other browsers. This is the suspicious behaviour that the original author is pointing out.
  - Anonymous
    
    I suspect there aren’t very many people here who are familiar with optimising compilers, much less optimising compilers for Javascript.
    
    The changes may be “insignificant” to us, because we know they are functionally superfluous. But that doesn’t mean that it won’t be able to fool an optimising compiler. This will come as no surprise to C and C++ developers should already be intimately familiar with the fickle nature of optimising compilers: seemingly innocuous and functionally irrelevant changes to code can easily defeat an optimising compiler.
    
    This is no exception. It is known that IE9′s dead code elimination optimisations DO work in general (i.e. outside of SunSpider). It’s just not very smart and isn’t able to recognise every variation and permutation of dead code. There more analysis at Hacker News, which demonstrates that IE9′s dead code elimination does work in general:
    
    http://news.ycombinator.com/item?id=1915380
    - the_madman
      
      Then you’re suggesting that Microsoft’s, “dead code” compiler is recognising that the javascript in this test is dead code and simply isn’t running it.
      
      However, this causes a problem: it’s a BENCHMARK. It’s SUPPOSED to run the code to see how quickly the code will run – simply not running the code, then claiming, “Look at how fast our browser runs this code!” is dishonest.
      - amazed
        
        Do you realize how stupid your comment is?
        
        “it’s a BENCHMARK. It’s SUPPOSED to run the code to see how quick[...]”
        
        If the browser _knew_ this was a benchmark and acted differently on this ground, then _that_ would definitively be cheating.
        
        IE9 behaves in this benchmark just like it does on any website. As has been pointed out in the comments, IE9 does _indeed_ perform dead code elimination on _real_ scripts. This is exactly the behavior that I want to measure.
        
        Because IE9 doesn’t run the dead code, I agree that the actual figures are pretty much meaningless… All they do is show that other browsers are behind because they don’t do dead code elimination.
        
        I wouldn’t call that IE cheating, it’s simply that the benchmark is crap. (And this isn’t just rhetoric, those micro-benchmark aren’t any good now that the engines have perfomances so close one from another.)
- Daniel
  
  Yes, over-optimising is possible, and it is a Bad Thing. Here’s an example (that is only slightly exaggerated from the real world too). Let’s say the MyHorseSoft racing stable found a way to breed horses that would win every race of exactly 2000 metres. But if the race was run over any other distance, even 1999 or 2001 metres, the horses would break all their bones and die horribly. Sure, MyHorseSoft would argue their horses are beautifully optimised. But to anyone else it would be clear that these horses were over-optimised and useless for the world of real racing that takes place over a range of distances.
- Jose_X
  
  >> I am confused as to how this is cheating? The engine runs faster than all other browsers and performs gets the correct result.
  
  The other comments covered “over-optimization”.
  
  I’ll just add that in fact they did not do well at all. That is the point of this article, that using a very slightly legally modified benchmark led to much worse results. IE did the worst.
  
  So IE is looking like the slowest browser… unless you run a very specific test.
  
  This is called cheating as it is useless in real life. It gives a data point that is nowhere generalizable to other cases (unless you are the marketing arm of the company and obviously then overplays that data point to suggest it is “typical”).
Pingback: Did Internet Explorer 9 Cheat In The SunSpider Bechmark? - internet
http://www.ecomstation.com/ FutureUser

Back in the early 90′s, Quarterdeck came out with a superior DOS memory manager called QEMM. However, a Windows 3.x update caused the QEMM to become unstable. The “fix” was simply to rename the QEMM directory to “QEMM386″ or something other than just “QEMM”. This was clearly a case of *Predatory Programming* by Mafiasoft. So, it comes as no surprise that two decades later, a standard benchmark would show “rigged” results for the convicted monopolist. Cheating is a major part of the Mafiasoft DNA.
- Jose_X
  
  And that is why they fear FOSS so much. They require darkness to allow their deceptions to flood the marketplace. Competitors that have an ounce of respect for society get left in the wash. Consumers lose.
Pingback: Microsoft caught cheating on IE9 SunSpider JavaScript tests? | ZDNet
AssumeTooMuch

Everyone assumes too much:

http://en.wikipedia.org/wiki/Hanlon%27s_razor

Never attribute to malice that which is adequately explained by stupidity.
- Robert J. Hanlon / Robert Heinlein
Pingback: IE9 SunSpider oddness prompts Microsoft benchmark-gaming rumors « Technology News Snippets
Pingback: IE9 Platform Preview 7: performance is the priority | MyGeist
Pingback: Can jailbreaking my iPad get me free internet?
Pingback: Tech Reviews » IE9 Platform Preview 7: performance is the priority
Pingback: Computer Hardware News » IE9 SunSpider oddness prompts Microsoft benchmark-gaming rumors
Jack Repenning

The differences among your three hypothesized explanations (cheating, over-optimization, test-specific bug) really amount to differences in the mental state of the engineer(s) who did the coding. Human mental state isn’t discoverable by benchmark or code inspection.

The Hacker News link seems to implicate dead-code removal, a notorious problem for benchmarking. Dead-code discovery is hard, generally idiosyncratic, useful, and yet problematic for benchmark design. An interesting exercise would be for someone to survey the Web at large, to see how often this particular dead-code pattern actually appears. If it’s common, IE has done a good thing. I suspect, in view both of the characteristic newness of JavaScript code in general (compared to legacy Enterprise code), and also of the apparent fragility of this particular dead-code recognition, that such a survey would show that this particular optimization by IE was pointless in the real world. But whether it’s malicious or just silly is a question beyond the mere state of the code.
- Robert O’Callahan
  
  The point is that inserting the statement “true;” into the code obviously changes nothing. Likewise changing the ‘for’ loop to an equivalent ‘while’ also breaks IE’s analysis. When an analysis only works on a piece of benchmark code and not on trivial variations of that code, that’s cheating; the developers have clearly produced an analysis to get great results on the benchmark and *nowhere else*.
  - Anonymous
    
    Except it’s already known that isn’t true. IE9′s dead-code removal DOES work in general, it’s just not very smart.
    
    http://news.ycombinator.com/item?id=1915380
    
    The above provides a javascript snippet that is completely different (functionally and semantically) from the Cordic math benchmark. IE9 correctly recognises it as dead code and removes the entire loop, completing the test in 1ms. Adding a superfluous “true;” or return statement defeats the optimisation here as well.
    
    So we know that IE9 isn’t optimising specifically for SunSpider: because their optimisations do work in general. It’s just that their compiler’s code analysis is not smart enough to recognise vacuously useless statements.
- Jose_X
  
  No, malice can certainly be suggested depending on what we would find in IE/Windows’ code.
  
  In any case, the most likely scenario is that IE is not nearly that fast.
http://allinfrastructure.blogspot.com/ FeriCyde

Wouldn’t be the first time that Microsoft’s benchmark statistics were called into question:
http://www.linuxtoday.com/news_story.php3?ltsn=1999-05-04-005-05-NW-LF

(link above from over 10 years ago).
Pingback: Did Internet Explorer 9 Cheat In The SunSpider Bechmark?
Pingback: Internet Explorer 9 incepe sa se clatine - kandrei.ro
Pingback: Discover the Benefits of Article Marketing
Pingback: Did Internet Explorer 9 Cheat In The SunSpider Bechmark?
Pingback: Affiliate Marketing- Most Profitable Affiliate Internet Marketing Tips | Affiliate Business Marketing and Hot Internet Marketing
Pingback: IE9 Preview 7 Now Available For Download: Real World Site Performance
Pingback: Microsoft News » IE9 Platform Preview 7: performance is the priority
Pingback: Did Internet Explorer 9 Cheat In The SunSpider Bechmark? - iKnew
Pingback: مايكروسوفت تغش في مقياس SunSpider | إلكتروني
Pingback: Dudas en los test de Internet Explorer 9 ( IE9 ) : Noticias2D
http://blogs.msdn.com/tims Tim Sneath

The official Microsoft response can be found at the bottom of the IE9 PP7 announce blog: http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx

Thanks, Tim
- Jose_X
  
  Yes, that was covered above. How did they sell their results ahead of time? It’s fun now to go back and say that if we all adjust the benchmark a dash then their browser is “in line” with the others.
  
  Optimizing to web pages is noble, but not if you sacrifice much elsewhere. It’s also easy to oversell that approach by claiming you are optimizing to patterns generally when in fact you might mostly be optimizing very narrowing to specific webpages or websites (perhaps even those using Microsoft development tools). Over time as the pages change the browser can become very mediocre (or the worst as this benchmark appeared to suggest).
Pingback: Notebooks 24
Socrate

Another sensational article.

The truth is that IE9 includes a new algorithm for “Dead Code Elimination”.

They just explained how it works here: http://blogs.msdn.com/b/ie/archive/2010/11/17/html5-and-real-world-site-performance-seventh-ie9-platform-preview-available-for-developers.aspx

Kudos to IE team.
- the_madman
  
  Like I said earlier, this is a benchmark. It’s supposed to run the code anyway, to see how well the code runs. Not running the code, then saying, “Look how quickly our browser runs this code!” is dishonest.
Ajzimmerman

Well you can spell BENCHMARK right and you’re good…
Pingback: Microsoft Releases IE9 Platform Preview 7 « The Social Developer
Pingback: Did Internet Explorer 9 Cheat In The SunSpider Benchmark? | Neorack Tutorials
Pingback: W32.Myzor.FK@yf Have you seen this virus problem?
Pingback: Cool-O-Rama » IE9 Platform Preview 7: performance is the priority
Pingback: IE9 may have been cheating in the sunspider benchmark | thefinalcastle.com
Pingback: Cool Internet Marketing images | Affiliate Business Marketing and Hot Internet Marketing
Pingback: IE9 Platform Preview 7 soll bei Benchmarks schummeln - Internet | News | ZDNet.de
Pingback: Do You recognize These Internet affiliate marketing Behind The Scene?
Pingback: Web Marketing Agency |
not cheating

As shown by

http://news.ycombinator.com/item?id=1915380

IE9′s optimizer does work on completely different codes, so we know it’s not cheating for Sunspider.
- Jose_X
  
  I’m going to guess that it was clear to competitors that the benchmark required you to run the code.
  
  Microsoft is making a legitimate argument that these benchmarks are not that valuable ..except to poke around to see certain general features and speeds of the browser (except to the extent you bypass what the benchmark intended to test).
Nathan

I think IE9 should be good to use atleast.
Pingback: IE9 Platform Preview 7: performance is the priority - Free Web Design Phoenix
http://www.mycooktube.com/Chumakov/ Chumakov

I have a problem with the overall premise of your article but I still think its really informative. I really like your other posts. Keep up the great work. If you can add more video and pictures can be much better. Because they help much clear understanding. thanks Chumakov.
http://gwan.ch/ Pierre

MICROSOFT IIS 7 also cheats:

IIS 7 rules under Windows (almost all other Web servers are slower) only because MSFT sabotaged the Windows user-mode to slow-down everybody but IIS (IIS is unaffected because it runs as a kernel driver), see:

http://gwan.ch/en_windows.html

But IIS is slower than all when you compare IIS with other Web servers on the same machine running Linux:

http://gwan.ch/en_linux.html

And the fraud is even bigger with C# as .Net is *far slower* than PHP and Java:

http://gwan.ch/

So, seing IE cheating is just business as usual.
- Jose_X
  
  That is why I wish that Mozilla would vocally and in action place more emphasis on open platforms (like Linux) where it actually has a chance to win long term, then to let Microsoft dictate the rules of the game on their field.
Pingback: Anonymous
Pingback: Tony Parker | Hot Music TV Information
Pingback: Car Moving Companies: Promoting A Auto From Place | Product Guide Reviews
Pingback: ¿IE9 hace trampa en el Benchmark SunSpider? : TecnoReviews
Pingback: Microsoft Caught Cheating in the Sunspider Benchmark
Pingback: How to Successfully Connect a Printer to a Windows Vista Laptop Over a Home Network to a Windows XP Desktop – Info Barrel | homehold pc
RP

If you read the MS blog post with an unbiased view then it makes sense completely. The original function used a lot of CPU cycles and produced nothing, so the simplistic dead-code elimination in the compiler simply ditched it. When you add a return or true then the function starts to return data, so it wasn’t ditched. Whilst this doesn’t help test the performance of this particular code fragment it would help in a real-world situation, which is exactly the thrust of the article.

Personally, I want the best-performing, most compliant and most functional real-world browser and I see these optimisations as appropriate for that goal. By all means criticise the performance of any aspect of any browser but please everyone leave your MS-bashing (or Apple/Mozilla/whatever) boots at the door and let’s have some reasoned, rather than prejudiced, arguments about relative merits.
- Jose_X
  
  For the given benchmark, they bypassed it and did not speak up despite likely knowing what was up when they did so much better (rather than the worst).
  
  Generally, if they have a different benchmark they prefer, then they should post those details clearly.
- Jose_X
  
  BTW, long-term, Microsoft will not be able to keep up with FOSS once we start optimizing to specific web pages.
Braden Keith

Its a wonder you keep writing with all the idiots tearing you down. You wrote a fine article. Thanks for the time you put on it. Ignore the haters and follow your passion.
Pingback: Microsoft IE9 SunSpider JavaScript results raise questions « NetSoft Consultants
Pingback: Workshop Internet Marketing at RSM, Erasmus – case studies | Affiliate Business Marketing and Hot Internet Marketing
http://www.chocolim.com Chocolim

if the dead code elimination if a actually a technique in the compiler why is cheating?
Is maybe the autor that hate so much IE and dont know that technique?
Pingback: ‘Microsoft is hypocriet en speelt vals bij browser benchmarktest IE’ |
Pingback: Mozilla’s Rob Sayre Claims to Have Revealed More Internet Explorer 9 Benchmark Fraud From Microsoft | Techrights
Pingback: Microsoft News » Feature: Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?
Pingback: Tech Reviews » Feature: Lies, damned lies, and benchmarks: is IE9 cheating at SunSpider?
Pingback: IE9 Platform Preview 7: performance is the priority - TechRunner
Pingback: ¿Internet Explorer 9 hace trampa en las pruebas de SunSpider? | Logit42
Pingback: مايكروسوفت: نحن لم نغش في إختبارات جافا سكربت | تيدوز
Pingback: eamcet*[SEO対策調査自動更新ブログ] | IE9、SunSpiderベンチマークでズルしてる？
Pingback: Internet Explorer 9 Test Drive | artiss
lisa

Microsoft has a decades long history of unethical distortions in comparisons and benchmarking. It would come as no surprise if this were the case. All ethical developers need to be vigilant against MS propoganda since as long as they exist their corporate culture will no doubt continue to be the same. Better still switch to open source development. In addition, they have frequently tried to prevent publication of independent or Enterprise benchmarks of their products in their contractual Terms and Conditions.
- Michael Westen
  
  All that comes out of your mouth is bile !!
  As I figure, if you go through the entire comments hierarchy you should have understand that in Computer Science jargon, the dead-code detection and elimination is not the sister word for cheating. So for the conclusion, it is just that there are some false-positives in IE 9 platform preview 7 (which is NOT the final version to cry about !) while attempt to optimize Chakra.
  
  PS: this.article = cry_wolf;
http://www.twitter.com/abionic ABK

though cheating could be a reason… but seeing past massive records of code-screw-up-s… the Paralytic Logic seems more strong reason
Pingback: World Spinner
Simon

I seriously disagree with this assessment. Microsoft has far more superior performance analysis tools and benchmarks than this test and they don’t use it while optimizing their product. The only reason they even brought it up is so that browser techies can see how IE9 peforms in an already-established standards test. Likewise, Microsoft doesn’t use any of the Acid tests when developing their CSS engine (nor do they seem to care about the results), but they may link to it just to show a real-world rating of where their browser is. Face it .. no one would ever be this determined to find foul-play if it wasn’t Microsoft in front. IE9 has a lot of optimizations for common-place scenarios and it is totally feasible, demonstrably so in fact, that adding code can cause these optimizations to fail. Furthermore, if you think the word “return” doesn’t “do anything”, you are sadly in need of more information on how IE works, or even a basic understanding of how a stack functions in procedural programming and what happens when you push things (null, in this case) onto the stack.
Pingback: Schummelt Microsoft im Browser-Benchmark? | Dimension 2k
Pingback: IE9 Platform Preview 7: performance is the priority | Sonjag01's Blog
Pingback: IE9 y el Dead Code Elimination « Techno Blog
Pingback: Search For Video Deals Coupons and Reviews
Pingback: PuhujainKulma #180: No it’s just PuhKu | PuhujainKulma - viikottainen paneelikeskustelupodcast teknologiasta
Fff

@FeriCyde
“Wouldn’t be the first time that Microsoft’s benchmark statistics were called into question:
http://www.linuxtoday.com/news_story.php3?ltsn=1999-05-04-005-05-NW-LF

(link above from over 10 years ago).”

Regarding the above, look at the fate of the owner of the company who performed that Microsoft’s benchmark statistic, and evaluate the honesty of the people that MS choose to run their paid benchmarks

http://nbs.gmnews.com/news/2007-04-26/Front_Page/043.html
http://www.thefreelibrary.com/Mindcraft+Survey+Fiasco+Leaves+Microsoft+Looking+Silly.-a054480430
Pingback: Internet Explorer 9 beta, ¿son fiables los datos sobre su rendimiento? | tuexpertoIT.com
Pingback: ¿Internet Explorer 9 hace trampa en las pruebas de SunSpider?
Vinicius

Microsoft is right. The test is loaded with dead code. Please use another benchmark.
Pingback: مايكروسوفت: نحن لم نغش في إختبارات جافا سكربت | أخبار تقنية - الاعلامي مازن عماد غوكه
Pingback: RealTime - Questions: "Some say life is a game. Do you know any cheat codes I can use?"
Pingback: Sarkem.Net | All About Free Download
Pingback: free ringtones with no internet? | Uncategorized | Find Popular Ringtons Items Informatin on Ringtones Update!
Carl

I have written a Javascript-intensive HTML5 application and released it initially as a Chrome Web App on Google’s new Chrome Web Store.

After reading this article, I decided to use this app as an informal benchmark to test the various browsers’ Javascript engines. My very informal benchmark has two advantages over Sunspider:

1) None of the browser vendors is likely to have optimized for it, and
2) It actually does something, so there is no “dead code.”

This thing grinds through highly-optimized Javascript to compute every Scrabble word that can possibly be played to a given Scrabble board from a given rack of letters. This is a search problem, and so the amount of computational effort devoted to the interface is very small compared to the internal search to find all words that can be made by placing rack letters among the pre-existing board tiles.

This app, once loaded, runs entirely on the client, and so there is very little that will affect its performance other than raw Javascript execution speed.

One of my benchmarks is to place three specific words on the Scrabble board and then type a specific set of seven letters, including one blank (a wild card that can represent any letter in Scrabble), onto the rack. This generates over 10,000 words, which are stored internally and displayed only when the user requests them, one at a time (such requests are not part of the test). Here are rough elapsed wall times for various browsers, from fastest to slowest on a 64-bit Vista PC:

Safari 5.0.3 (7533.19.4): 5 seconds
Chrome Beta (9.0.597.19 beta): 6 seconds
Opera 11: 17 seconds
IE9 Beta 9.0.7930.16406: 26 seconds
Firefox 3.6.13: 45 seconds

To give IE9 credit for improvement, IE9 is now much faster than IE8, for this particular test. I can’t reliably quantify the amount of improvement, but I think it is at least 50 percent.

I must also list an important caveat: my Javascript was produced by the Google Web Toolkit compiler, and I believe that GWT is still in the process of optimizing its output for IE9, so it may be that there will be additional improvements for IE9 as a result of that.

As a developer, I could not be more pleased that IE9 is finally addressing its Javascript performance issues. I have heard that they are still working on this, and so I have high hopes for even better results before the first stable version is released.
Pingback: Top 10 “Breaking Stories” Of 2010 At Digitizor.com
Pingback: Google Sting Operation: Bing Caught Copying Search Results From Google!
http://twitter.com/betamos Didrik Nordström

I’m a web developer and have always hated Microsoft for their lack of a decent browser. However, if they want to make a decent browser instead they are in my opinion more than welcome. But when they cheat for benchmark points even before the release just to throw a hype, I’m not really confident about IE9 at all. It will probably be better but still a pain in the ass to develop for IE. Hope users are smart enough to swtich to Chrome/Firefox.
- Anonymous
  
  The bummer is that 95% of people out there really don’t care. I’m also a developer, and MS’ inability to create a browser that behaves predictably in any way puts me in the red several times a day. After messing with the IE9 beta and now RC versions, I’m experiencing rendering artifacts and layout glitches that I’ve never seen in any other IE (even IE6). What I’m eventually led to is web apps that work in all versions of all browsers, and now require IE9-specific code/styles to function properly. This is what IE9 was SUPPOSED TO FUCKING FIX, ASSHOLES. Arg.
Frazer Reg

============================================RESULTS (means and 95% confidence intervals)——————————————–Total: 16.4ms +/- 2.3%——————————————– math: 16.4ms +/- 2.3%
cordic: 1.0ms +/- 0.0%
cordic-with-return: 14.4ms +/- 2.6%
cordic-with-true: 1.0ms +/- 0.0%
Pingback: maweki.de › MSDN on Tour auf dem HTWK-Campus
Anonymous

Regarding the HTML5 tests, your thoughts mirror my own: http://www.arcreative.net/blog/2011/02/microsoft-being-dishonest-about-ie9-beta-performance/

I received and email from their IE team, you can see the comments on my post.
Pingback: El IE9 de 64-bits es notablemente más lento que el de 32-bits | HardwareMX
Pingback: HardwareMX – El IE9 de 64-bits es notablemente más lento que el de 32-bits
Pingback: IE9 Cheats! | Ivo's Blog
Pingback: HomeShop18 .com Teams Up With Microsoft, Brings Customers Closer To Their Favorite Online Store
Flammon

In IE’s favour…
Jose_X

This is so embarrassing for Microsoft. It is so easy to cheat on benchmarks. It’s in character with what this company does, but it’s still nice to get an example.

Just recently they were claiming they did a good job in supercomputing but just didn’t get chosen over Linux. A “bug” they called it.

Gaming benchmarks is the easiest thing to do. The skill is in doing it well so that it is a dash robust and not overdoing it since that will just incur suspicion.

“Everyone does it” is probably their only excuse, except that “everyone” can’t include open source because the cheating would thus be obvious.

Anyway, optimizing for specific websites is fine but is not a reflection of overall capabilities (including security) of the product and algorithms. There is so much you can hide when you are closed source. They hide interop traps (“bugs”). They hide backdoor access (since as malware has shown for decades, backdoors are simply bugs exploitable for access and control).

I don’t trust or touch their proprietary garbage if I can help it.
M. Mouse

this “rumor” is backed by verifiable fact, including a link to the benchmarks which you can run for yourself.

when it’s that easy, YOU are the fool for calling the author ethically challenged.

but thanks…for the good thoughts. don’t give up, you were almost right this time.
topic

Please stay on topic. Are you challenging his work?
- if so.. Please publish your rebuttal.
- are you venting your emotions as a perceived insulted?
- Do you own MS stock or feel bleeding heart for them?
If so then please see item #1.
Bar Foo

I get it. You work for the MS Explorer team
nobody

Here you go, jackoff. Firefox results. Was it so hard to clicky click yourself?

============================================
RESULTS (means and 95% confidence intervals)
——————————————–
Total: 23.2ms +/- 1.9%
——————————————–

math: 23.2ms +/- 1.9%
cordic: 7.8ms +/- 3.9%
cordic-with-return: 7.8ms +/- 3.9%
cordic-with-true: 7.6ms +/- 4.9%
Sjabanjo

You are right that the most likely explanation is usually the right one. But your claim that “fishy numbers almost always indicate someone is trying to game the system” is wrong. Weird numbers from benchmarking are likely to be caused by dead code elimination, not by cheating.
Kevin

“Remember, the most likely explanation is usually the right one.”

What a wonderful tautology you have there. Yes, the most likely explanation *is* the most likely. Or were you trying to misapply a woefully twisted version of Occam’s Razor?
http://roy-t.nl Roy

Tbh, if you rename quake3.exe to quack3.exe you will probably see a slightly lower fps because the video card drivers try to detect known programs and change some settings to optimize for this. So your analogue is quite flawed .
http://rustb.lt The Rust Belt

Next time try to judge an article by something more than its headline and the first line. It helps in avoiding embarrassment.
bowlingb

Except that functionally similar dead code situations don’t get eliminated.
Ice_shack

A little testy aren’t we now!
Point is I shouldn’t have to clicky click.

Maybe you like reading articles that have no credibilty.
But then your just a nobody!
Brad

Wrong again.

The only verifiable fact is that IE9 performed worse on the modified test. The “WHY” of the question is purely speculative and therefore the headline as well as the authors favored opinion is highly judgmental and misleading.

The truth is the author has no damn clue “why”, and his bias does the rest of the talking. In the first update, it is admitted that the original headline was inappropriate, and that betrays the mindset of the author – who clearly believes MS has been caught stealing cookies from the cookie jar, even though nothing is “proven”.

We’ll know for certain soon enough.
Jose_X

No credibility you say, yet you can’t point to a reason why adjusting the test a slight amount leads to Microsoft having such a horrible performance.

Wow, in the real world, to think that if I adjust my webpage a dash it will create performance nightmares for IE users. Ouch.

Cheaters always want others to look the other way. “Tattletale,” they say.

Well, their piece of crap should be shown in the light because it is real people that will be cheated (consumers as well as all of those that have placed their business behind a greater quality product).

If this was a game, they’d have fouled out a long time ago.

Even in court, they got penalized tens of millions of dollars for repeatedly ignoring the judge’s order and trying to influence the jury illegally. [I think this was the i4i case, though not sure]

Microsoft is dirty and it disgusts me to think about patronizing them. Their whole closed source lock-in monopoly based business reeks. They have no sense of social obligation or respect for honesty. Nothing matters next to $$$. They flunk out of science and ethics class, and that is one reason why they so fear the open source light.
Jose_X

No, because a benchmark is supposed to represent general performance measures. That it doesn’t do so in practice just means that people writing closed source software can’t resist cheating for marketing purposes.
Jose_X

>> The “WHY” of the question is purely speculative and therefore the headline as well as the authors favored opinion is highly judgmental and misleading.

It is not highly judgmental. It is very reasonble to suspect gaming is going on to have something be so many times faster in a benchmark, have it fail so badly with such a small change to the benchmark, yet expect that this was a coincidental bug that does not affect the general case capabilities of the software.

Yes it is possible they have refactored some mathematics significantly, etc, but it is unlikely that this generalizes very much.

“Optimizing” to a benchmark is at least a little cheating (since benchmarks are not real-world scenarios) but becomes significant cheating if the optimizing doesn’t scale much at all to even similar benchmarks.

Anyway, the speculation is warranted. According to you though, this is likely an innocent mistake Microsoft will correct very soon. If that’s the case, then no prob, and it doesn’t take away from the reasonableness of the suspicion. Especially, since using closed source to hide gaming of benchmarks is a very real temptation for a for-profit business largely dependent on monopoly levers and finding a market it used to control (and important for their server aspirations) slipping away steadily.
Ice_shack

Hmm… Stick to the point.
I don’t believe I was defending IE. I was just blasting a poorly written article. As a web developer it is my responsibility to know everything I can about ALL browsers. I learn 90% from articles written.
If you want to bash Microsoft, go ahead, I don’t care, I use chrome when I am on my windows box.
Fact.. I was just pointing out an article written about IE and Firefox with no results for Firefox published. No information given as to how the results were compiled = No credibility.
Now there are two sets of results for Firefox in these posts that don’t even come close to matching.