At about the same time, Adrian Kingsley-Hughes at ZDNet posted on some questions around some anomalies in the IE9 SunSpider results, having picked up on an accusatory post over at the Digitizor, which in turn took a fairly innocent paragraph in Mozilla engineer Rob Sayre’s post on SunSpider results for Firefox
One last issue that can crop up has to do with over-specialization for a specific test. While I was running the SunSpider tests above, I noticed that IE9 got a score that was at least 10x faster than every other browser on SunSpider’s math-cordic test. That would be an impressive result, but it doesn’t seem to hold up in the presence of minor variations. I made a few variations on the test: one with an extra "true;" statement (diff), and one with a "return;" statement (diff). You can run those two tests along with the original math-cordic.js file here.
All three tests should return approximately the same timing results, so a result like the one pictured above would indicate a problem of some sort.
After a flurry of Twitter retweets, an appearance on Slashdot, some on-the-fly blog post title editing to calm down on the “cheating” allegations, and even some backlash from within ZDNet, a number of bloggers posted their views:
- Lies, damned lies, and benchmarks- is IE9 cheating at SunSpider- Ars Technica
- Internet Explorer 9 Is Not Cheating on SunSpider Benchmark OS News
- Microsoft’s new IE9 triggers speed-test squabble CNET News
However, Rob Sayre responded in turn to that update, giving another in depth analysis of our new favorite term, “dead code elimination”. In the post (and we wonder if we, or the IE team, are the “beginners” he refers to in his blog post title ) he turns the tables back on IE:
Curiously, the IE9 team has chosen not to eliminate more expensive operations such as multiplication or division. The set of analyzed operators do happen to be the ones that appear in the math-cordic benchmark.
We’re interested to hear more about IE and SunSpider, but if Sayre’s latest post is correct, IE still has some explaining to do.