By Scott M. Fulton, III,Betanews,03/27/10 5:00 AM PT
There are a lot of ways to test a Web browser, and also a lot of ways to compare the results. How important the score in a given area is depends on what kind of Web work the user wants to do. The test preview of IE9 that Microsoft has given to developers scored 8.75 on the scalability scale, meaning its Chakra JavaScript engine performs nearly 9 times better than IE7 when it comes to scaling heavy workloads.
How long ago would you have thought it absolutely impossible for the slowest Windows Web browser currently under development to be coming ... from Mozilla? Granted, the Internet Explorer 9 Tech Preview isn't a real browser (typically, these things need their own address bars and Back buttons). But unless Mozilla gets its JaegerMonkeys in a row in time for Microsoft (Nasdaq: MSFT) to debut IE9 with real features like buttons, the number two reason users cite for switching from Internet Explorer ... will be wiped off the map.
In the most sophisticated system Manage and monitor your systems with Landscape for Ubuntu. Free 60 day Trial. of browser tests ever developed -- reconstructed by Betanews in anticipation of the IE9 preview last week -- IE9 in Windows 7 registered a comprehensive index score of 13.17, representing over 13 times the performance of IE7 in Vista SP2. By comparison, IE8 in Windows 7 scored a mere 2.20, representing about six times the performance of Microsoft's current production browser. That's down from our preliminary estimate from last week, but still a very commendable performance gain. Typically, when developers add real features to their browser projects, that tends to slow down overall JavaScript performance. But that doesn't mean Microsoft won't continue to compensate as they improve their own new JavaScript engine, code-named "Chakra."
Last week's latest daily preview build of Firefox 3.7 Alpha 4, meanwhile, scored a 10.76, using the same tests on the same machine. The new round of Alpha 4 previews represent Mozilla's fastest browser to date, well ahead of the current Firefox stable browser score of 9.08.
That's not the only headline to emerge from the latest tests: Although it had appeared to be holding back in recent weeks, Google's (Nasdaq: GOOG) Chrome 5 team managed to find another gear (what would you call it? Eighth gear? Ninth?). On both our old tests and the new, Chrome snatched back a handful of speed points, to post a total score of 23.32. The latest stable Opera 10.51, released just recently -- including bug fixes to the hurriedly released Opera 10.5 -- scored a 23.17 on our new tests, still a big improvement over the stable version score of 21.85. The stable Chrome 4 scores, by comparison, are behind Opera's at 20.05.
Why a New Test Suite, Again?
As Web "browsers" evolve to become Web applications platforms, and as Web "pages" evolve to become applications, it becomes more and more critical for us to understand the differences between the browsers as though they were machines. Readers have told me recently that it might be unfair to keep comparing IE to Chrome, for instance, because (in their words) folks tend to use IE just to browse pages, whereas they may be using Chrome to run Google Apps. For those readers, continuing to declare Google 20 times greater, or so, than IE is like saying over and over again, a tractor's more powerful than a lawnmower. Sure it is. We get it already. But that's not to say lawnmowers don't have their place.
The average BlackBerry has a far slower processor than the common iPhone. That fact is obvious whenever you zoom in and out of a page (the word "zoom" doesn't really apply to most BlackBerrys). And up to now, the standard browser on a BlackBerry is, in my totally unbiased opinion, terrible. (If you're like me, you've replaced it with Opera Mini.) But that doesn't mean the BlackBerry is useless or even inefficient for what it is capable of doing, when it does it well.
Efficiency, for me, is the capability to find another gear and crank out greater work product when the workload increases. You hear marketing Free Report - Discover the Difference of Email Marketing 2.0! folks misuse the word scalability; to me, it's the capability to get more efficient as work gets tougher. Theoretically, if a processor has to do a job x 100 times, you can expect time consumed to grow to 10x if the workload increases to 1,000. If it becomes 15x instead, that's bad.
Modern Web browsers like Firefox utilize just-in-time (JIT) compilers that look ahead through the batch of upcoming JavaScript instructions, to break down jobs into more efficient, more digestible work units. Theoretically, that means when the workload increases to 1,000, time consumed should be more like 7x or 8x. That's the type of scalability I want, and now expect, to see from a modern browser -- more so from a development build of Internet Explorer now than ever before.
When Opera Software last month told me its developers' opinion of the relative efficiency of one of the tests I had been using in our Relative Performance Index suite, I decided to pursue whether they were right. They were. Months earlier, I had resurrected an old test battery used by magazines in the Netscape days, which spun a single instruction a few thousand times and measured time elapsed. Well, in modern days, when a single instruction does nothing, and a thousand or a million repetitions of that instruction do nothing, just-in-time compilers see that it does nothing and, quite efficiently, "compile" that instruction to ... nothing. So when it takes no time at all to do nothing, I frankly shouldn't be all that amazed.
That's when I became more curious about the way JIT compilers work. If the "digestibility" of a sequence of instructions depends on its sameness, then a more appropriate test of a JIT's efficiency would be to throw different algorithms at it whose relative efficiency sometimes depends on its capability to be differentiated rather than the same, throw varying workloads of the same test at it (100, 1,000, 10,000, 10 million iterations), and see how browsers perform under the stress. Will they scale up to meet new demands? Will they opt for easier breakdowns or faster run time?
That's the inspiration behind the latest battery of tests in the Betanews suite: a way to see not only how fast a browser runs and how fast it can become, but why. Back in the 1980s and '90s when I used to test BASIC compilers, I used some common reiterative algorithms and math tests, and I published the results under my old pseudonym, "D. F. Scott." So in honor of my past life, I've christened this new battery "DFScale," a test of varying speeds and scalability under varying conditions.
The IE7 Baseline
The reason Opera 10.5 went from zero to hero as fast as it did was because of how well its JIT compiler breaks down instructions at greater and greater workloads. The reason Internet Explorer 8 cannot catch a break is because it does not scale workloads as well as IE7. In fact, simply overcoming that problem is why IE9 has come so far.
As with the other tests in our suite, we compare our results against Internet Explorer 7 as a baseline, as a way of extracting the relative speed of the machine from consideration. When something scores a 2.00, for example, that's our way of saying it's double the performance of IE7, and it would be double no matter what machine you tested it on.
Everybody who has used IE7 knows what "the speed of IE7" means, generally speaking. But "the scalability of IE7," or any other browser, is not something you can easily see, so drawing a comparison between it and any other browser might not make immediate sense. So here's a general rundown of what I learned: In solving problems that tend to scale linearly as workload increases linearly, IE7's scalability is surprisingly not bad at all. For example, using the algorithm Microsoft used to fix its widely reported choice screen randomization bug last month (yes, the new "shuffler" is indeed one of the algorithms we decided to use), IE7 starts out by shuffling a 250-unit array at about 125,000 units per second. When we change the array to 250,000 units of unique values, the estimated speed is closer to 195,000 units per second. And that's actually very good.
For mathematical problems that become exponentially more complex as the workload grows linearly, IE7 struggles. There are two ways of expressing an algorithm for finding the first numbers in the Fibonacci sequence: one which is compact and easy on the programmer, and another which is more drawn out but easier on the processor. For the compact version, for the first 20 numbers in the sequence, IE7 runs at about 132 iterations per second. Increase the workload to just 30, and the run time slows to just 1.5 iterations per second. Make it 35, and IE7 hangs interminably.
IE8 is only marginally more scalable overall than IE7, partly because in some instances, it's actually a lot worse. The surprise here is, the simpler and more linear the algorithm, the poorer IE8 is at finding that extra gear when it needs it. For simple reiterative problems such as the Sieve of Eratosthenes and a wonderful discovery dubbed "Euler's Problem #14," IE8 can actually become slower than IE7 over time -- its scalability goes way down, below the 1.0 mark. So if you can picture IE7's scalability as 1.0, IE8's is just 1.31.
By comparison, the IE9 Tech Preview issued last week posted a scalability score of 8.75. That means, by our estimate, the "Chakra" JavaScript engine does a nearly 9 times better job of scaling heavier workloads than IE7. Compare that to a 4.47 scalability score for the daily private build of Firefox Alpha 4, and you see some evidence of Microsoft's claim that managing Windows' background/foreground process timing can be more efficient than using tracing and JIT compilation. (Of course, now that this particular cat is out of the bag, imagine what Mozilla could do with it.)
The New Categorical Breakdown
Relative performance of Web browsers in Windows 7 by category, March 22, 2010.
What our new test suite also enables us to do, that we couldn't do before is give you a single graph (rather than 10) that breaks down the browsers' performance by category. Here, our final score is broken down into three groups: computational speed (33 percent), rendering speed (34 percent), and scalability (33 percent).
Just last week, the Chrome 5 development build was posting speed scores that were lower than those of the stable Chrome 4. The score from last Friday's dev build 356.2 (which we verified by running the whole suite two more times completely on a rebooted system) gained back about 14 points in speed alone. One has to wonder just what it was that Google was holding back.
The latest Opera 10.51 is the fastest rendering Web browser in the field, at 19.62; here Chrome 5's scores actually dive below not only Chrome 4 but Safari. While I was devising these tests, the Chrome 5 dev build at the time was posting scalability scores that were not as good as Chrome 4's. Now they're well ahead at 10.93 versus Chrome 4 at 8.25, and Opera 10.51 at 9.47.
Oh, yeah, Safari! Lost amid all this talk about everyone else is the fact that the new stable Safari from Apple (Nasdaq: AAPL) is faster and more scalable, and also that it's a better platform for the daily WebKit development builds. WebKit scores had been suffering, but joined with the latest Safari 4.0.5, they've improved dramatically, now with better rendering scores than either the stable or dev build of Chrome.
Ladies and gentlemen, welcome to the five-way shootout. With Opera and now IE officially in the hunt for supremacy, the Web browser battle may as well follow Joe Bob Briggs' first rule of horror pictures: Anybody can die at any moment.
© 2010 Betanews. All rights reserved.
© 2010 ECT News Network. All rights reserved.
