DrunkenBlog

Web browser benchmark

Someone not too long ago woke up and said, "I wonder just how many people I can piss off in one day, in how many ways." In other words, they did some benchmarks of various web browsers on multiple platforms, including Mac OS X, Linux and Windows.

Even under the most stringent and clean conditions this is just asking for trouble, as caveats and clauses get up in arbitrary ways, but you have to give the guy props for trying to do it in such a thorough way. Someone gave up some serious time.

This clip especially stuck out at me, and is guaranteed to cause a fuss depending on who it seems to help out more:

Hardware; 400 MHz G4, 256 MB RAM. Supposedly the Harvard architecture of the G4 chip makes this approximately equivalent to an 800 MHz Intel Pentium chip, and therefore this computer is of comparable spec to the one used to run Windows and Linux.

Thank god Java and Flash weren't tested, or those figures would have freaked people out more than they already will. Again on the Mac side, I was surprised to see Safari 2.0 included, considering that it's not out yet... someone's gonna get sued.

It pretty much backs up everything I've seen, including the currently horrific state of scripting in Safari, as well as the improvements I've heard mentioned by just about every Mac developer having to make use of WebKit in some way. Everything from AdiumX to Fire to your RSS app of choice will see a huge benefit. Look forward to it, and hope to hell the rumor that Safari/WebKit will be backported to at least 10.3 is true.

I mentioned those caveats when it comes to benchmarks, and there are going to be quite a few here, especially for OS X. It doesn't mean that these benchmarks are invalid; they're not. They're just a benchmark, for these set of circumstances, and I know someone is going to bring up the Firefox builds optimized for the G4 and such. I wouldn't really look to that, there are more basic things going on that could trip it all up.

Aside from JavaScript, which nothing would help, there are some basic ways I'd want to see these benchmarks extended:

• Raise the RAM in the tested machines to at least 384MB, preferably 512MB. A porsche can only go so fast between red lights, and different engines are optimized for different requirements.
• Raise the complexity of some of the test samples drastically, with tons of nasty embedded tables and different compatibility crud. A large nested-table discussion, ala slashdot, or large forum thread should do the trick nicely. :)
• A majority of the web still isn't doing this as XHTML/CSS, and in general all the browsers aren't completely optimized for them. Testing full plain HTML 3.x to 4.x sites is still a really, really good idea. Not saying not to bench CSS sites, just saying to do both.
• Do do mixed tests. This means including sites that have bits of flash, animated gifs, JavaScript, and even applets. Even sites that are CSS but also have to swap in things like tables are good to look at. While the browser maker might not have complete control over any one thing, it's more real world, and that's primarily what people are going to care about... not SPEC scores.
• HTTPS is something that often isn't tested, yet is something we use more and more of, and there are major variances between browsers.
• Concurrent loads: what happens when you try to open 30 pages at once from a folder? If it's 30 tabs?
• By giving it more RAM, you give the browser headroom... but running it awhile, you test its legs and whether it'll trip over itself after a mile of running.* I'd really like to see tests take into account what happens when the browser has been loading things for an hour of web browsing. You could start to see shades of issues with the 30-tabs test I mentioned above, but there are some differences between engines that would really show up here.

* IE, Safari generally caches everything, including the entire DOM structure of the document, which means things are fast when you reload a page you've been to, or when you're going backwards or forwards, but RAM usage after a bit of browsing gets insane.

An engine like Firefox stores the files but rerenders the page, which means RAM usage starts out larger than Safari but starts to even out real quickly. These are the types of things you pick up in general use, but wouldn't notice with a quick benchmark.

From what I can tell this is a real problem with his benchmarks, at least on the Mac. He initially loads the file so that it will be cached, and discards whatever that first number was. The idea is that it's the best way to test the renderer itself and not its routines for pulling in data from the disk. Or something.

Not sure I'm down with that. Assuming you disabled things like Safari's DOM caching, you could get theoretically get away with that, otherwise you're not being real-world about what people will actually see when they first hit a site.

 Posted by drunkenbatman
    February 13, 2005, at 04:03 AM