DrunkenBlog

Gecko's unsightly ass

I've been spending some time playing with the new Firefox builds for Mac OS X, and I have to say I've been impressed. In grabbing the nightly build, I was able to have links sent to it from other apps finally open in a tab instead of a new window, something I've been whining about forever. Keyboard shortcut goodness is in too, and combined with the potential of it's extension system... Firefox is getting really, really solid on OS X.

It's so solid that I wanted to take a look at it in the context of Safari and the larger browser sphere as a whole. As a word of warning, this one is going to be long-ish, and has been banged out before my normal morning-after doses of B1 and hair of the dog. I.E., par for the course.

Going back, Apple was put into the position of needing a solid browser engine for the platform. While it's true the Mac version of Internet Explorer was long in the tooth and becoming a liability, there was more to it than that as prefaced by some Microsoft initiatives.

Basically, while MS was getting absolutely raked over the coals for integrating IE into the OS (and, in many cases it was warranted) the idea behind it was sound: having access to the web was becoming a ubiquitous, or expected, feature for consumers. Having an engine built-in was becoming expected by developers.

Remember, there was a time when your Mac didn't even come with a TCP/IP stack or email client, let alone a web browser. You actually had to buy that separate, or your ISP would license it and include it on the disks they gave you for getting connected.

Now that need has become mainstream: not everyone needs a TCP/IP stack, but a large enough amount of users do that users just expect an OS to wrap in that functionality.

Apple basically had a couple of options:

• License the Opera browser/engine
• Work out some licensing deal with MS
• Adopt the Gecko engine from the Mozilla group, and roll their own
• Go with Netscape/Mozilla as the standard browser
• Adopt Chimera the Camino browser, which was essentially the Gecko engine wrapped in a Cocoa interface
• Adopt the KHTML engine from the Konqueror project and roll their own

Something akin to adopting the Netscape/Mozilla browser (Firefox wasn't around at the time) would have given them an OK browser, but would have fallen up short in the other services they wanted to provide. They really needed a core included with the Operating System.

For reasons I'm not going to waste time on, this basically came down to the KHTML engine and the Gecko engine (if you email me about something like iCab, I'll beat you). KTHML won, for a variety of reasons, but the biggest one being the size and scope of the Mozilla project itself.

The Mozilla project was ambitious beyond all reason. Things that Apple is doing with Dashboard, and Microsoft is doing with XAML, were pioneered with the Mozilla project through XUL. To think of this simply, imagine it as using the browser engine to render the interface in a disconnected fashion. In the case of XUL and XAML it's a form of XML, in the case of 'Dashboard' it's HTML/CSS. Instead of just building the core of the engine, and then porting that to each platform with native chrome... why not just make the base portable, and render the interface with the engine itself.

What the Mozilla group set out to do was impressive and ambitious stuff, essentially creating an entirely new platform for cross-platform app creation. However, it produced a multi-pronged quagmire of code constantly being pulled in six different directions... I believe the technical definition for the situation is 'cluster-fuck'. A cluster-fuck of coolness, but still a cluster-fuck with the occasional actual product regurgitated out once in awhile.

A programmer just trying to dive in and make something useful out of it was going to facing one hell of a learning curve, let alone the effort to try to shoe-horn it into what Apple wanted. Even when you took out all of the intertwined crud by separating the browser from chat, email, etc., Mozilla wasn't specifically geared for a tiny footprint... that was supposed to come later. Much, much, much later. Well, someday.

And Apple didn't and doesn't have a ton of people working on WebCore, which put a lot of stars in the KHTML column when it came time to deciding which way to go. KHTML was reasonably standards-compliant, small in terms of footprint and reasonably quick to render. Think of this as trying to take the MS Word codebase to build something new out of it, and the codebase from a shareware editor. Chances are, you'll be up and running a lot faster with that little editor's code than MS Word's.

Of course when you have a codebase like MS Word, a lot of that 'crud and bloat' is actually doing something, in what you might think of as the 'long tail' of features and compatibility.

By 'long tail', let's assume you are graphing features on a chart in relation to how many users will be served by each one you add in. For something like Word, you might think of the spell checker as having high relevance to a majority of users. Having support for tables will have relevancy for a large number of users, but not quite as many as the spell checker.

The graph goes out and out like this:

...and it generally keeps going out and out until you get to features that are somewhat laughable.

And yes, I completely made the specific features and number of users up, but you should be able to get the idea, and the concept of the long tail is important in general. If you look at the chart, you can get the idea that while the vast majority of users are served by a few main features, if you add up the users using the 'esoteric' features they make a really sizable percentage of the whole.

This is why when you're using Word, you might go "Dear gawd this is bloated, why is feature x in a word processor?". It's in there because there are Word users out there using it, even though they are a small percentage of the whole user base. If the percentage of the base wanting it is too small, it probably will be left to a specialized editor to implement.

But because the base of Word is so large, for a feature not to have enough users to make it worthwhile to implement, it has to be a serious niche feature. And this is where marketshare rears it's ugly head, and what's known as 'opportunity cost' starts to come into play.

Basically, lets assume that a programmer spends 8 a day working on their program which has 10,000 users. If 100 people are asking for a specific feature, that's 1% of the user base.

The programmer is faced with a dilemma: if it spends time implementing the feature requested by that 1%, it's time spent not working on either features requested by a larger percentage of users, or brand new features that would appeal to a broader percentage of people in general.

To put it another way: if the programmer spends time implementing a feature that might get it 100 sales (that 1%), it's time it can't spend working on a feature that might get it 2,000 sales (say, 20%). If you assume the app costs $50, that is the difference between $5,000 and $100,000.

The developer has to sit down and figure out how much time a feature will take to implement (scarily, an equation like this is used for bugs too) versus how much they stand to gain, but also what they lose by not devoting that time elsewhere.

There are a lot of areas where this starts to get really complex, and downright weird:

• It's fiscally worth your while
  There's a big difference between talking about 1% of a base of 10k users and 1% of a base of one million users (100 versus 10k, respectively). If the upgrade to your app will cost $50, and you have 1% of a million users that will upgrade because of a specific feature, that's $500k, while 100 users would only net you $5k. A great programmer is $50-$120k depending on where you're located, so if it's only going to bring in $5k, a feature that is going to take real effort to get in isn't going to make the cut. But when you are talking about a million users, if one half of a half percent of them will upgrade, you're still talking $25k.
• You've picked all the low-hanging fruit
  Applications do reach a point of maturity until the Next Big Thing™ comes along, which is why software companies are always trying to create the Next Big Thing™. Unless you're able to create the Next Big Thing™, you can end up with an app that much of your core user base can sorta yawn at. At that point, you start finding every little feature you can add and seeing what it'll take to get it in, hoping that you can wear the user down by having a bunch of features that they are lukewarm about but when added up give them a significant reason to upgrade.
• The 'product' is part of something larger
  It's easy to say "I'm selling program x for y", but what if you're selling something which program x is a part of? An example might be an Operating System, which might include a whole slew of applications for the price. You have to start attaching value to them, or a share of importance, as compared to everything else in the product, and then give them appropriate resources based on that.

Once you start having some of these concepts in your head, it starts to get obvious why, as an end user, marketshare and the size of the base as a whole are really, really important.

Annoyed that the game you just bought for your Mac seems a lot slower than on the PC? Instead of just ripping on the company that ported it, realize that the company porting it was probably given a specific budget to work with by the company who owns the game, and that budget is arrived at by figuring out how many sales they're likely to generate. I can understand why, as a Mac user, it can be annoying.

However, it's simply not economically viable for the company to put the same amount of polish and optimization into the port relative to the PC version. This becomes even more of a burden when you realize that in most cases your average Mac is underpowered relative to your average PC, meaning you'd have to optimize above and beyond what was done for the PC version to get the Mac version close to the performance of the PC.

Luckily this usually works out in the end; when you're porting a game that is two years old, it's easier to find a larger percentage of the Mac base that can run it well without going above and beyond in optimization.

Flash is another example of this: Flash development sucks on the Mac, and most serious Flash guys have abandoned the platform. Being an end-user of Flash also sucks on the Mac, and it's a little embarrassing when a dually 2GHz G5 feels slower than a 500MHz Celeron when Flash is involved. It's gotten better over the last few updates, but it's still really, really bad and Flash has become ever more prevalent.

Most people have moved away from having Flash splash-pages, but as far as advertising goes, it's way too easy to hit a page with a few flash ads and watch your Mac flash to a crawl. Open a bunch of them in tabs and you'll feel it.

It's hard to fault Macromedia for this; they are offering a platform and service to the people they are trying to sell on Flash. They could discontinue the Flash player on the Mac altogether and just tell Apple to implement their own if they really want it, but Macromedia would like Flash to be ubiquitous. The more platforms that can view it, the easier it is to convince someone to use flash in their banner ads.

However, for a market at 2-3% marketshare, it doesn't mean Macromedia is going to spend a lot of resources actually optimizing the Mac version relative to the Windows version, as any extra resources being applied to the Mac version are resources being diverted from the Windows version.

Which brings us back to Safari. Safari needs a lot of work. You can use a browser like Shiira, and get the idea that there's a lot of room for optimization just within Safari itself (to be fair, Safari does more), or use something like Omniweb to get an idea of how far Safari has come. And yes, just ignore the "fastest browser on the Mac" bit on their page; if you believe anything Apple says in their marketing without real independent verification... well, caveat emptor people.

Safari has made some big strides since it was originally released. I remember not being able to even log into a site that used a self-signed certificate for encryption, and even the ones I could log into where so slow I would fire up other browsers.

It'd only allow four connections at all, so pity the fool who tried to open up a bunch of tabs at once and spent ages waiting for them to load because one of the sites was slow, and Safari was still trying to render three banner ads on one of them... or had four downloads going and then tried to open another site. It's performance in general has improved, as has compatibility with what you'll encounter on the web.

While there have been improvements, Safari has a long way to go. The idea that Safari is the fastest browser on the Mac at this point would be a hard topic to debate, if not downright laughable. For a 'simple' site, this might not be that noticeable. But the web is getting really rich, primarily because on a PC it's able to be very rich without the end user having their browser get choked and stalled. I won't even browse web forums with Safari, I have to switch over to Mozilla or Firefox for that. Compatibility is still a big issue.

I mentioned the 'crud and bloat' feeling you can get from certain apps, like say, Mozilla or MS Word. But the simple truth is that a lot of that crud comes about from years of playing whack-a-mole with compatibility and bug issues.

Thousands of fixes for little esoteric issues accumulating over time can give the appearance of crud. Which isn't to say that some significant re-factoring might not be in order, but rather that some of the crud is thousands upon thousands of little patches and bits of glue that make Mozilla or Firefox much more compatible with whatever the web throws at them.

Apple had to do most of the above mentioned improvements to Safari, and will also have to do most of the improvements to Safari and WebCore going forward. This might seem a little bit confusing, since Apple adopted KHTML. As a quick refresh: WebCore is Apple's name for what they've turned the KHTML engine into on the Mac; WebKit is the service on OS X which developers can plug into to render HTML/CSS/etc.

KHTML is primarily used by KDE, a Linux desktop environment, and more specifically used in it's Konqueror browser. When Apple took KTHML, the license required them to keep the source public. I.E., if Apple adds improvements to an area, they have to give them back to the KHTML guys, who can then incorporate them into KHTML for the benefit of others (via Konquorer).

Apple announced Safari, and released their changes back to the KHTML team, and the KHTML team was ecstatic. Their only real regret was the wish that Apple would have let them know what they were up to sooner, so they could better coordinate efforts. This was a serious win-win for the KHTML team, as before Safari came along Konqueror was the only browser out there using KHTML in a big way, and Apple putting it on their desktops increased it's 'base' in a huge way.

Apple also had every incentive to help improve the weak areas of KHTML, and more hands and eyes on the code needing that much work is always a good thing - building a browser engine that has to deal with all the things browsers have to deal with is a daunting task for anyone at this point. The benefit to the KHTML base shouldn't be understated; the simple act of an increased base going out and finding rendering problems and reporting them is in itself a big boon.

Unfortunately, some things have cropped up, and that big boon has become just kinda nice sometimes.

When Safari was announced, it was another catalyst towards getting the Mozilla group to smell the coffee and step out of the echo chamber for a bit, as everyone had just assumed Apple would be doing something with the Gecko engine. When they went with KHTML, and gave their reasoning behind the decision, the Mozilla group had to ask themselves a lot of questions... and they did so.

Mozilla is still the main trunk project, but the individual parts (browser, email, chat, kitchen sink) have been spun out to allow them to focus on what will make them excel, while being able to suck out the marrow of Mozilla whenever it's progressed enough. Firefox (browser) and Thunderbird (email) are the two standout examples.

Firefox is easily the fastest browser on Mac OS X, but if you haven't used it on a PC, especially Windows, you probably won't believe just how much faster and better put together it is. Linux is sort of the reference platform of Firefox and Thunderbird goodness, but the brunt of optimizations have taken place on the Windows side. It's made a serious, serious amount of noise.

One of the big benefits of testing with/using something based on Mozilla is that if it looked ok using Mozilla under Windows, you could 99.99999% sure it'd look great under Mozilla under Linux or Mac or FreeBSD... anything that had Mozilla. Firefox changes the equation, because people actually want to use the browser on the other platforms.

If you'll humor me for a moment while we explore that changing equation, we need to take stock of where the engines are. Wen it comes to browser engines, you basically have four credible varieties at this point:

• Gecko-based engines
  Used within anything from the Mozilla group, such as Netscape and Mozilla, and embedded within a host of spin-offs, like Firefox, Thunderbird, Camino and Netscape. Also embedded into Gnome, the *nix Desktop Environment.
• KHTML
  Used in Konqueror, Apple's Safari, and embedded within KDE, the other major *nix Desktop Environment.
• Opera
  On the desktop, it's used primarily on Windows machines where it has a minor share. Also available for Linux and OS X, with more people liking it on Linux than do on OS X. Has recently made inroads into licensing it's engine to others for embedding. This includes cell phones, and even web design software.
• Internet Explorer
  The old mainstay, used primarily as the default for the desktop on Windows machines, as well as being used in Windows in an embedded way (AOL, etc.), as well as embedded devices. The Mac version was based on a different engine, and has fallen out of favor and ceded much share. But in terms of overall share, it has an incredible piece of the pie. It owns the pie. This can't be understated: IE has something like 93% of the total market, while all the others, everything, splits the rest among themselves.

Internet Explorer and Opera are both closed-source projects, meaning the companies putting them out are completely responsible for them. In the case of Opera, they charge, either via your eyes (advertising) or your wallet. In the case of IE, Microsoft has to decide how many resources to throw at it, and does so.

When it comes to OSS projects, many of the contributions add up to more than the sum of their parts. I.E., while in many cases a lot of the optimizations built into Firefox are Windows-specific, a lot of it isn't and benefits those using it on Linux, and vice-versa. The exact same should be true of KHTML, considering both the KDE team and Apple are working on the source.

While I'm not sure of the specific point at which it happened, this stopped being true a good while ago. Apple has been a good OSS citizen and released their changes back to the KHTML team, but as it turns out, for all intents and purposes, the code has been forked into a separate project.

When a project 'forks', versus just being an implementation, it means the base code is used as a reference and you can go nuts shoving it into whatever goals you have. You don't have to worry that when your code might break something when it goes back into the trunk, you just have to worry about whether or not it'll break what you're doing.

Forking generally happens because the person doing the forking has goals that don't align with the original project, and they need to do things differently or add things that aren't relevant to the original project. Something like Firefox isn't a fork, but rather a branching off of the main trunk of code, which it eventually merges back and forth with.

While Apple gives the code for WebCore back, their code has forked from the main code base in such a large way that Apple's changes and fixes can't just be merged with the trunk, and vice-versa. This means if KHTML makes big strides via the KDE team, Apple is pretty much out of luck, and again, vice-versa. The 'greater than the sum of its parts' axiom no longer really applies.

Another interesting thing has happened, which was KDE bowing to pressure and doing a bunch of work to allow someone using the KDE Desktop Environment to choose to use the Gecko engine from within Konqueror, rather than KHTML.

There were reasons for this, but much of it goes back to the 'compatibility crud' that the Gecko engine brings. As far as I'm aware, it still defaults to KHTML, so I'm not writing KHTML off as far as the KDE platform is concerned. But it's no longer a given, and in the future KDE may decide it's just not worth their time to expend their finite resources developing a browser engine which, as was mentioned, are really hard to do.

Linux has also made huge strides on the desktop, adding a lot of share. While many of you will say, "Yeah, but a ton of those are corporate, government installations, etc.", for the purposes of browser share it doesn't really matter. While many sites care what you're browsing with at home, a huge amount care what you're browsing with while at work.

It's a fair question to wonder why it matters who is gaining share, and why it matters if things seem to be going the Gecko engine's way in a big way. As long as Microsoft is ceding share, it should promote 'standards-based' websites, with less reliance on Active-X and things specific to Windows.

There's a lot of truth to that, and standards-based sites are a very important ideal, and a saving grace. But as any website developer/designer knows, a site being coded to standards does not mean it'll render correctly and look the same across all browsers, and Safari is not the most compatible browser in the world, not by a long-shot.

Its full open-standards support at this point is an ideal, not a reality, let alone that now-fondly-considered-compatability-crud that takes ages of bug repots and developer time to diagnose and then fix. And Apple will be doing it all.

Mac users don't enjoy hearing this, but while it's very easy for them to get on the web, they've been second-class citizens for years. Honestly, the introduction of Internet Explorer 5 was the high point for the Mac, everything else since then has been trying to catch up.

Yes, Apple has some neat technology in their Java implementation, but because of the speed of 95% of the Mac hardware base and the insane amounts of optimizations Microsoft did to theirs, something that loads quickly on Windows can make your Mac browser slow to a crawl, if it doesn't outright crash.

While Java applets should be cross-platform, way too many sites rely on Windows-specific technologies to allow applets to communicate with other parts, meaning Mac users are often out of luck or just see downright weird behavior in way too many sites, no matter what browser they are using.

While performance has improved on the Mac, the others have started to widen the gap even further. Apple's marketing primarily shows off Safari's performance compared to Internet Explorer 5 for the Mac, an application that was released five years ago. It's simply not a realistic impression of where the market really is. And while Apple can close the gap (not having animated gifs play in a hidden tab would be a start), they'll be doing it and expending all the resources, rather than building on and using OSS as a 'force multiplier'.

We've covered Flash indirectly.

Compatibility is still a major problem on the Mac, and even though it's improved greatly with Safari, way too many Mac users have to open other browsers in order to do their banking or other tasks. Yes, lots of times sites will lock you out from areas 'needlessly' by checking what browser you're using, when if they'd just let you in, you'd be able to use it just fine. But many of them, when you have Safari 'spoof' what it actually is and you get in, it still don't work.

You're often 'locked out', because the people who built the site don't often don't feel it's worth their resources to test against everything out there. It doesn't mean it won't work, they just can't say with a certainty that it will work and to deal with whatever support hassles may come. Yes, you can vote with your wallet, but in many cases they still won't care because as we've covered, the equation just doesn't make it worth their while. Frustrating, but realistic.

Right now, the old mainstays that are tested are Internet Explorer variants and, if you're lucky, Netscape 4.x. In most cases, it's not the Gecko-based Netscape versions, but this will probably change soon if Firefox keeps building up steam, especially on the corporate desktop where various forms of malware are taking a vicious toll. And anything based on the Gecko engine will benefit.

As 'Gecko share' grows, it'll be tested for, and considerations made on the back-end of sites. These may be standards-based considerations, or it may just be testing and seeing where a slight Javascript tweak will make it work, but at least they'll know if users using Netscape or Firefox will actually be able to make use of the site.

And remember, Gecko-share is starting to become larger than the sum of it's parts: if you're using Camino, Mozilla, Netscape, Firefox, Thunderbird, etc. you're part of the club, no matter what the specific platform. Firefox on Windows, Firefox on Mac, Firefox on Linux... it all adds up to a much larger wave. Safari is just... Safari on Mac.

This does not mean testing won't happen with Safari, just that the odds of it being a platform that it tested for are becoming smaller and smaller.

Hell, it's actually really difficult for a non-Mac web designer to have any idea how to even know what their site is doing in Safari, as getting Konqueror up and running in WindowsXP is a little above and beyond where many are able to go, and with the divergence in the code bases you still can't trust that what displays OK in Konqueror will display OK in Safari, or vice-versa.

And most of them aren't going to buy a Mac to test on. Hell, I've been surprised to see a few of them pop up on lists asking for a Mac user to take a screenshot and send it to them. They're put into an awkward position: they have to justify testing and sometimes development for a platform with a small and continually shrinking share, while they're already going to be testing for Gecko.

It's just the broadest swath: test for Gecko, you get the broadest amount of users across platforms. In other words, get used to seeing "IE v.x and Netscape v.x or Gecko-variants supported". This might be a good time to find the nearest Camino developer and buy them a beer.

Apple seems to have gotten some whiff of this, as they've decided they're going to back-port the 10.4 version of Safari for 10.3, which is a good thing as making site owners test for not only a browser on a small platform, but multiple versions of that browser split across those few percentage points of share is asking a lot.

But wait, you say, if the site is building with standards-compliance in mind, this again can only benefit Safari. If that's what happens, that's again a very important point: but implementing all of those standards, and all of that crud, and all of that testing... again falls on the shoulders of Apple.

Apple then has to devote a portion of the resources allocated to the development of Mac OS X to it, instead of just riding the Gecko's tail and putting that work towards Mac-specific refinements of it. This is assuming they'll be willing to divert those kinds of resources towards Safari instead of other parts of Mac OS X, instead of going with "good enough" while staring at the Gecko's ass ahead of them as it gets smaller and smaller.

The web is shaking down to Gecko and IE across multiple platforms for the foreseeable future, with Opera as a niche, and Safari as an even smaller niche. The KHTML code basically got Apple running faster, but now they're on their own again: marginalized due to share.

I am not saying Apple made the wrong decision by going with KHTML over Gecko, and I'm not in any way calling out the Safari team. They're doing great work, but there are only so many of them.

What I am saying is there are trade-offs in everything, and while Apple gained much by choosing KHTML over Gecko, the things that have been traded away may become much more important down the road.

When it comes to something as intrinsic to computing as the web and internet have become, I've grown weary of staring at the asses in front of me by using Mac OS X, and when you're not leading the view in front of you never changes.

The Lizard's posterior has broken the monotony of IE's, but I'd rather be riding its tail.

 Posted by drunkenbatman
    November 08, 2004, at 07:00 AM