Suffer the reviewers
I was chewing through AnandTech's No More Apple Mysteries, Part Two which was making my happy place all tingly. Now, lots of things make my utility belt tingle, but if you're not aware, Part 1 of their article focused on benchmarking Mac OS X for many server related tasks (I.E., Apache, MySQL, etc.) and came up with some abysmal results for the Mac compared to other platforms.
It caused a hell of a lot of smack to be thrown around -- primarily at them -- with only about 5% of it being deserved. There were all sorts of theories being thrown around as to why the results might have been showing so badly in OS X's favor, especially when it came to databases, all of which clearly ignored...
- Apple had all, in not so many words, told the AnandTech guys the issue they were seeing was real.
- The issue itself -- OS X being slower by an order of magnitude at certain types of server-related tasks -- was well known to a lot of people.
I think one of the reasons why the AnandTech guys caught so much flak was that users just weren't used to seeing number two thrown about, primarily because these types of real comparisons of OS X are so rare, and they were seeing it at a site primarily devoted to PC's, no less. This is compounded by the fact that Apple's server share just can't get off the ground, so few are really putting it through its paces.
About little less than a year ago, I did an interview with Pieter Van den Abeele of Gentoo Linux for PowerPC and basically asked him what someone would get by running Linux on Mac hardware instead of OS X:
For some people OS X or Darwin might be too slow or bloated. Companies wanting to squeeze every last bit of performance out of their server farm might choose to install a customized Operating System to do the job. Boeing for instance used a G5 based Gentoo Linux solution for its satellite software.
Pieter got some flak for that around the web -- people in particular seemed to object to the use of the term "bloated" -- but what he was saying was sound. With these types of things, you're probably only going to find the issues if you're actually comparing hardware and OS configurations to find what works best for what you want to do, rather than looking for an excuse to get a Mac in the door.
We know Apple sold a bunch of XServes to the Navy to be embedded inside a submarine, and we also know they chose to use Yellow Dog Linux instead of Mac OS X. Obviously we don't know the story behind that, but we can assume there was something about the hardware that rubbed them the right way for what they wanted to do -- most probably something to do with AltiVec -- but that something about the OS wasn't working for them in the same way Linux was.
There are other areas in OS X where there are similar types of bottlenecks. Back in the Red Shed chat, Rentz said:
Cocoa is kind of weird concerning threading. It supports multiple threads, but the first thread you spawn, Cocoa takes notice and sends out an application-wide notification that the app has gone multithreaded. If your app was a Star Trek episode, it would be the Klingon ship decloaking right off starboard bow with a system-wide RED ALERT and raising of shields.You also can't use most of Cocoa from nonmain threads (the "main thread" is the thread that handles events like mouse clicks and key presses). It's really not bad, as it's typically not the UI you want to thread, it's calculations and I/O.
Both of those are points which one could make an entire interview about, because they do affect us in our day to day use of apps, but to my knowledge no one has really sat down and explained to a Casual the implications of the above. Developers just work around it as best they can, as they have little other choice, and it'll take a lot of work from Apple to improve it.
Basically, these things are real, you just won't find them in a marketing brochure and they're complex enough that few want to tackle explaining to Casuals what's really going on.
If you think about it, the most thorough reviews of OS X to date haven't come from the Mac side, but rather the PC side dipping a toe into the Mac world, but even then there are limitations. It isn't as though OS X server hasn't been reviewed around the web, but people's time costs money, so they're generally fairly shallow, even at places that often do solid work.
It amounts to plugging it in and starting it up and playing around with it in a test network, while getting a feel for it and making sure it does some of the things it says on the box. Remember, the more time spent on testing something, the more costly the final review which has to be weighed against how much cash they're bringing in from doing it. The types of things AnandTech is trying to do actually take a decent amount of time, and they should be encouraged.
Another problem is that the reviewers are rarely people who are using it day-in-day-out, which means they're going to miss the things that are going to crop up with OS X Server. I remember reading reviews of it that just didn't mention what was really going on in the trenches with it: Apple probably could have been sued over the quality of the eMail server they were releasing. (Note: The Mail Server has improved)
However, you'd only really find out these things after using it for awhile, and the majority of the people who have been using it for awhile are pretty worthless when it comes to asking what they're seeing, because they're using OS X Server because they want to use a Mac.
You might think that's hyperbole, but Apple has barely been able to make a dent in the server market whatsoever. They've had some successes in the high performance computing market (HPC), where someone wants to hook up a bunch of Macs in a cluster for calculations, but the dirty secret about these is they're more about PR than actual unit sales and segment growth. We don't know actual sales of the XServes, because Apple lumps them in with the PowerMac numbers, but we know that number keeps going down and down.
The place to look for people who are primarily using them are pro-Mac shops that in the past just had no choice but to use something from Windows or Unix. Not long ago, you'd walk into a design house or pre-press shop that would have Macs on every desk -- even the secretary -- but a big Unix or WindowsNT server feeding all of them, because:
- Apple just didn't make hardware that was server-ish whatsoever, they were basically PowerMacs with extra software installed. This was fine unless you needed certain things big iron can provide, or wanted to throw them in a rack for better management.
- The Classic Mac OS was killing them in this market. Running OS 9 on a dual-CPU PowerMac as a server wasn't going to do you much good, and even if you were just needing to run FileMaker server the performance differences (and stability) was going to be huge.
These people just had little choice but to cede the sale towards a Windows or Unix-based solution, but with the XServe there's at least something the faithful can buy if they want to keep it all-Apple. As always, this isn't the whole story, but these people are tantamount to worthless when it comes to getting a handle on what they're really dealing with, because they chose a Mac because they want to use a Mac and wouldn't really consider something else.
They'll pull their hair out for weeks straight on something sub-par in OS X Server, and their shop has lost huge amounts of time because of it, but when asked by someone else how its gone for them they'll recommend it whole-heartedly and mentally gloss over just what they've gone through.
Let me put it this way: If you are trying to find a place to co-locate a Mac Mini as a server, you either have incredibly specific needs or you're one of the people I'm talking about... When all you have is a hammer, everything looks like a nail.
If you can't tell, I'm a little put out at the level of thoroughness of some of the reviews of different products out there. It's not just an Apple thing, it's just very prevalent in the Apple camp. You have a problem where the publications that are left on the Mac with the resources to do thorough reviews don't have a base that would be really interested, so the reviews are basically rewritten marketing materials about what's new.
There's also the problem of Apple often being the largest advertiser in these publications, and pissing Apple off in a major way is tantamount to seppuku. Apple dropping their ad buy by 40% for a few months because a review decided something had major problems would hurt the publication a lot more than Apple, so it's generally 4 out of 5 stars across the board, and you get a 4 instead of a 5 if the computer has a habit of frying its logic board.
Note I'm not calling the publications out about this, they're trying to survive and pay the mortgage and all that, and this goes across all industries in some form. The gaming industry would be a good example, although they're generally at the far end of the scale and approaching corruption. It's just a little larger than it should be on the mac, because everything is so single-source: Apple.
AnandTech is going to catch flack for their article I'm sure, but on the whole I'm just glad someone is putting in the time while being upfront about what they're seeing and trying to figure out what's going on. If you're upset at what they're doing, the least you can do is ask yourself whether you're more upset at what they're finding (and showing) or how they're doing it.
The latter can be rectified if you're willing to give them feedback, the former is... well it is what it is. At the end, they're trying to give information that wasn't out there in a readily widespread way before (But is on MySQL lists and such), and trying to get to the bottom of it. That's cool.
In Tiger, the locking is finer. Although Apple's documents indicate that it is still rather coarse grained, it is clear that more than two locks into the kernel can exist at the same time. In the case of MySQL, this should be a very important improvement, but we didn't see any improvement at all when performing the tests on both Panther and Tiger. This is speculation, but based on our data, we are tempted to hypothesize that the new locking system isn't really working right now, and that Tiger continues to behave like Panther.
The way the above is worded is going to cause the AnandTech guys headaches. The funnel situation has been improved for OS X, as someone can see by say, saving out a Snapz Pro movie or QuickTime export while trying to do other things in Panther or Tiger. In Tiger, it actually feels like a multitasking OS.
This doesn't mean its perfect, and isn't rough and shouldn't be looked at as something contributing to the problem. Obviously there's a huge difference between one process completely blocking anything from doing I/O, and two things trying to access, and twenty things trying to access, but there has been improvement here.
Mac OS has a lot of strengths, even when it comes to speed sometimes. I.E., there are some good APIs in CoreAudio now that can give you good sustained low-latency support for Real Time-ish audio work, where it stands up great to the other players. The types of things being shown here are just an area where it's very weak, and there's a reason why Apple doesn't have a lot of benchmarks for OS X doing server related tasks up on the web.
Sure, they did back when the G5 came out, but if you'll notice much of them were compared to their older designs, and related to showing off the speed of their new DMA (direct memory access), which allowed the CPU to offload a lot of tasks to other chips for file-serving related things. There's a huge difference between serving static pages, or even dynamic pages, with serving pages that have to make a bunch of database hits at the same time.
It can be improved over time in some areas, but many of them are just things that have to be lived with due to design decisions in OS X, like having the Mach layer, and those aren't things that are going to go away. Mach brings some things to the table, but much of the way OS X works now is because it's "good enough" to get it out the door now, and changing it would be more than Apple could stomach in development costs while getting anything out the door in a reasonable time frame.
Posted by drunkenbatmanSeptember 02, 2005, at 08:57 PM
Comments (34)
Posted by: David Sanchez at September 2, 2005 10:15 PM
This sentence really captured my attention:
"Mach brings some things to the table, but much of the way OS X works now is because it's "good enough" to get it out the door now, and changing it would be more than Apple could stomach in development costs while getting anything out the door in a reasonable time frame."
I agree with it. Mac OS X when it went out was not a finished product, and even today it is not.
I know many people complaining about Mac OS X kernel design because of its speed. I would like to see a clear explanation why Apple chose such a complex design. I cannot believe an originally one-man project, like Linux, could beat up an army of engineers at Apple.
Even the BeOS kernel with its TCP stack seems faster than XNU after 5 years (BeOS kernel is POSIX compliant too). Is it so hard to make a better/faster kernel? Linus Torvalds is not such a genius and even if he were, anyone can see his design (It is open software).
There has to be some things XNU is good for and better than others, and if it is not, it should be replaced with something better. OS X is designed with modularity in mind, it should not be so difficult and if it is, then Apple is the one to blame.
The Intel transition is going to place a very big zoom on Apple OS design, since it is going to be the main player getting new customers. We are going to see more reviews like the Anandtech's. Mac hardware cannot be dramatically different from x86 PCs so the magnifying glass over the operating system is gonna be huge. This flaw in XNU design could hurt Apple in many ways, especially since SMP machines get more common and programmers want to exploit all the hardware possibilities.
The "Good enough" part was accepted 5 years ago, but as days go by, users and programmers are expecting Mac OS X speed problems gone.
It seems the XNU design is too complex, bloated, unrealistic or Apple is just not paying attention to it.
Posted by: Engrish at September 2, 2005 10:37 PM
We don't know actual sales of the XServes, because Apple lumps them in with the PowerMac numbers, but we know that number keeps going down and down.
You don't know the actual numbers, but you know they're going down. WTF ? :-)
We know some of the numbers, they're far from stellar and going up and down, and then up and down again. T,FTFY.
Posted by: Carl at September 2, 2005 11:06 PM
The word is "hara-kiri," and it's pronounced as spelled, not like the baseball announcer. Of course, Japanese people usually just say seppuku anyway…
Posted by: Jay Tuley at September 2, 2005 11:06 PM
Not to nit pick, but the cocoa threading comment seems out of place. Yes cocoa threadings not the greatest, but it doesn't seem to relate to slowness. The fact that cocoa gui classes aren't thread safe should theoretically improve performance rather than contributes to any slowness. Just as Rentz said, normally you don't want multiple threads acting on the gui, so why have extra code doing extra work just to make it safe when everyones going to be running it in one thread anyway. It's the same for the red-alert part, the cocoa frameworks are reducing their overhead to theoretically run faster if they know they are one thread only. It's weird but it fits in to the design philosophy you see every where in objective-c which was designed to run on much slower computers with little ram and do only what's the bare minimum that's necessary do the rest in C. Basically its just that cocoa is still living in the 1980's. My biggest problem with cocoa threading is that NSThread is too bare bones of a wrapper on pthreads, it doesn't offer any high level apis for thread management. Wow, one instance method that's barely useful and a couple of basic functions that can only effect the thread they are running in.
Posted by: robert at September 2, 2005 11:27 PM
The anandtech article seemed to me to be more about trying to answer the hardware side of the question-just how fast is the powerpc platform compared to the fastest x86-the first article did not succeed in doing that because it got sidetracked by the mysql performance difference, which they sort of addressed here. My take on the article was that they demonstrated that powerpc was kicking butt in many areas and holding its own as an architecture versus Intel/AMD, it was just getting held back by the OS primarily. Unfortunately I do not see this getting better under Intel unless apple does some serious work on 10.5, but then I think 10.5 will primarily be about universal binary compatibility and no code optimizations at all-just make it work. I guess we wait for 10.6 then?
Posted by: drunkenbatman at September 2, 2005 11:27 PM
You don't know the actual numbers, but you know they're going down. WTF ? :-)
I didn't remember them from the conference call, I usually just reference the financial docs I keep on hand, so thanks for filling it in.
Not to nit pick, but the cocoa threading comment seems out of place. Yes cocoa threadings not the greatest, but it doesn't seem to relate to slowness.
Feel free to pick my nits, however in this case I was more giving an example of something that would surprise Casuals if it was laid out for them, and that someone could write volumes on.
And with that, I am off to find more bottles.
Posted by: Bob Ippolito at September 2, 2005 11:32 PM
Yeah, I'll have to second Jay here.. Cocoa's desire for single-threadedness has nothing whatsoever to do with the rest of the article. Single threaded programs are easy to make correct, so Apple is really doing you a favor here by encouraging you to really think about what the hell you're doing before you wantonly throw concurrent computation around. It's more about bugs than performance.
Naively written threading code is almost without fail inherently busted. Smart people like Rentz, who can and have written their own pre-emptive threading libraries, are the kind of people who can get concurrency right. Normal people can not without better help from the platform or a hell of a lot of practice. Objective-C is not such a platform (and neither is C++, Java, or anything I've seen on .NET), and most people don't have that kind of practice.
The saving grace here is that Apple has a bunch of these experienced folk on-staff, and most of the places in Cocoa that really do take a while allow you to do them asynchronously (which may be implemented using a background thread that you don't normally know nor care about) from the main thread. If you're doing a task where you need threads to do something they can't already easily do, then you're probably doing it the wrong way anyhow.
Posted by: Twist at September 3, 2005 12:13 AM
Apple really needs to spend some time focusing on performance (both speed a bug wise) instead of new features for 10.5. 10.4 is really the first step backwards as far as speed goes but it is noticeable enough to be very annoying. 10.2 was almost enough of an improvement to make it feel like I had gotten a faster CPU or quadrupled my RAM but 10.4 makes my iBook feel like it did when I first got it and only had 256 megs of RAM. They need to make Spotlight faster and give us the option to turn off the live searching since that is what lags us people on the low end more than anything. The engineering time spent on Dashboard or iMovie HD (if you can afford an HD camera you can afford Final Cut Express) could have been put to better use elsewhere. At least the Safari team sounds like they have their shit together. Reducing 5000 memory leaks on average down to 5 is one hell of a way to spend a few weeks of their time. Now they just need to get those improvements pushed out to us non-bleeding edge users.
Posted by: tester2 at September 3, 2005 01:53 AM
Problem is they keep focusing on the mysql/apache test, they were told ab was flakey on Mac OS X and still they publish the wrong number ... so 1 out of 2 test is already wrongly conducted. (Don't get me started on the synthetic benchmarking)
Apache is able to sustain over 6 000 red/seq if you run ab from a Linux box over the network to a Mac OS X server, I tested this, and to find a PC source read this http://www.pcmag.com/article2/0,1895,1637655,00.asp
In fact Mac OS X easily outperforms Linux on pure Webserver performance.
Mac OS X - mysql is not a good combo, but claiming this is due to Mac OS X solely is short sighted.
Yes you will find app's running faster on Linux and vice versa, but please keep this in perspective and don't draw the wrong conclusions ...
Posted by: Anthony at September 3, 2005 02:46 AM
For all their problems, the Linux crowd does a LOT of optimizing. The commercial OS vendors have a hell of a time keeping up and if they manage it's because they had a ten-year head start. FreeBSD is the only open-source OS that even comes close to keeping up. Anyone not focusing on performance to the point of obsession is going to fall behind.
I know many people complaining about Mac OS X kernel design because of its speed. I would like to see a clear explanation why Apple chose such a complex design. I cannot believe an originally one-man project, like Linux, could beat up an army of engineers at Apple.It's what they inherited from NeXT, and NeXT probably chose it because it gave them a working kernel without starting from scratch. Linux didn't yet exist and BSD wasn't ported to anything interesting at the time.
In fact Mac OS X easily outperforms Linux on pure Webserver performance. Mac OS X - mysql is not a good combo, but claiming this is due to Mac OS X solely is short sighted.Static content isn't the problem. The problem is that dynamic content is much more demanding, MySQL is the most common SQL server, and SQL is a critical part of dynamic content.
Yes you will find app's running faster on Linux and vice versa, but please keep this in perspective and don't draw the wrong conclusions ...You should take a look at some of the other server benchmarks there. I only looked at the Sun V20z running Linux, but it beat the XServe's scores pretty easily on that site. Not that they give enough information to take the benchmarks seriously...
This was a narrow review, but given that almost every benchmark given in that article and the previous one favor Linux it's pretty hard to imagine anything that spends much time in the kernel will be faster on OS X.
Posted by: matt_tracker at September 3, 2005 03:26 AM
Feels rushed and sloppy and not close to your usual quality DB.
Posted by: Rosyna at September 3, 2005 03:30 AM
The problem isn't so much with OS X as it is with Anandtech's methods. I think a poster on /. summarized it very well.
This article certainly doesn't prove that, or anything, really. I don't know what the guy's thinking. He dismissed the fsync issue without looking into it at all, and without bothering to configure MySQL to work around it even just to see if he could possibly be wrong. No, it has to vindicate his pet theory: there must be something wrong with Mac OS X's threading architecture. He still can't be bothered to compare apples to apples.
The fact is, Johan seems to have ignored everything the engineers had told him and corrected him on. Then he continues to use the official GCC tree (with none of Apple's optimizations, the GNU Foundation won't accept performance patches to GCC).
In fact, you could almost call Johan a troll. He ignored all the facts, and started making up new "facts" to support his inaccuracies. But this seems to be the norm with all of Anandtech's Mac OS related articles.
Posted by: dave at September 3, 2005 03:31 AM
Let me put it this way: If you are trying to find a place to co-locate a Mac Mini as a server, you either have incredibly specific needs or you're one of the people I'm talking about...
Thought you might be amused to see Positive Internet's (Debian Linux) Dolphin Mini webserver.
Posted by: dave at September 3, 2005 03:38 AM
Sorry, messed up the link, it shoud be..
this Dolphin Mini Webserver
Posted by: Rory at September 3, 2005 05:43 AM
Developers just work around it as best they can, as they have little other choice, and it'll take a lot of work from Apple to improve it.
This is a bit of a non-issue really, if you're following the model view controller (MVC) design pattern (as any good programmer should be these days developing desktop apps) then the chances of you running into threading problems are miniscule. Cocoa provides a bunch of useful things for locking, executing methods on the main thread and so on. There are some issues with the event loop being tied into the main thread (specifically menus that freeze your app until you've made your selection), bur other than this there isn't much to worry about.
Posted by: David Sanchez at September 3, 2005 01:23 PM
It's what they inherited from NeXT, and NeXT probably chose it because it gave them a working kernel without starting from scratch. Linux didn't yet exist and BSD wasn't ported to anything interesting at the time.Not really, by 1996, the time Apple acquire NeXT, Linux already existed.
Apple really needs to spend some time focusing on performance (both speed a bug wise) instead of new features for 10.5.
I agree with it, although 10.4 feels faster in my G5 and even in some G3. The problem is the leaking memory in some apps.
Problem is they keep focusing on the mysql/apache test, they were told ab was flakey on Mac OS X and still they publish the wrong number ... so 1 out of 2 test is already wrongly conducted. (Don't get me started on the synthetic benchmarking)
It is true, but it is well known Mac OS X is not the fastest OS in town.
XNU is not a microkernel, so it can be optimized a lot. If mySQL and Apache need optimization, then Apple should include an optimized version with Mac OS X server.
Posted by: Anthony at September 3, 2005 01:50 PM
Not really, by 1996, the time Apple acquire NeXT, Linux already existed.They already had their working kernel inherited from NeXT. I doubt they could afford to change by that point, it was already a race to get something out.
Posted by: Chris at September 3, 2005 08:45 PM
It is true, but it is well known Mac OS X is not the fastest OS in town.
Got benchmarks?
DB, you're backing an article by someone who appears to be a Linux wingnut with an axe to grind. If I were you, I'd retract this post. I'm not saying "OS X is slower than Linux on the same hardware" is necessarily wrong, but the article uses highly questionable test methodology and the author shows dubious understanding of reliable performance measurement techniques. It doesn't prove the thesis by any reasonable standard. If we're going to call a spade a spade, let's make sure it is actually a spade first.
Posted by: drunkenbatman at September 3, 2005 08:50 PM
DB, you're backing an article by someone who appears to be a Linux wingnut with an axe to grind. If I were you, I'd retract this post.
Not going to happen, because I've seen the performane issues for these types of tasks first hand.
Posted by: Miles at September 3, 2005 09:43 PM
There are a lot of comments here that criticize DB for not having hard data showing that Linux is faster than OS X Server. Getting as few as two people to agree on what this "hard data" would be is this side of impossible. People have tried, and they've had to weather attacks on their character and a lot of backseat driving. The software we are talking about is complicated and everyone is going to miss some tiny detail. To claim that missing this or that little thing makes all of their work null and void is the cry of the angry and irrational.
AnandTech tried, and people are crying foul. It's unfair, I don't think that AnandTech struck out to attack Apple's server OS. I definitely do not believe that they're trying to twist the facts.
I manage two XServe machines and they do a good job. One XServe manages OS X clients and this is a reasonable use for the XServe and OS X Server. The other runs our custom application, if it was up to me this machine would be running Linux. I don't have an axe to grind, I'm trying to be pragmatic.
I would be curious to know how many self-appointed defender's of OS X Server are running demanding application's on the XServe. I'm not talking about file servers for Apple clients, I'm thinking more along the lines of core business applications or things that have a serious database back-end.
No matter what, talking about problems and pointing out places where Apple is weak encourages them to fix those same things. There is nothing more irrational than ignoring a problem in the hope that it fixes itself.
Posted by: Equinix Rat at September 3, 2005 11:55 PM
Xserve are now about 30% of the colo servers in major facilities. They aren't selling because of their performance, but because they're cheap storage solutions. Xserves are gettting all sorts of consideration, if not the sales.
Posted by: Peter at September 4, 2005 12:27 AM
Ugh. Wrong wrong wrong wrong wrong.
Anandtech is full of crap. The dominating factor of these benchmarks is that MySQL on Mac OS X is able to call a OS X specific API that actually flushes the disk cache. Not the OS disk cache, but the RAM on the disk drive itself.
This is a VERY slow API, because it actually works, as opposed to every other OS where the similar API just pretends to work to make the benchmarks faster.
If MySQL was compiled with this special feature disabled, speed would be nearly the same. I don't know exactly how "nearly", but it would at least allow people to focus on real problems instead of specuklating about a bunch of stuff that they have NO clue about. Threads are NOT the problem, and if anandtech actually could write a benchmark to save their ass, they'd see that.
Instead, MySQL is WAY more reliable on Mac OS X, and therefore, slower.
And I say anandtech is full of crap, because they've been told this, by lots of different Apple engineers, and they didn't do squat to verify it. Instead they stick with their utterly unsubstantiated rant about mach threads.
Posted by: AC-DC-Syb at September 4, 2005 12:57 AM
And I say anandtech is full of crap, because they've been told this, by lots of different Apple engineers, and they didn't do squat to verify it. Instead they stick with their utterly unsubstantiated rant about mach threads.
Oh, shut the fuck up already you're repeating shit you picked up off of slashdot comments. An Apple engineer posted a theory on a blog somewhere, but Apple admitted the problem! From Anandtech's site: "The server performance of the Apple platform is, however, catastrophic. When we asked Apple for a reaction, they told us that some database vendors, Sybase and Oracle, have found a way around the threading problems. We'll try Sybase later, but frankly, we are very sceptical."
Posted by: ex2bot at September 4, 2005 01:57 PM
So the obvious next step is to look at Oracle and Sybase on OS X Server and see if they actually perform well.
If they can perform, then it's really a non-issue, isn't it?
ex2bot
Posted by: cabbey at September 5, 2005 11:23 PM
(with none of Apple's optimizations, the GNU Foundation won't accept performance patches to GCC).Actually Rosyna, the gcc team is happy to accept patches that improve the speed of generated code within a few simple guidelines: you can't drop it once they close a tree down for new function without a lot of agreement, you can't do something you know will break any code paths without emiting workarounds, and if the compile time performance penalty is too much you usually have to make your optimization switched. Oh, and they usually don't much like graceless, ugly, unmaintainable hacks that don't play well with the rest of the tree.
Posted by: Iljitsch van Beijnum at September 6, 2005 01:26 PM
Well, someone must be drunk here...
Those Anandtech types really don't know how to do a decent test. I'm not sure what the deal is with MySQL, but the problem seems too big to be caused by something trivial. There's probably more than one factor at work.
But even though they do test Apache 2 this time, they don't tell what kind of threading was used! This is ridiculous!
Just for the record: Mach threading is very good. When Linux and *BSD were still figuring this stuff out, there were already multi-CPU Macs running multithreaded apps that actually take advantage of the extra CPU. (This should be interesting when Windows desktop apps are suddenly confronted with more than one core, while Mac apps have been doing this for years.)
The trouble with things like threading and network interaction is that an application programmer really has to talk to the system in a way that agrees with what the system expects. So if you take an application that spends a lot of its time in the kernel that was developed on one OS, you're going to see strange things when you run it under another OS. That's why Apache has a bunch of different threading mechanisms: one size very definately does not fit all.
Just in case anyone cares: I wrote a little program that does a bunch of busywork in a configurable number of threads. By varying the number of threads you can easily see how much overhead is incurred by thread handling. I'll be happy to send the program to anyone who wants to test it on at least two different operating systems or hardware configurations. Unfortunately, I only have a 5 year old Linux box so my own test results are mostly meaningless.
Posted by: Anthony at September 6, 2005 08:13 PM
This should be interesting when Windows desktop apps are suddenly confronted with more than one core, while Mac apps have been doing this for years.Hyperthreading's been around for years as well, and it tends to get used when there's a substantial benefit.
Just in case anyone cares: I wrote a little program that does a bunch of busywork in a configurable number of threads. By varying the number of threads you can easily see how much overhead is incurred by thread handling. I'll be happy to send the program to anyone who wants to test it on at least two different operating systems or hardware configurations. Unfortunately, I only have a 5 year old Linux box so my own test results are mostly meaningless.Does it do I/O in the threads? AFAIK OS X uses an M:N threading model, and AFAIK M:N threading does alright in CPU-bound situations but not so well when there's I/O involved.
Posted by: Iljitsch van Beijnuum at September 7, 2005 02:46 AM
Not really. It mostly executes a mostly empty loop.
What I wanted to test is thread creation, context switches between threads and destroying the threads again. People keep saying the Mac is bad at this, but in my tests the difference with FreeBSD and Linux was minor, although I don't have the hardware to do a proper test.
I'm not sure how useful it is to do I/O in many threads at the same time anyway...
Posted by: at September 7, 2005 03:52 AM
I'm not sure how useful it is to do I/O in many threads at the same time anyway...
Like in a multitasking OS with different processes and threads all vying for I/O...
Posted by: Iljitsch van Beijnum at September 7, 2005 07:24 AM
As a rule, if you need to do 10 things, it's faster to do one at a time in a single thread/process, rather than fire off 10 processes or 10 threads and have each do one thing.
There are two reasons to have threads (or processes, but threads have less overhead). The first one is so you can take advantage of multiple CPUs, and the second one is to do something that takes a lot of time but not much processing power in a way that's easier to program.
For instance, if 10 people ask a web server for a 10 MB file, and each of those people has an 1 Mbps link but the web server has 10 or 100 Mbps, sending all the files at the same time is faster.
You don't really need threads to implement this: you can just have a loop that writes bytes to the network on the TCP sessions that are ready to accept more data. However, this is hard to program, and it requires that ALL operations are "non-blocking". With threads, all of this happens lower inside the system so it's much easier to write the program that performs simultaneous actions.
But for regular I/O such as reading/writing a disk, having several threads all some reading/writing is not a good thing, because soon the disk heads spend more time moving from one place on the disk to another than doing actual reading or writing. Having the threads preprocess the data and then let one central thread do the I/O in the largest possible chunks works better here. (But it's more work to program.)
Posted by: at September 7, 2005 11:20 AM
What I wanted to test is thread creation, context switches between threads and destroying the threads again. People keep saying the Mac is bad at this, but in my tests the difference with FreeBSD and Linux was minor, although I don't have the hardware to do a proper test.Are you saying you did tests with FreeBSD in lieu of MacOS?
This is a fairly complicated situation. I don't think results on FreeBSD can be applied to MacOS because FreeBSD has 3 pthreads libraries, none of which are that similar to what MacOS has. Also, a lot of this stuff is kernel-dependant, and FreeBSD and MacOS X have completely different kernels.
I'm not sure how useful it is to do I/O in many threads at the same time anyway...Multiple CPU-bound threads aren't what tax the implementation. That's easy. It's lots of threads doing I/O, blocking, restarting, etc. This can really screw up some implementations, and if MySQL is having problems with the threading some empty loops will not reveal it.
You don't really need threads to implement this: you can just have a loop that writes bytes to the network on the TCP sessions that are ready to accept more data. However, this is hard to program, and it requires that ALL operations are "non-blocking". With threads, all of this happens lower inside the system so it's much easier to write the program that performs simultaneous actions.Yes. This is why Zeus and thttpd win so easily with static content.
But for regular I/O such as reading/writing a disk, having several threads all some reading/writing is not a good thing, because soon the disk heads spend more time moving from one place on the disk to another than doing actual reading or writing.I agree. It would have to be network I/O, or specifically set up to do sequential operations on the hard drive, or something similar to that.
Posted by: Iljitsch van Beijnum at September 8, 2005 08:40 AM
Of course I tested MacOS, in addition to Linux and FreeBSD. If you want to do the same, here's the source:
http://www.muada.com/thtest.c
Note that this test is meant to test thread creation/switching/joining ONLY, which works without trouble: with 50 threads this took 12.119 seconds on my PB 1.25 GHz (Tiger), with 500 threads 12.161 seconds. Under FreeBSD 5.4 on a P4 2.4 GHz it's 7.551 vs 7.606 seconds. (Keeping THREADS * BUSYWORK constant for easy comparison.)
Obviously MySQL is doing some other stuff that rubs MacOS the wrong way (or the other way around).
Posted by: Johan of Anandtech at September 11, 2005 01:26 PM
Just a quick note to the people who think I am "drunk" and the article is "full of crap".
1. The MySQL tests used the MyISAM engine. fsync() is not an issue there. Sorry Peter.
2. Secondly, we used Apache and MySQL that came with Mac OS X server. Configuration files were only changed if we saw that this improved performance.
3. Indeed, Apple will never confirm this publically, but when we confronted them with our numbers, the first thing on the Apple Tech support guys was "a threading issue".
4. FYI, Apple did sent a few people to our lab to investigate our claims.
It's so nice to see rational mac users around.
I work as a lowly tech dude for a reseller. The sales people make all these wonderful promises about the systems and how fast they are. We techs come along and install it and cop the blame when it doesn't meet their expectations.
For an all mac internal environment it would be difficult to find a better server system to manage it than OS X Server. However, anything that's out in the real world, or not explicitly requiring OS X generally gets run on FreeBSD with me. (Or DragonflyBSD)