PostgreSQL, FreeBSD, and Free Dog Food

This week I did something I’d prefer to never repeat:  I left the country, did something useful, and made it back again in the same day.  The occasion was the FreeBSD Developer Summit, held just before BSDCan–the convention that happens in Ottawa the week before PGCon every year.  So I get to head right back again next week, but stay a while that time.

The FreeBSD developers were nice enough to sponsor my trip so that we could talk about both the business and technical hurdles that I felt were keeping the sort of companies I work with from deploying their databases on FreeBSD more often than they do.  My slightly updated slides are available on our talks page, I cleaned up a couple of things from what was presented (the most important rewording I’ll talk about below).

I was very pleased at how friendly and receptive the developers were even to some of my critical comments.  FreeBSD and PostgreSQL have very like minded communities:  open for any purpose BSD license, academic roots, developers focused on stability, and even a strong documentation culture.  There’s been plenty of cross-over too.

Much of the PostgreSQL infrastructure has been run using FreeBSD jails for quite some time (although plans are moving to use more Debian in its place, details on why at Inside the PostgreSQL Project Infrastructure).  My running joke during the talk was that if PostgreSQL developers are eating their own dog food by deploying critical infrastructure that depends on the database, much of that has been served in a FreeBSD bowl.  (The lunch at the conference session was pizza, much better choice)

And there’s been plenty of FreeBSD development that’s used PostgreSQL benchmarking as a measuring stick for the success of their advances.  The very popular Introducing FreeBSD 7.0 slides that not only showed their achieving performance parity against Linux during that release, it doubled as a document showing how PostgreSQL outscales MySQL.  Cheers all around for community driven, BSD licensed code.

One bit of audience contention during my talk came from my assertion that not having support for Emulex fiber channel cards in FreeBSD was preventing a significant amount of “big iron” adoption for databases, due to their perception as the market leader for connecting up expensive hardware like SANs.  The guys from FreeBSD hardware and support vendor iXsystems called me out on that, suggesting that the alternative vendor here–QLogic–is both completely trusted by the big boys and has top notch FreeBSD driver quality.

I did a bit more research into whether I was suffering from sampling bias from the set of people I’d talked to about this, and it looks like that was the case.  While Emulex claims they’ve been named Sun’s “Best-in-Class Supplier for OEM products“, and all the Sun FC cards I’ve personally run into came from them, there are tons of Sun rebrands of both Emulex and QLogic cards.  Same thing is true at all the other vendors I mentioned in my talk; you can get FC cards from both manufacturers via HP and Dell too.  I think my general point, that not supporting both Emulex and QLogic hurts the perception of FreeBSD as a serious choice for large businesses, still stands; it’s just not quite as bad as I’d feared.  Accordingly, I tweaked the wording in the slides I’m publishing, to better match reality here than the ones I presented.

In additional to the solid core they’ve been growing for years, FreeBSD’s license has allowed them to incorporate two very valuable features Sun released as open-source, ZFS and DTrace, into their operating system, both of which are incompatible with Linux’s license and are extremely valuable for PostgreSQL deployments.  It’s still not ideal yet; FreeBSD DTrace can currently be used only by root for example.  Limitations such as these have in the past kept me from being particularly motivated to work with FreeBSD.  The existence of a free commercial Solaris that ran on generic hardware, combined with the steady progress and open enough community around OpenSolaris, satisfied my needs better.  While not many of my PostgreSQL installations have been on Solaris, its has a monopoly share for hosting the terabyte scale databases I’ve worked with.  High quality filesystem snapshots via ZFS and the additional piece of mind you get from disk block checksums alone justified those platform decisions.

The problem today is that hating everything about how Oracle does business is what got me working with PostgreSQL in the first place, and now that they own Sun they’re doing the same things to Solaris.  No more Solaris on non-Sun hardware, serious cutbacks on the open-source version (OpenSolaris looks like a walking corpse to me), cutting off even basic OS patches unless you have a support contract–that’s what we’ve seen just in the first round from Oracle here.  Solaris isn’t free in any sense of the word again, we’re right back to the same dynamics that pushed me away from them and toward Linux fifteen years ago.

But I continue to be dissapointed at how little focus there is on quality control in Linux.  How poorly the filesystem mechanics work for the sorts of database work I do doesn’t help either.  The Linux OOM killer might as well be named the Linux PostgreSQL Hater for how it acts on my servers.  And those sexy Solaris features I know work so well for databases, still not there (even if SystemTap is getting better at DTrace emulation).

Meanwhile, FreeBSD has the whole “free” thing sorted out right in their name, and their quality control paranoia is similar to that of your typical good DBA.  It looks to me like they’re very close to fully assimilating ZFS and DTrace to the point where they can start improving them, rather than just working on getting the original feature set Solaris already had complete and the matching code stable.  I think all of us who work on business critical PostgreSQL deployments and who value free software should do a sanity check on just what dog food we’re chewing on, and start making sure there’s a FreeBSD bowl there at least sometimes.  From what I heard this week, the FreeBSD developers are gearing for another round of chewing on ours too.  They’re looking into database oriented performance improvements as part of future development, and they’re not any happier about using MySQL for that than I am about running PostgreSQL on Solaris.  Looks like it might be bowls of dog food all around.  Nobody said that leading the software industry was going to be tasty.