And then I found out that most of that mucking around in the kernel build process isn’t necessary.

RHEL5 u6 (at least my copies — note that I have the -xen kernel packages installed, which might affect things) actually has the XFS kernel module in /lib/modules/kernel/fs/xfs — it just doesn’t have the xfsprogs package, and the xfsprogs package is not available via any supported means. I downloaded the xfsprogs srpm from the XFS project page and used rpmbuild to build and install it myself.

Does anyone else in this day and age find it ridiculous that Redhat does not support filesystems larger than 16TB without some form of hackery? I can buy that much disk storage at my local Office Depot.

Note that I’m pretty sure that the mount option ‘inode64′ is required if you’re over 16TB, and that 32-bit NFS clients do not like the inode64 option one tiny bit.

One nitpick about the article: They pointed out that the Gawker servers were running Linux 2.6.18 — that likely means that they’re running Redhat Enterprise Linux, or CentOS, or a RHEL derivative. RHEL’s current kernel is 2.6.18-194.26.1 — the high final numbers mean that they’ve done a lot of backporting for security reasons. Of course, for those of us tied to RHEL, it serves as a reminder of why we should consider something a little fresher…

And man, doesn’t that make Gawker look like a bunch of incompetent dicks? You’ve got the iPhone thing, and now this…

The web servers are *mostly* running Drupal installations, backed by MySQL. Drupal is a fairly hefty CMS and for a long time we’ve used several physical servers fronted with Pound. This is very obviously expensive — two servers alone from Rackspace cost $1500/mo. While we could do it cheaper in the cloud, we weren’t comfortable with the faceless/nameless cloud’s ability to meet the uptime requirements in our SLA, nor were we comfortable with the ability of the cloud vendors to provide the support levels that were required by our customers.

Building this out —

First, I decided to use VMWare ESXi and vCenter to fulfill the high availability requirement. Since this was going to be a remote installation, I needed to be able to quickly and effectively diagnose problems and needed to have support due to the nature of the clients we were hosting. While I could’ve gone with Xen and SLES’s HA package, the license for a Essentials Plus version of VMWare was actually cheaper than the necessary bits from Novell and Citrix… and unlike those products, VMWare supports failover and migration between hosts out of the box instead of requiring weird Pacemaker/OpenAIS cluster configurations.

Second, after pricing comparable hardware solutions, we ended up going with Dell. We priced HP, Sun/Oracle, and SuperMicro systems. The SuperMicro systems were closest in cost, but ended up failing on support. Compared to Dell’s Same Day/Next Business Day offerings, SuperMicro’s “go through your reseller” strategy falls way short of business expectations. Also, one neat thing about the Dell MD3200/3220 is that it can handle up to 8 hosts connecting via 6 gb/s SAS at once to the same LUN. Since we don’t expect to have anywhere near that number of hosts in the short term, we can effectively use VMFS. In the longer term, if we outgrew that number of hosts without upgrading to a fiber SAN, we could share VM partitions out via NFS and offload our highest concurrent storage user, MySQL, to it’s own storage.

Third, for the gateway/router/etc, we chose the Cisco 5505 ASA Security + device. Two of them will fit in a 2u rack mount kit (actually 1.5, which is ideal for the crowded top of the colocation cabinet we’re in) and each has 8 1gb/s ports. They’re pretty full-featured little boxes (note that you MUST buy a support contract if you want a warranty past 90 days though!)… sort of like a home cable modem, they’ll gateway, NAT, provide DHCP, and other basic networking functions out of the box … unlike your average home gateway, though, they support active/standby failover, a ton of diagnostics, routing, and shaping, and speak the same asa/pix language that we’re all familiar with. They also do VPN via everyone’s favorite Cisco VPN client. My other choice, besides the comparable Juniper Networks equipment, was to build my own system using BSD & CARP or WRT-DD… while both of those might have been cheaper for the client, they didn’t fulfill the support requirement.

Last, we needed to provide an onsite and offsite backup capability. With 15k RPM disks running up to $1k, the best solution I could find was a 2TB Buffalo TeraScale NAS. I configured it as a 3-drive RAID5 with 1 hot spare.

When we were done, the total configuration looked something like this:
- 3x Dell R610 w/ dual X5620 + 32gb RAM + no drives (boot from internal USB)
- 1x Dell MD3200 w/ 2 MM, 7x 300gb 15k drives
- 1x 2tb Buffalo TeraScale III NAS
- 2x Cisco ASA 5505 + rack mount kit
- VMWare Essentials Plus

For a few of the infrastructure bits, we purchased a license for Redhat EL 5.5. For most of the webheads and other ‘disposable’ VMs, we use CentOS 5.5. VCenter runs in a Win2k8r2 VM.

Here’s a few examples.

MySQL

Theoretically, you could recover anything you needed from the binary log, as long as you’ve got a good starting point and a good ending point. (This, by the way, is a good reason to flush the binary logs and take a backup on a regular basis.) What if your binary log’s corrupted, though? You need to fall back to a full SQL backup … which you’re doing regularly, right?

If your binary log is corrupted, any mirrors you are using that are based on that binary log are corrupted as well.

Case in point: I had a client with a very active, very large database… north of 15GB in InnoDB. The binary log hit a bug and corrupted itself. The backups were being done from that mirror so that they didn’t interrupt the main machine’s processing, but they only kept a few days worth, so we couldn’t use those backups to restore. The most recent un-corrupted dump from the main machine had been taken three months before. Luckily, the client had done some application-level backups to an XML format, and we were able to (laboriously) restore from that. It cost about $3,000 because they didn’t want to degrade their forum’s performance for a half hour every night and pay for an extra TB or storage or so to keep more than a few days worth of upgrades.

Servers

Scenario: Hard drive gets corrupted or dies. You need to get the machine back up quickly. You have a snapshot of the machine … but your snapshot is on the same storage as that machine unless you back it up somewhere else.

On top of that, storage requirements have been growing rapidly for servers. Where a linux server take less than 1GB, Windows 2008R2 can take up 20GB with system files alone. (In fact, if you plan to have any data on that server, or keep any logs, we’d recommend going with 40GB minimum for your C: drive.) It’s important to back that up to something that’s not on the same system disks.

Better yet, take a hint from the application-level backups — and back up your registry, configuration files, and data separately from the snapshot. We tend to use RSync for this role and put it in a rolling-backup mode with the –link-dest option to ease recovery.

VMWare

Same principle as above. Snapshots are usually stored in the same datastore. Datastore goes bye-bye, so do your snapshots.

There’s some great products out there that can really help with this issue. The one we use is VEEAM Replication and Backup. It can be used to replicate a snapshot to another VMWare cluster, or back up the datastore files at a consistent snapshot point and then copy them elsewhere all in one step. We use a two-step process — we keep them locally on the backup server and also transmit them to another datacenter across campus.

When using VEEAM with Windows, make sure that VMWare Tools is installed and that you enable the VSS integration. (You’ll also need to make sure that the administrative share option on the system drives are enabled, and that the appropriate firewall ports are opened.) This ensures that you’ve got a transactionally consistent backup snapshot.

Practice, practice, practice

The only way to make sure that you can recover from a disaster is to test recovering from a disaster. At least once a year, we practice recovering from a worst-case scenario. That means bringing up a new machine from scratch, re-implementing all of the options and configurations, and then restoring the data. Despite that kind of restoration being something that should never happen, it does — and practice gives you insights into how to improve the processes and turns a recovery operation from an expensive nightmare that sets back all of your other processes into something that you can execute quickly and professionally.

After triaging and addressing the main issues based on the logs, we were left with two more issues. The first was the inability of Drupal to perform well in an environment where it had to rebuild every page from source for every page view. This is well documented in the drupal community; there are many pages inn the documentation area of Drupal that deal with caching and performance optimization. The second issue was MySQL performance and the long table lock/scan times we were seeing on some queries that could not be further optimized.

We scheduled a 2 hour downtime with the customer to install some tools. Our checklist was installing memcached and PHP-APC. I also wanted to take the time to back up the MySQL database and run a good check_table on each of the MyISAM tables. (Yes, I know. MyISAM. More on that later.)

Side note: I would typically prefer xcache, which in my mind is superior to APC because I have an easier time working with it and prefer it’s management interface and tuning parameters. However, APC was available as a binary package for the platform we were on, and xcache was not. To make things faster and easier, we chose APC. Despite the endless debate about which is superior, both are usable and work. I have not run into problems using APC on an 8-core system, despite oft-reported-but-never-proven flock() issues.

APC was fast to install and required minimal tuning. It produced a noticeable performance improvement. However, the number of deadlocked apache threads (and total number of apache threads) went up, and the other Apache errors that dealt with clients timing out did not cease.

We installed the Drupal Memcache implementation along with the appropriate PECL module. We configured two pools, both using up to 1 GB of RAM (which we had to spare on the web server.) The ‘hot’ pool would mostly handle cached pages for non-logged-in users, and the other one would handle some higher volume caching for users that are logged in, as well as some internal/custom functionality to go along with specialized RSS feed parsing. (Side note: We found that the Cache and Cacherouter plugins did not work as expected. Rather than waste downtime troubleshooting them, we used what worked.)

Again, we saw a huge performance boost. We needed to do some tuning (changing certain cache settings and analyzing performance, but that was essentially everything that we could find to do from a single-server web server side of things.

While we’re on the topic of drupal: Don’t forget that Drupal has a ‘cron’ program that should be getting called remotely. It’s sort of a poor man’s cron solution, but it works. It was causing our load to spike every 20 minutes. We occasionally disabled it during testing to be sure we understood it’s effects.

The next beast to tackle was the database. As previously mentioned, it was on MyISAM tables. Obviously, this isn’t ideal. We found that node lookups, statistics lookups, and searches were taking up a disproportionate amount of server time because they were both The weirdest part was that we were seeing some full table scans in the slow query log (i.e. 3 million rows scanned) but a later ‘explain’ statement couldn’t replicate the performance recorded in the slow query log.

We batted around adding indexes. The issue was that Drupal’s search and nodes tables are frequently altered, which means the indexes become scrambled quickly. And really, what was taking time was the size of the table we were dealing with — the table wouldn’t fit in memory, so it was copying it to a disk temporary table and then doing a filesort.

Running check_table did the trick to re-sort the indexes and ‘defrag’ the files, but the benefits only lasted so long.

What we ended up doing was taking the database down, dumping everything out to a SQL file, and re-importing everything to InnoDB. Make sure that innodb_files_per_table is enabled, or you might end up with some unexpectedly big files — this depends on your architecture and filesystem. Remember that InnoDB files can not currently shrink. (Also: You can do the table changes online, but it’s really not recommended. It takes a long time, especially when some of your tables are larger than 1gb.) Don’t forget to switch to set innodb_buffer_pool_size appropriately.

The change to InnoDB, the implementation of both PHP engine-level opcode and actual built pages, and the careful tuning of Apache and MySQL parameters led to stability for this client.

There were some further problems, but they were with an unrelated product that causes a nightly load spike on the database machine. Tomorrow night I’ll covering the cleanup work: NFS iops vs. local disk, binary logging and the lack of backups in the original configuration, and building some redundancy into the system so that it can tolerate faults more smoothly.

The below comes from a consulting gig that I’ve been working on recently. The parties will remain nameless. I’m going to break this into several parts, since it took over three weeks to resolve all of the immediate problems with the site, and we’re still not all the way done with the task list.

Going in, I knew that we were dealing with a heavily loaded Drupal site that shared a mysql database with a wiki and a forum. The site would go down at random times — sometimes multiple times per hour. Upon logging into the server the first time, it seemed slow — so I immediately called ‘uptime’ and the answer came back with all three time period load averages over 90 on an 8-core server. There were 125 Apache processes running, but most of them were in Deadlocked state. The very second command I ran on the server was killall -9 httpd, which is never the way you want to start out a consulting gig…

While that was busy killing off processes, I checked the Apache configuration. Sure enough, it was still at the stock settings. I immediately cranked up the requests per process to 20,000 and upped the server limit to 300. (Remember, we’re dealing with prefork here.) I restarted Apache and watched it churn. It handled the load far more gracefully with some room to move around, and I quickly saw the number of Apache processes spike, and then sink down to about 80 and stay there.

The next step was looking through the logs. A quick aside about logs: I like my logs to be clean. I don’t like debug messages, I don’t like status messages, and I don’t want to see either of them. If I have a lot of a certain type of status message that I *do* want to trap, I make sure that syslog puts it into it’s own file or I handle the problem that’s causing it. In this case, /var/log/messages had a bunch of SNMP messages logging each get, and some messages about martian packets. The martian packets issue could be (and was) resolved with a quick firewall tweak to reject packets from an illegal source. The snmp issue was resolved by editing snmpd’s startup configuration to log to local1 instead of the default (check your man file for snmpd to make sure you get the right flags, it’s changed…), and then editing syslog’s configuration to log everything on local1 to /var/log/snmpd — and don’t forget to add it to logrotate!

Now we were down to two classes of errors. The first was obvious and sort of easy to troubleshoot: “MySQL server has gone away.” Log into the MySQL server. See if there’s slow-running queries. Nope? Well, double check the timeout that’s set in /etc/my.cnf — on this server, slow-query-time was set to twenty seconds, but timeout was set to ten seconds. Well, that’s not very useful. Also, check your caches and table types. In this case, everything was MyISAM. More on that later — for now, just make sure we’re using the right kind of caching strategy for your table type and system specs, which in this case is MyISAM key cache (and lots of it!). Try to fit all of your most-used tables in memory.

On this gig, we got the site back on it’s feet with these things. Downtime went from multiple events an hour down to one or two events per six hour period. Unfortunately, we were also out of easy things to change. Next time I post, we’ll start to get into fixes that will cause downtime.

As a smaller customer, I’m far more interested in the SunFlash F20 PCIe card — which I don’t see many people blogging about. Looks like I could add that to not only my existing systems, but non-Sun systems that can make use of that sort of storage. That, ladies and germs, is something worth the name “OpenWorld” — as in, a world of open wallets.