After triaging and addressing the main issues based on the logs, we were left with two more issues. The first was the inability of Drupal to perform well in an environment where it had to rebuild every page from source for every page view. This is well documented in the drupal community; there are many pages inn the documentation area of Drupal that deal with caching and performance optimization. The second issue was MySQL performance and the long table lock/scan times we were seeing on some queries that could not be further optimized.

We scheduled a 2 hour downtime with the customer to install some tools. Our checklist was installing memcached and PHP-APC. I also wanted to take the time to back up the MySQL database and run a good check_table on each of the MyISAM tables. (Yes, I know. MyISAM. More on that later.)

Side note: I would typically prefer xcache, which in my mind is superior to APC because I have an easier time working with it and prefer it’s management interface and tuning parameters. However, APC was available as a binary package for the platform we were on, and xcache was not. To make things faster and easier, we chose APC. Despite the endless debate about which is superior, both are usable and work. I have not run into problems using APC on an 8-core system, despite oft-reported-but-never-proven flock() issues.

APC was fast to install and required minimal tuning. It produced a noticeable performance improvement. However, the number of deadlocked apache threads (and total number of apache threads) went up, and the other Apache errors that dealt with clients timing out did not cease.

We installed the Drupal Memcache implementation along with the appropriate PECL module. We configured two pools, both using up to 1 GB of RAM (which we had to spare on the web server.) The ‘hot’ pool would mostly handle cached pages for non-logged-in users, and the other one would handle some higher volume caching for users that are logged in, as well as some internal/custom functionality to go along with specialized RSS feed parsing. (Side note: We found that the Cache and Cacherouter plugins did not work as expected. Rather than waste downtime troubleshooting them, we used what worked.)

Again, we saw a huge performance boost. We needed to do some tuning (changing certain cache settings and analyzing performance, but that was essentially everything that we could find to do from a single-server web server side of things.

While we’re on the topic of drupal: Don’t forget that Drupal has a ‘cron’ program that should be getting called remotely. It’s sort of a poor man’s cron solution, but it works. It was causing our load to spike every 20 minutes. We occasionally disabled it during testing to be sure we understood it’s effects.

The next beast to tackle was the database. As previously mentioned, it was on MyISAM tables. Obviously, this isn’t ideal. We found that node lookups, statistics lookups, and searches were taking up a disproportionate amount of server time because they were both The weirdest part was that we were seeing some full table scans in the slow query log (i.e. 3 million rows scanned) but a later ‘explain’ statement couldn’t replicate the performance recorded in the slow query log.

We batted around adding indexes. The issue was that Drupal’s search and nodes tables are frequently altered, which means the indexes become scrambled quickly. And really, what was taking time was the size of the table we were dealing with — the table wouldn’t fit in memory, so it was copying it to a disk temporary table and then doing a filesort.

Running check_table did the trick to re-sort the indexes and ‘defrag’ the files, but the benefits only lasted so long.

What we ended up doing was taking the database down, dumping everything out to a SQL file, and re-importing everything to InnoDB. Make sure that innodb_files_per_table is enabled, or you might end up with some unexpectedly big files — this depends on your architecture and filesystem. Remember that InnoDB files can not currently shrink. (Also: You can do the table changes online, but it’s really not recommended. It takes a long time, especially when some of your tables are larger than 1gb.) Don’t forget to switch to set innodb_buffer_pool_size appropriately.

The change to InnoDB, the implementation of both PHP engine-level opcode and actual built pages, and the careful tuning of Apache and MySQL parameters led to stability for this client.

There were some further problems, but they were with an unrelated product that causes a nightly load spike on the database machine. Tomorrow night I’ll covering the cleanup work: NFS iops vs. local disk, binary logging and the lack of backups in the original configuration, and building some redundancy into the system so that it can tolerate faults more smoothly.

The below comes from a consulting gig that I’ve been working on recently. The parties will remain nameless. I’m going to break this into several parts, since it took over three weeks to resolve all of the immediate problems with the site, and we’re still not all the way done with the task list.

Going in, I knew that we were dealing with a heavily loaded Drupal site that shared a mysql database with a wiki and a forum. The site would go down at random times — sometimes multiple times per hour. Upon logging into the server the first time, it seemed slow — so I immediately called ‘uptime’ and the answer came back with all three time period load averages over 90 on an 8-core server. There were 125 Apache processes running, but most of them were in Deadlocked state. The very second command I ran on the server was killall -9 httpd, which is never the way you want to start out a consulting gig…

While that was busy killing off processes, I checked the Apache configuration. Sure enough, it was still at the stock settings. I immediately cranked up the requests per process to 20,000 and upped the server limit to 300. (Remember, we’re dealing with prefork here.) I restarted Apache and watched it churn. It handled the load far more gracefully with some room to move around, and I quickly saw the number of Apache processes spike, and then sink down to about 80 and stay there.

The next step was looking through the logs. A quick aside about logs: I like my logs to be clean. I don’t like debug messages, I don’t like status messages, and I don’t want to see either of them. If I have a lot of a certain type of status message that I *do* want to trap, I make sure that syslog puts it into it’s own file or I handle the problem that’s causing it. In this case, /var/log/messages had a bunch of SNMP messages logging each get, and some messages about martian packets. The martian packets issue could be (and was) resolved with a quick firewall tweak to reject packets from an illegal source. The snmp issue was resolved by editing snmpd’s startup configuration to log to local1 instead of the default (check your man file for snmpd to make sure you get the right flags, it’s changed…), and then editing syslog’s configuration to log everything on local1 to /var/log/snmpd — and don’t forget to add it to logrotate!

Now we were down to two classes of errors. The first was obvious and sort of easy to troubleshoot: “MySQL server has gone away.” Log into the MySQL server. See if there’s slow-running queries. Nope? Well, double check the timeout that’s set in /etc/my.cnf — on this server, slow-query-time was set to twenty seconds, but timeout was set to ten seconds. Well, that’s not very useful. Also, check your caches and table types. In this case, everything was MyISAM. More on that later — for now, just make sure we’re using the right kind of caching strategy for your table type and system specs, which in this case is MyISAM key cache (and lots of it!). Try to fit all of your most-used tables in memory.

On this gig, we got the site back on it’s feet with these things. Downtime went from multiple events an hour down to one or two events per six hour period. Unfortunately, we were also out of easy things to change. Next time I post, we’ll start to get into fixes that will cause downtime.

Today, the bug was updated and changed to “unsupported” status. According to the MySQL website, the only supported platform for MySQL Workbench is Windows. This is quote odd, because it seems that MySQL has been working hard to release a cross-platform version of it’s workbench as recently as one week ago.

I suppose this is why Drizzle is getting such a huge groundswell of support behind it: MySQL’s development efforts have been fractured and mismanaged so badly at this point that nothing can save the company or it’s products.

Update: Apparently they’re phasing out Query Browser and the other GUI tools in favor of making workbench cross-browser. I got confused because I filed a bunch of bugs at the same time, most of them dealing with Workbench crashes.

group_concat. To use it is to know it, and to know it is to love it.

LÖVE looks interesting, kind of like a relatively polished PyGame without the annoying Python.

Update: Jay posted his sample code.

(Then again, I should know better than to work off of someone’s code when they term themselves a “Web Craftsman” and have personal philosophies as headlining elements of their “about me” page.) See also: New Media Douchebags Explained.)

If you’re banging your head against this like I have been for the past four hours, you’ve probably figured out a few things. First, the library files that are required to make the javascript work need to be called in a particular order. That order is:

1. The MooTools 1.2 –FULL– compressed library. Yes, I know he says you only need a few components. Yes, he lied. Selecting the components he specified will give you an error along the lines of “window.scroll is not a function at Swiff.Uploader.js (line 62)” … so save yourself the time and just pull down the full compressed/mangled MooTools.
2. Fx.ProgressBar.js
3. Swiff.Uploader.js
4. FancyUpload2.js
5. script.js

Stray from that order, and you develop problems. Also, I found that instead of using

window.addEvent('load',function ...

as the first line of script.js, it was better to use

window.addEvent('domready',function ...

.

Second, let’s take a look at the sample file that’s supposed to magically save what we spit out. Specifically, script.php — which receives the ajax update based on a session variable that’s set, and then saves the information with an indexed .ini file. First off, it’s hard to debug. Let’s use PHP’s output buffering functions to help that. You can use this snippet of code to write a log of what’s going on:

 13  $logfile = fopen($folder.'logfile','a');
 14  ob_start();
 15  var_dump($_FILES);
 16  var_dump($_POST);
 17  var_dump($_GET);
 18  fwrite($logfile,ob_get_contents());
 19  ob_end_clean();

What you’ll notice is that, with the default code that comes with the script, you’ll have the same session ID and since the session ID is created in the javascript and each session ID is incremented in the PHP script, you’ll overwrite your file if you’re uploading multiple files — which was the whole purpose of using flash and AJAX in the first place! This is what we call a brillant(sic) programming decision… or really, one where the developer just didn’t really THINK before he wrote it. I’ve rewritten my function to simply create a directory in the ‘uploads’ folder with a batch name that I enter on the main form. After all files have uploaded successfully (or not), I have a ‘continue’ button that does a bunch of database work and sucks the text of the files into the database.

Now, I’m working with a Zend Framework application here — this means that I need to make a few changes. First, the script depends on not having any authentication ahead of the request. While the browser will still maintain the session, the ajax requests won’t have the browser’s cookies and therefore will bounce off of any preDispatch authentication. To get around this but still stay secure, I generate a (hidden) temporary session ID, store it in the database, and clear the table on a regular basis.

I’m on a crunched deadline and still don’t quite have things working yet, but in the next day or two I’ll post the fruits of my labor. Too bad this uses MooTools and not Dojo… as far as I know, Dojo still doesn’t have a flash-compatible uploader that will allow multiple selection.

But back to where I started: Harald, dude, you ain’t no web “craftsman” if someone needs to spend 8 hours learning your code and writing workarounds before they can implement it.

The problem was caused by our new configuration of MySQL. The usual approach to setting MySQL’s table cache size is to make it large enough that it can hold all of your tables open, but we have too many tables for that to be practical. Instead, we decided to make it big, to improve the odds that tables we need were already in the cache. And since we had new, bigger hardware with more RAM and faster disks, we decided to cache even more tables.

Unfortunately, while opening tables is fast in MySQL, the speed at which it can close tables gets slower and slower as the table cache grows. Linearly, in fact. And since the cache will always end up full, the speed that MySQL can close tables becomes the speed that MySQL can open tables. And to exacerbate the problem, opening and closing tables in MySQL uses a global lock, so only one thread can be opening or closing tables at a time, regardless of how many threads are executing in parallel.

Once we understood this, we realized that if the cache is going to be full, it needs to be kept as small as possible. We reduced it to the minimum we could get away with without having to reopen tables in the middle of one session interacting with FreshBooks. And now that closing tables is faster, everything’s faster.