Karl Katzke

This article got posted on Digg earlier today: When Acts of God Bring Down Web Hosts … and it’s got to be the lamest collection of such stories that I’ve ever seen. Anyone who deals with other people’s servers

Here’s my list of stories from the past year:

- Softlayer Dallas: When switching back from generator power after a tornado passed within a mile of the datacenter, they blew their 600 amp main breaker, which brought down the entire place. It blew at about midnight and was down until 4am because that’s the soonest that they could get a licensed master electrician out there to tag the system out, install the replacement breaker that they keep on hand (because these things aren’t exactly carried in home depot) and

- CiHost Dallas: Sites suddenly inaccessible. What happened? Apparently the DNS servers crashed. Both of them. A little badgering of the lead sysadmin later got him to inadvertantly admit that the servers had crashed to the floor, and they’d be back up as soon as the got them back on the table and got them started again.

- CiHost Chicago: Dallas is bad enough, but if you had servers in Chicago, you’d no longer have them. This thread at WebhostingTalk.com details a sordid story of an inside job and some nice corporate fibbing — to the extent that their data center was robbed not once, not twice, but possibly as many as six separate times.

And stories that I’ve heard but can’t verify:

- Delta’s Florida Datacenter: Big corporate datacenters have to maintain standby centers that they can transfer to in case of problems at their primary center. But before finding a good backup center, they should probably check the skies around their primary site. When Hurricane Andrew came through Florida, it blew a water tower over onto the main operations site for Delta’s flight routing and ticketing software. Obviously, this made a bit of a mess. As quickly as they could, Delta’s Engineers collected the soaking wet online backup tapes and, drying each one by hand on the long flight to the alternate datacenter in Portland, Oregon, brought the system back up. After the initial data load, the engineers were sleeping in shifts on the datacenter floor. (This story was related to me by a Fortix, now ViaWest, sales rep. I haven’t been able to verify it.)

If you’re bored like me on a Saturday, you can probably find more interesting stories of drama and dirt over at the WebHostingTalk Forums … it’s a good place to check up on a potential host before you do business with them.