Accomplished; The Spring Break 09 Edition

Any IT guy is familiar with the “one or two times a year when we’re allowed to turn things off” problem. The higher ups insist that email (or insert other service here) always needs to be available, and anything that causes a delay in email (or other service) is met with protests. And god forbid that we need to schedule downtime to fix something…

I’m fortunate enough to work at a public university these days that has regular staff holidays. During the staff holidays, the university is officially closed — and us techies are permitted to do periodic maintenance. I thought it would be entertaining to write up what we did this time around.

• We’ve been installing APC 7901 switched PDUs in all of our racks, which is great for servers that run on two power supplies, but that’s rare indeed for network switches and other devices. As a result, we installed an APC AP7752 (or similar) in each of our racks. This required a lot of rewiring and we had to take the main datacenter rack down completely for a brief period.
• We moved our AC&NC JetStor 516F fiber to a new QLogic SANBox 4gb/s switch.
• We have two machine rooms with equipment; the central campus site and our building. In the central campus site, which houses a university-specific installation of a streaming media machine, we added another AC&NC JetStor 416F to our ZFS pool.
• Using g4u (aka Ghost For Unix), we cloned two potentially dead or dying machines on nine year old hardware up to an FTP server. Hopefully, we’ll get these up in a virtual machine next week and then rsync a delta over so that we no longer need to worry about hardware failure.
• We finished rolling out our APC 5616 KVM to the main datacenter. It’s nice to be able to access console on my machines from home no matter what machine I’m using. They work great on Mac, PC, and Linux.
• Some assorted firmware upgrades, flashing, and configuration.
• We finished installing APC AP9618 cards in all of our UPSes.

The pending release of MySQL-Proxy 0.7 means that I should be able to get my mysql cluster done in the next week. When we get a new rack full of equipment for the media server, we should be able to grab the old hardware to expand our Xen application server cluster a little bit, which will give us the room we need to roll out OpenSuSE 11.1. Hopefully before the end of this calendar year…