The web servers are *mostly* running Drupal installations, backed by MySQL. Drupal is a fairly hefty CMS and for a long time we’ve used several physical servers fronted with Pound. This is very obviously expensive — two servers alone from Rackspace cost $1500/mo. While we could do it cheaper in the cloud, we weren’t comfortable with the faceless/nameless cloud’s ability to meet the uptime requirements in our SLA, nor were we comfortable with the ability of the cloud vendors to provide the support levels that were required by our customers.

Building this out —

First, I decided to use VMWare ESXi and vCenter to fulfill the high availability requirement. Since this was going to be a remote installation, I needed to be able to quickly and effectively diagnose problems and needed to have support due to the nature of the clients we were hosting. While I could’ve gone with Xen and SLES’s HA package, the license for a Essentials Plus version of VMWare was actually cheaper than the necessary bits from Novell and Citrix… and unlike those products, VMWare supports failover and migration between hosts out of the box instead of requiring weird Pacemaker/OpenAIS cluster configurations.

Second, after pricing comparable hardware solutions, we ended up going with Dell. We priced HP, Sun/Oracle, and SuperMicro systems. The SuperMicro systems were closest in cost, but ended up failing on support. Compared to Dell’s Same Day/Next Business Day offerings, SuperMicro’s “go through your reseller” strategy falls way short of business expectations. Also, one neat thing about the Dell MD3200/3220 is that it can handle up to 8 hosts connecting via 6 gb/s SAS at once to the same LUN. Since we don’t expect to have anywhere near that number of hosts in the short term, we can effectively use VMFS. In the longer term, if we outgrew that number of hosts without upgrading to a fiber SAN, we could share VM partitions out via NFS and offload our highest concurrent storage user, MySQL, to it’s own storage.

Third, for the gateway/router/etc, we chose the Cisco 5505 ASA Security + device. Two of them will fit in a 2u rack mount kit (actually 1.5, which is ideal for the crowded top of the colocation cabinet we’re in) and each has 8 1gb/s ports. They’re pretty full-featured little boxes (note that you MUST buy a support contract if you want a warranty past 90 days though!)… sort of like a home cable modem, they’ll gateway, NAT, provide DHCP, and other basic networking functions out of the box … unlike your average home gateway, though, they support active/standby failover, a ton of diagnostics, routing, and shaping, and speak the same asa/pix language that we’re all familiar with. They also do VPN via everyone’s favorite Cisco VPN client. My other choice, besides the comparable Juniper Networks equipment, was to build my own system using BSD & CARP or WRT-DD… while both of those might have been cheaper for the client, they didn’t fulfill the support requirement.

Last, we needed to provide an onsite and offsite backup capability. With 15k RPM disks running up to $1k, the best solution I could find was a 2TB Buffalo TeraScale NAS. I configured it as a 3-drive RAID5 with 1 hot spare.

When we were done, the total configuration looked something like this:
- 3x Dell R610 w/ dual X5620 + 32gb RAM + no drives (boot from internal USB)
- 1x Dell MD3200 w/ 2 MM, 7x 300gb 15k drives
- 1x 2tb Buffalo TeraScale III NAS
- 2x Cisco ASA 5505 + rack mount kit
- VMWare Essentials Plus

For a few of the infrastructure bits, we purchased a license for Redhat EL 5.5. For most of the webheads and other ‘disposable’ VMs, we use CentOS 5.5. VCenter runs in a Win2k8r2 VM.

In CentOS, the default Postfix package doesn’t have MySQL support built in. If you’ve got the priorities plugin installed, you’re either going to need to disable it or otherwise work around it so that you get the one from the centosplus repo. Other than that, it’s been made about as easy as it can be… just keep in mind as you’re reading this tutorial that I really loathe administering mail servers and consider it to be a quite onerous chore that’s been made even more onerous by spammers and hackers and script kiddies and what have you.

You’ll need to install:

yum install gcc postfix clamav mysql-server mysql-devel spamassassin
 dovecot php php-mbstring php-mysql rpm-build

. There’s no RPM for Postfixadmin, but it’s available from the project’s site on Sourceforge.

And now the fun begins. This howto assumes that you have a decent level of knowledge and skill setting up services that run on Linux.I’ve included my configuration files where appropriate, please note you’ll need to establish your own files associated with mysql, since the main purpose of those files is authentication. Please read the documents on how to do this. I’ve even linked to the correct pages.

Since Dovecot and Postfix hang off of postfixadmin, let’s get that installed first. Download the tarball, untar it somewhere (I use /opt/postfixadmin/), and point apache at it… basically, just add this into your virtual host container on the server:

Alias /pfadmin/ "/opt/postfixadmin/"

        Order allow,deny
        Allow from all
        DirectoryIndex index.php

Don’t forget to set up a database. I’m assuming you’re using mysql; as root:

mysql> create database postfix;
Query OK, 1 row affected (0.01 sec)

mysql> grant all on postfix.* to postfix@localhost identified
 by 'passwordgoeshere';
Query OK, 0 rows affected (0.03 sec)

Change the settings in /opt/postfixadmin/config.inc.php to reflect your server’s setup. If you are using MySQL >= 4.1, don’t forget to change the database type to mysqli… you’ll get much better performance. To finish setup, head to http://www.yourserver.com/pfadmin/setup.php and follow the prompts. You’ll want to have already pointed the domain’s MX at this IP address — that’s outside the scope of this tutorial, but rest assured that if you haven’t already done it, you’re going to want to go take a nice long nap … say, for a day or two … and come back when it resolves. Don’t add a domain or a mailbox yet.

Postfix

On to Postfix. Postfix is the mail transfer agent. In post office terms, Postfix is the guy at the central office who receives your mail from the truck or airplane, takes it out of the big bulk bag, filters it into your correct mailbox after applying any additional filters (i.e. “I’m on vacation, hold my mail”) to your individual account, and puts it in your P.O. box ready for you to pick up.

The official word on how to do postfix with mysql is available here. Go read that, I’ll wait.

Before proceeding with the Postfix configuration, make sure that the postfix version you have has MySQL support built in. To do this, run postconf -m and verify that mysql appears in the list. If mysql isn’t there, uninstall postfix using yum, download the version from the centosplus repository, and install that one manually.

To get postfix working, you’ll need to tell it to use mysql virtual maps for it’s user and domain tables. I’ve posted my configuration files below to make it easy on you. There’s a decent walkthrough in the postfixadmin DOCUMENTS folder.

• master.cf

main.cf

A few things of note:

• Note in postfix’s master.cf that I have ports 25 and 587 open. 587 is known as the alternate SMTP port, and us poor peons in the US with horrible ISPs that block port 25 for some misguided reason need to use it to pass mail to our servers.
• The virtual_uid_maps and virtual_gid_maps need to be set to a user that postfix has access to (the postfix user is fine, but I created an extra user called vmail.) You will need to create /var/spool/mail/vmail and chown it over to the group ID and user ID that you’ve set in the configuration file.
• Note the smtpd_auth_type and smtpd_auth_path settings — these are important to make sure that you are not hosting an open relay. Postfix can use dovecot instead of saslauthd — there’s no reason you should have to run two authentication mechanisms… is there? Here’s documentation on sasl with dovecot.
• Note that we also are using a static user ID of 502 for the vmail user that owns the virtual mailboxes — this MUST be set in both the postfix and dovecot mail configurations! Create a vmail user by calling `adduser -s /sbin/nologin vmail` as root, and then `addgroup vmail` — cat /etc/groups and /etc/passwd to get the user and group IDs, and set the group ID definitions in postfix (look for the lines that say static:###) and dovecot.

Dovecot

The next step is to get Dovecot configured. Dovecot is the part that allows users to authenticate to the mail server and to get the mail from their mailbox. In post office terms, it’s the authentication mechanism (key or combination) that lets you into your private P.O. box.

Again, there’s a DOVECOT.txt in the postfixadmin/DOCUMENTS directory that will tell you what settings you need to change, but I’ve attached my file below. In round terms, what you need to do is set up the pop3 and IMAP servers for the appropriate locations, tell them to look in the mysql tables for the authentication information, etc. Please note that Dovecot changed it’s configuration file schema, so default_mail_env is now mail_location and a few other details. Depending on what version of dovecot you’re using (I’m using 1.0-1.2.rc15, which is the latest from centos) you might have to make some changes to the configuration files below. There some very good information in the Dovecot wiki that you want to read before we go any farther.

/etc/dovecot.conf

Securing / Opening things up

With both of those servers running, make sure you don’t have any errors in /var/log/maillog. Then make sure that your firewalls are open on ports 25, 110 and 143. (This tutorial doesn’t cover SSL/TLS. I’ll cover it at a later date — but I recommend that you go from here and follow the very easy instructions at both the postfix and dovecot websites to configure it yourself.)

With both dovecot and postfix running with users from the database, we should have a secure and working server. Let’s test it. Before we start testing, you’re going to need a hash of your username and password… this is the virtual mailbox you set up when you were setting up postfixadmin. The command to get this hash looks like:

perl -MMIME::Base64 -e ‘print encode_base64(“\000foo\@foobar.com\000foobar“)’;

… replace the first green part with the email user’s name, then the second green part with the virutal domain we’re handling mail for, and the third green part with the user’s password.

Which would result in a hash of:

AGZvb0Bmb29iYXIuY29tAGZvb2Jhcg==

SMTP first. Open up a shell and telnet to your mailserver on port 25 or 587. You should be able to hold a session that looks something like this:

:~ karlkatzke$ telnet mailserver.com 587
Trying xx.xx.xx.xx...
Connected to mailserver.com.
Escape character is '^]'.
220 mailserver.com ESMTP Postfix
ehlo mail.foobar.net
250-mailserver.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-AUTH PLAIN
250-AUTH=PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
auth plain foobar
535 5.7.0 Error: authentication failed:
auth plain AGZvb0Bmb29iYXIuY29tAGZvb2Jhcg==
235 2.0.0 Authentication successful

At this point you could send mail and see if it actually goes through. Google it if you need to figure it out, since I’m not going to really cover it much here. Let’s test Dovecot some though. Telnet to port 110 on your mailserver, and let’s use the same password hash to do something similar. (Note that you’ll have to have sent an email to the user first in order for the mailboxes to be created, so if you haven’t, do so!)

:~ karlkatzke$ telnet mailserver.com 110
Trying xx.xx.xx.xx...
Connected to mailserver.
Escape character is '^]'.
+OK Dovecot ready.
user foobar@mailserver.com
+OK
pass foobar
+OK Logged in.
list
+OK 2 messages:
1 601
2 486
.
quit
+OK Logging out.
Connection closed by foreign host.

Ok, you can receive mail, send mail, and everything else. If that’s all you want, you’re done. Just to make sure, head over to the abuse.net relay test and run a test against your server to make very sure you don’t have any open relays that could be used by spammers. Having open relays on your server WILL get you blacklisted almost immediately.

Spam Assassin

The easiest way to get spamassassin running is to follow this tutorial. Just remember that if you’re running SMTP on port 587, run spamassassin there too. Don’t forget to start and chkconfig the spamasssassin service before you restart postfix.

Squirrelmail

Install squirrelmail via yum. It’s easy — just install it, restart apache, and you’re good to go. Update: See this blog article for SMTP auth.

Wrapping up

This configuration, it should be noted, is the one that works for me. It may not work for you. It may not also follow the best security principles — I would especially recommend getting SSL/TLS working for any sort of deployment in the workplace. If you identify any security holes or problems, please post a comment below or email me via the contact page and I’ll update the tutorial. Also, please note the date the tutorial was published and check documentation files accordingly. Reading this tutorial and/or downloading the files is no replacement for actually reading the documentation.

If you’re installing on CentOS, you’re going to want to

yum install elinks

, then run cpan as root and install

install Nagios::Plugin

… after that, you can install and add to your nagios command config. I usually run it in NRPE on remote hosts.

Unfortunately, no one has released 5.1 for RHEL 4 / CentOS. However, there is a 5.1.2 SRPM available in Fedora Core 5. The SRPM can be built using rpmbuild after making just one small change to the spec file.

Steps to build on CentOS:

1. yum groupinstall ‘Development Tools’
2. download the Fedora Core 5 php-5.1.2 SRPM from a mirror
3. rpm -ihv (the php SRPM)
4. edit the CFLAGS= line (~line 310) of the php.spec file – remove ” -Wno-pointer-sign”
5. rpmbuild -bb ./php.spec; you’ll need to fulfill some requirements, like having Apache, MySQL and a bunch of other libraries installed
6. cd /usr/src/redhat/RPMS/-arch-/
7. rpm -ihv (the php RPMs for all of the modules you need)

That should do it. That will provide you with a PHP 5.1.2 binary on CentOS. Please note that if you need support for this binary somewhere, you won’t get it from Redhat or the CentOS team, as they purposely haven’t released a more advanced version of PHP due to stability concerns. My personal opinion is that it’s a necessary and incredibly beneficial upgrade.

As a side note, Apache 2.2 is available in much the same way (it’s worth noting that you’ll need to build PCRE, APR, and APR-Utils RPMs from FC5 source as well), and contains a similar level of improvements — such as a stable and configurable mod_cache and an optimized prefork mpm.

Edit: Here’s my php.spec file. It’s got MSSQL enabled and I didn’t put in conditionals for it because we always need it, so you’ll just need to remove any references to MSSQL or PDO-DBLIB. It requires the freetds & freetds-devel RPM from Dag’s RPM repository… don’t try to build FreeTDS .64 on your own, it won’t work.

(Note: This entry was restored from archives and the spec file is unfortunately no longer available — or applicable.)