Here’s a few examples.

MySQL

Theoretically, you could recover anything you needed from the binary log, as long as you’ve got a good starting point and a good ending point. (This, by the way, is a good reason to flush the binary logs and take a backup on a regular basis.) What if your binary log’s corrupted, though? You need to fall back to a full SQL backup … which you’re doing regularly, right?

If your binary log is corrupted, any mirrors you are using that are based on that binary log are corrupted as well.

Case in point: I had a client with a very active, very large database… north of 15GB in InnoDB. The binary log hit a bug and corrupted itself. The backups were being done from that mirror so that they didn’t interrupt the main machine’s processing, but they only kept a few days worth, so we couldn’t use those backups to restore. The most recent un-corrupted dump from the main machine had been taken three months before. Luckily, the client had done some application-level backups to an XML format, and we were able to (laboriously) restore from that. It cost about $3,000 because they didn’t want to degrade their forum’s performance for a half hour every night and pay for an extra TB or storage or so to keep more than a few days worth of upgrades.

Servers

Scenario: Hard drive gets corrupted or dies. You need to get the machine back up quickly. You have a snapshot of the machine … but your snapshot is on the same storage as that machine unless you back it up somewhere else.

On top of that, storage requirements have been growing rapidly for servers. Where a linux server take less than 1GB, Windows 2008R2 can take up 20GB with system files alone. (In fact, if you plan to have any data on that server, or keep any logs, we’d recommend going with 40GB minimum for your C: drive.) It’s important to back that up to something that’s not on the same system disks.

Better yet, take a hint from the application-level backups — and back up your registry, configuration files, and data separately from the snapshot. We tend to use RSync for this role and put it in a rolling-backup mode with the –link-dest option to ease recovery.

VMWare

Same principle as above. Snapshots are usually stored in the same datastore. Datastore goes bye-bye, so do your snapshots.

There’s some great products out there that can really help with this issue. The one we use is VEEAM Replication and Backup. It can be used to replicate a snapshot to another VMWare cluster, or back up the datastore files at a consistent snapshot point and then copy them elsewhere all in one step. We use a two-step process — we keep them locally on the backup server and also transmit them to another datacenter across campus.

When using VEEAM with Windows, make sure that VMWare Tools is installed and that you enable the VSS integration. (You’ll also need to make sure that the administrative share option on the system drives are enabled, and that the appropriate firewall ports are opened.) This ensures that you’ve got a transactionally consistent backup snapshot.

Practice, practice, practice

The only way to make sure that you can recover from a disaster is to test recovering from a disaster. At least once a year, we practice recovering from a worst-case scenario. That means bringing up a new machine from scratch, re-implementing all of the options and configurations, and then restoring the data. Despite that kind of restoration being something that should never happen, it does — and practice gives you insights into how to improve the processes and turns a recovery operation from an expensive nightmare that sets back all of your other processes into something that you can execute quickly and professionally.

First, a security note with people who have their machines on a “public” network: You’ll want to disable or set a password for the ‘null’ default user for ipmi. This may be done for you in recent versions of dell firmware, but it isn’t on some older stuff. Also, IPMI v. 1.5 doesn’t do encryption by default; you’ll want to make sure you set an encryption key in the bios settings and then use that when you connect (with the lanplus interface, and I have no idea what the syntax is with the stonith resource agent…) or you’ll be sending your authentication information “in the clear”. Since you need to have ADMINISTRATOR permission to reboot the machine as far as I know, you will want to make sure you have this secured… or anyone who’s sniffing can power your machines off at will.

You can test the sequence by using ipmitool. ipmitool -H (ip address) -U (user) -a chassis power cycle should cause your server to reboot after you enter the password. Please test this first.

Now all you need to do is add a configuration to your cluster. Using the pacemaker clm command line tool, just go configure primitive (host)-stonith stonith:external/ipmi \
params hostname=(host) ipaddr=(ip address) user=(user) passwd=(password) interface=lan. Double check your stonith resource agents to be sure of the syntax. ( crm# ra meta external/ipmi stonith )

You’ll want to add constraints (out of the scope of this document, since it depends on your cluster design) so that a host won’t have it’s own stonith running on it. Although I’m thinking that a game of stonith-based russian roulette would be kinda fun as a computer art installation someday.

When you want to check in /etc/logwatch/conf/logwatch.conf. Your working directory is /etc, but /logwatch and /logwatch.conf are not yet added. You don’t want to import everything under /etc/logwatch/scripts nor do you want to import anything under the /conf directory besides logwatch.conf.

svn clients >= version 1.5 have a really cool new feature. From an existing working directory, you can run svn add logwatch/conf/logwatch.conf --parents instead of first having to add logwatch with the -N option, conf with the -N option, and then finally logwatch.conf, … makes for a decent time savings.

Problem: Audio doesn’t work when the machine is docked in a Dell dock.
Solution: You need to enable the IEC958 Switch in your Volume Control application.

HowTo:

1. Click Computer, More Applications, and Voume Control
2. Click Edit, Preferences, and check IEC958. Then click Close.
3. Switch from the Playback tab to the Switches tab, and check the IEC958 checkbox.

I also clicked the IEC958 Default PCM box on my OpenSuSE 11 machine; this gave me control over the dock’s specific output port as opposed to the headphone port on the laptop.

The SCPlugin available via the Tigris.org website is -OK-, but not great. There’s some things that it does that just don’t feel right… I’ve just never gotten used to using a GUI (even TortoiseSVN) to manage repositories. Plus, my Vim plugins don’t work with SCPlugin and I’d rather be able to control things without switching from terminal windows into gui windows.

Yes, I -am- old school …. but I know there’s others of me!

To install up-to-date versions of the svn command-line client, just follow these easy steps.

1. Install X11 from your OSX DVD. It’s necessary to have X11 installed before you install xCode so that it pulls in the X11 sdk.
2. Download and install XCode 3.0 if you haven’t already.
3. Download and install MacPorts and follow their directions until you can run #4…
4. … sudo port -d selfupdate. The -d flag tells it to output debug information, which gives you a better idea of what went wrong if it errored out.

   Now, I ran into a small problem with the next step. You SHOULD just be able to sudo port -d install subversion… but it errored out because I didn’t have awk installed, or some other dependency was missing. At this point, I ran sudo port install gawk and then sudo port deps subversion … I installed the first few dependencies by hand, and after that it let me run sudo port install subversion and it safely installed the subversion client.

And voila! You should be able to do things like you normally would with subversion from the command line in linux. Maybe it’s just how *I* learned, but I really do prefer working with subversion repositories via the command line.

When you’re configuring Squirrelmail, you need to make sure that it does smtp authentication. It doesn’t seem that the version that comes with any way to that that *I* could easily find. However, when you set up your config_local.php in squirrelmail (which with the RPM installs in /etc/squirrelmail/config_local.php), set up a user for smtp auth with it’s own password… example:

$smtp_sitewide_user = 'smtpauth@katzke.net';
$smtp_sidewide_pass = 'somepassword';

… and now Squirrelmail will be able to authenticate to send email!

Alternately, you could set email sent from localhost to always be allowed without having to authenticate, but this will still work if your mail server and webmail server are on different machines.

In CentOS, the default Postfix package doesn’t have MySQL support built in. If you’ve got the priorities plugin installed, you’re either going to need to disable it or otherwise work around it so that you get the one from the centosplus repo. Other than that, it’s been made about as easy as it can be… just keep in mind as you’re reading this tutorial that I really loathe administering mail servers and consider it to be a quite onerous chore that’s been made even more onerous by spammers and hackers and script kiddies and what have you.

You’ll need to install:

yum install gcc postfix clamav mysql-server mysql-devel spamassassin
 dovecot php php-mbstring php-mysql rpm-build

. There’s no RPM for Postfixadmin, but it’s available from the project’s site on Sourceforge.

And now the fun begins. This howto assumes that you have a decent level of knowledge and skill setting up services that run on Linux.I’ve included my configuration files where appropriate, please note you’ll need to establish your own files associated with mysql, since the main purpose of those files is authentication. Please read the documents on how to do this. I’ve even linked to the correct pages.

Since Dovecot and Postfix hang off of postfixadmin, let’s get that installed first. Download the tarball, untar it somewhere (I use /opt/postfixadmin/), and point apache at it… basically, just add this into your virtual host container on the server:

Alias /pfadmin/ "/opt/postfixadmin/"

        Order allow,deny
        Allow from all
        DirectoryIndex index.php

Don’t forget to set up a database. I’m assuming you’re using mysql; as root:

mysql> create database postfix;
Query OK, 1 row affected (0.01 sec)

mysql> grant all on postfix.* to postfix@localhost identified
 by 'passwordgoeshere';
Query OK, 0 rows affected (0.03 sec)

Change the settings in /opt/postfixadmin/config.inc.php to reflect your server’s setup. If you are using MySQL >= 4.1, don’t forget to change the database type to mysqli… you’ll get much better performance. To finish setup, head to http://www.yourserver.com/pfadmin/setup.php and follow the prompts. You’ll want to have already pointed the domain’s MX at this IP address — that’s outside the scope of this tutorial, but rest assured that if you haven’t already done it, you’re going to want to go take a nice long nap … say, for a day or two … and come back when it resolves. Don’t add a domain or a mailbox yet.

Postfix

On to Postfix. Postfix is the mail transfer agent. In post office terms, Postfix is the guy at the central office who receives your mail from the truck or airplane, takes it out of the big bulk bag, filters it into your correct mailbox after applying any additional filters (i.e. “I’m on vacation, hold my mail”) to your individual account, and puts it in your P.O. box ready for you to pick up.

The official word on how to do postfix with mysql is available here. Go read that, I’ll wait.

Before proceeding with the Postfix configuration, make sure that the postfix version you have has MySQL support built in. To do this, run postconf -m and verify that mysql appears in the list. If mysql isn’t there, uninstall postfix using yum, download the version from the centosplus repository, and install that one manually.

To get postfix working, you’ll need to tell it to use mysql virtual maps for it’s user and domain tables. I’ve posted my configuration files below to make it easy on you. There’s a decent walkthrough in the postfixadmin DOCUMENTS folder.

• master.cf

main.cf

A few things of note:

• Note in postfix’s master.cf that I have ports 25 and 587 open. 587 is known as the alternate SMTP port, and us poor peons in the US with horrible ISPs that block port 25 for some misguided reason need to use it to pass mail to our servers.
• The virtual_uid_maps and virtual_gid_maps need to be set to a user that postfix has access to (the postfix user is fine, but I created an extra user called vmail.) You will need to create /var/spool/mail/vmail and chown it over to the group ID and user ID that you’ve set in the configuration file.
• Note the smtpd_auth_type and smtpd_auth_path settings — these are important to make sure that you are not hosting an open relay. Postfix can use dovecot instead of saslauthd — there’s no reason you should have to run two authentication mechanisms… is there? Here’s documentation on sasl with dovecot.
• Note that we also are using a static user ID of 502 for the vmail user that owns the virtual mailboxes — this MUST be set in both the postfix and dovecot mail configurations! Create a vmail user by calling `adduser -s /sbin/nologin vmail` as root, and then `addgroup vmail` — cat /etc/groups and /etc/passwd to get the user and group IDs, and set the group ID definitions in postfix (look for the lines that say static:###) and dovecot.

Dovecot

The next step is to get Dovecot configured. Dovecot is the part that allows users to authenticate to the mail server and to get the mail from their mailbox. In post office terms, it’s the authentication mechanism (key or combination) that lets you into your private P.O. box.

Again, there’s a DOVECOT.txt in the postfixadmin/DOCUMENTS directory that will tell you what settings you need to change, but I’ve attached my file below. In round terms, what you need to do is set up the pop3 and IMAP servers for the appropriate locations, tell them to look in the mysql tables for the authentication information, etc. Please note that Dovecot changed it’s configuration file schema, so default_mail_env is now mail_location and a few other details. Depending on what version of dovecot you’re using (I’m using 1.0-1.2.rc15, which is the latest from centos) you might have to make some changes to the configuration files below. There some very good information in the Dovecot wiki that you want to read before we go any farther.

/etc/dovecot.conf

Securing / Opening things up

With both of those servers running, make sure you don’t have any errors in /var/log/maillog. Then make sure that your firewalls are open on ports 25, 110 and 143. (This tutorial doesn’t cover SSL/TLS. I’ll cover it at a later date — but I recommend that you go from here and follow the very easy instructions at both the postfix and dovecot websites to configure it yourself.)

With both dovecot and postfix running with users from the database, we should have a secure and working server. Let’s test it. Before we start testing, you’re going to need a hash of your username and password… this is the virtual mailbox you set up when you were setting up postfixadmin. The command to get this hash looks like:

perl -MMIME::Base64 -e ‘print encode_base64(“\000foo\@foobar.com\000foobar“)’;

… replace the first green part with the email user’s name, then the second green part with the virutal domain we’re handling mail for, and the third green part with the user’s password.

Which would result in a hash of:

AGZvb0Bmb29iYXIuY29tAGZvb2Jhcg==

SMTP first. Open up a shell and telnet to your mailserver on port 25 or 587. You should be able to hold a session that looks something like this:

:~ karlkatzke$ telnet mailserver.com 587
Trying xx.xx.xx.xx...
Connected to mailserver.com.
Escape character is '^]'.
220 mailserver.com ESMTP Postfix
ehlo mail.foobar.net
250-mailserver.com
250-PIPELINING
250-SIZE 10240000
250-ETRN
250-AUTH PLAIN
250-AUTH=PLAIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
auth plain foobar
535 5.7.0 Error: authentication failed:
auth plain AGZvb0Bmb29iYXIuY29tAGZvb2Jhcg==
235 2.0.0 Authentication successful

At this point you could send mail and see if it actually goes through. Google it if you need to figure it out, since I’m not going to really cover it much here. Let’s test Dovecot some though. Telnet to port 110 on your mailserver, and let’s use the same password hash to do something similar. (Note that you’ll have to have sent an email to the user first in order for the mailboxes to be created, so if you haven’t, do so!)

:~ karlkatzke$ telnet mailserver.com 110
Trying xx.xx.xx.xx...
Connected to mailserver.
Escape character is '^]'.
+OK Dovecot ready.
user foobar@mailserver.com
+OK
pass foobar
+OK Logged in.
list
+OK 2 messages:
1 601
2 486
.
quit
+OK Logging out.
Connection closed by foreign host.

Ok, you can receive mail, send mail, and everything else. If that’s all you want, you’re done. Just to make sure, head over to the abuse.net relay test and run a test against your server to make very sure you don’t have any open relays that could be used by spammers. Having open relays on your server WILL get you blacklisted almost immediately.

Spam Assassin

The easiest way to get spamassassin running is to follow this tutorial. Just remember that if you’re running SMTP on port 587, run spamassassin there too. Don’t forget to start and chkconfig the spamasssassin service before you restart postfix.

Squirrelmail

Install squirrelmail via yum. It’s easy — just install it, restart apache, and you’re good to go. Update: See this blog article for SMTP auth.

Wrapping up

This configuration, it should be noted, is the one that works for me. It may not work for you. It may not also follow the best security principles — I would especially recommend getting SSL/TLS working for any sort of deployment in the workplace. If you identify any security holes or problems, please post a comment below or email me via the contact page and I’ll update the tutorial. Also, please note the date the tutorial was published and check documentation files accordingly. Reading this tutorial and/or downloading the files is no replacement for actually reading the documentation.

If you don’t know basic OOP (Object-Oriented Programming) principles, now would be a good time to go do some reading on them. We’re going to make heavy use of inheritance, as well as overriding. Just keep in mind that PHP is not a strongly typed language.

Let’s get started. For authentication, we’re going to build the form the old fashioned way so that you can get a good example for ‘how it was done’ versus ‘how smoothly it can be done’ with Zend Framework. Let’s create the login form. In /app/views/scripts/index/, create login.phtml — do note the .phtml extension on it. To keep us moving right along, here’s the code for it.

<? if(!empty($this->message)): ?>
    <?=$this->message?>
<? endif; ?>
<form name="login" action="index/login" method="post">
Login: <input type="text" name="login" /><br />
Password: <input type="password" name="password" /><br />
<input type="submit" name="submit" value="submit" />
</form>

Now we’ll head back to your controller. If you look at the work you did in the form, you’ll see that the index controller’s login action will be responsible for both displaying and processing this form.

First, to get the form to display, you’ll need to add just a blank login action to your IndexController. For security purposes, I’ve cleared out the ‘list of users’ that was in yesterday’s indexAction() (we don’t really want to give an attacker a list of users to try, do we?) and replaced it with a call to $this->_forward() to direct users that come to the index page to the login. Later on, when we’re checking for authenticated users, we’ll replace this with a conditional.

class IndexController extends Zend_Controller_Action {
  function indexAction() {
     $this->forward('login');
  }
  function loginAction() {
 
  }
}

Now, if you visit /index/login in your browser, you should get the login form. Let’s add some logic to make it do something. We’re going to use Zend_Auth’s database methods for this.

function loginAction() {
  $params = $this->_getAllParams();
 
  if(!empty($params['submit']) && $params['submit'] == 'submit') {
        $auth = Zend_Auth::getInstance();
 
        $adapt = new Zend_Auth_Adapter_DbTable(Zend_Registry::get('db'));
        $adapt->setTableName('users');
        $adapt->setIdentityColumn('id');
        $adapt->setCredentialColumn('password');
        $adapt->setCredential(sha1($param['password']));
        $adapt->setIdentity(htmlspecialchars($param['login']));
 
        $result = $auth->authenticate($adapt);
        $this->initView();
        if($result->isValid()) {
	          $storage = new Zend_Auth_Storage_Session();
                  $storage->write($adapt->getResultRowObject(array('id')));
	          $auth->setStorage($storage);
 	          $this->view->message = 'Login Successful!';
        } else {
	          $this->view->message = 'Invalid login. Please try again.';
        }
    } 
}

There’s a few very important things we want to note here.

1. Zend_Auth is written in a Singleton Pattern.
2. We make use of the database handle that we stored in the registry in the front controller.
3. We’re encoding the password in the code before we send it to the database. In the database, the password is already encoded in sha1 format.
4. We’re setting the identity for later retrieval.
5. We pass a message back to the user via the view script.

You know what? I know I promised that we’d get into some fancier stuff using Zend_Form and Zend_Mail, but I think that’s enough new concepts for one day. We’ll extend the tutorial into next week and spend tomorrow working with Zend_Form and Zend_Mail.

Happy coding!

The general procedure when setting up Zend_Db is to create a bunch of objects that extend Zend_Db_Table_Abstract. When defining an object, you can define what table it connects to, what the primary key is, and any foreign keys that relate to it. Upon instantiating the object in your code, you can use it to build queries against it’s related table, which returns a collection of row sets for direct use in your applications. Today’s tutorial assumes that you’re very comfortable with looping and other control structures in PHP.

Configuration File

You should have a MySQL database accessible to your webserver. Knowing how to use MySQL is outside of the scope of this tutorial but the basic gist of what you need to do is — create a database, create a user and password for that database, and insert the SQL to create the tables.

ZFApp (Day 3) SQL

When you’ve collected that information, let’s write a configuration file. This file will go in app/config/db.ini.

[dev]
database.params.host=localhost
database.adapter=pdo_mysql
database.params.username=foobaruser
database.params.password=foobar
database.params.dbname=zfapp

Save that file, and then let’s go tell the front controller how to access the zfapp database.

Front Controller

The first thing we need to do is go back and visit our friend the Front Controller. It needs some setup if we’re to use Zend_Db. Let’s open up index.php and add a few lines. We’ll be using Zend_Config_Ini to load up the configuration file. Note the ‘dev’ – that tells us to load up the area after the [dev] in the above .ini file. I use this to tell the difference between my dev and live configuration, since that keyword is set by a SetEnv statement in the Apache configuration file.

$dbconf = new Zend_Config_Ini(BASEDIR.'/app/config/db.ini','dev');

Now that we’ve got a handle to the configuration file, we’ll use Zend_Db’s factory method to create a DB Connection.

$db = Zend_Db::factory($dbconf->database);

That’s it. That’s all we need to do. $db is now a database handle. Two lines of code! Zing! The only other things that we’ll find helpful here is setting this DB handle as the default handle for tables. I also register the DB handle with the registry so that I have an easy way to get it back later if I want to do something deep and complicated with the db.

Zend_Db_Table_Abstract::setDefaultAdapter($db);
Zend_Registry::set('db',$db);

Using the DB

Tomorrow, we’ll really get into using the database for something, but for today let’s just finish the prep work and make a basic use of our new toy.

First, we need a model to access. Make sure you’ve got the models on your include path in your front controller:

set_include_path(ini_get('include_path').PATH_SEPARATOR.BASEDIR.'app/models');

Now, we’ll create our first model. A model is a basic file that just tells the database driver where a table is, and provides some defaults. In /app/models/users.php, we’ll want to extend a Zend_Db_Table_Abstract as follows:

class Users extends Zend_Db_Table_Abstract {
   protected $_name = 'users';
   protected $_primary = 'id
}

… and that’s it. Since you included the models directory in your include path, all you need to do is instantiate a Users module anywhere in your application, and you can access the users table via the db handle you created in your front controller and assigned as default. Pretty neat, huh?

Ok, let’s select some rows out.

Head over to the /app/controllers/IndexController.php that you made yesterday. In the indexAction() function, place this code:

public function indexAction() {
  $this->initView(); // Initialize the view object for this module
  $user = new Users(); // Users Table Object. 
  $rowset = $user->select(); // Created a default 'select all' query
  $list_of_users = array();
  foreach($rowset as $row) {
    $list_of_users[] = $row->id;
  }
  $this->view->userlist = join($list_of_users,', '); // Concatenate the list of users and make it available to the view. 
}

And we’ll head over to the view script you created yesterday … /app/views/scripts/index/index.phtml … and add in a line to echo that userlist. Don’t forget to add the carat-questionmarks here, of course.

 echo $this->userlist;

… and as Emeril says, Bang! You’re reading from your database.

Tomorrow we’ll do some ‘heavy’ lifting — we’ll set up Zend_Auth to authenticate users from the database, a Zend_Form to allow users to sign up, and a Zend_Mail to send users their passwords when they’ve forgotten it.