Centralized Logging
It’s been quiet around here for the past few months. I’ve been busy working on central syslogging of all kinds of different things, and the project’s been killing my free time because it’s downright interesting to put together this many things all focused on one task.
The end goal of the syslogging project is for me to be able to tell an auditor or supervisor exactly when a particular user logged into a particular machine or service, down to the second, and grouped by months or other information. There’s actually a lot of uses for this information that aren’t apparent at first, from demonstrating increased usage of resources to justify upgrades through forensic logging of an intrusion or internal compromise.
A bunch of things end up being required. First, a central gathering point of logs from different types of machines and services needs to be generated. Second, it’ll require that these logs be parsed and processed into discreet information — i.e. plucking the date/time, user ID, and machine ID/name out of the raw logs. This processing can require a lot of CPU power. Third, the information needs to be stored in an index so that you can quickly pluck summaries and drill-down detail as needed.
I’m still on the first one. Until you dig through it, you don’t realize how few linux applications actually use the syslog functionality, even with the much more modern syslog-ng. For instance, Apache can pipe error logs, but not access logs (or even filtered access logs pertaining to authentication) through syslog. MySQL doesn’t use syslog at all except for messages when mysqld starts or stops unexpectedly. You can tee or pipe some of this output through the /usr/bin/logger utility, but this can be problematic load-wise.
The solution I’ve arrived at is writing a daemon that will monitor the logfiles and send chunks of them either directly to a remote syslog-ng or pipe them through the local syslog first and then through a filter and log statement that’ll transmit them to the remote syslog. I thought of only sending them over as they rotate using the logrotate script, but that reduces the effectiveness of remote logging when doing intrusion forensics — hard to wipe a brute force attack from the logs on a local machine that got pwnt when the logs are also recorded in real time elsewhere, eh? Keeping server load low is a huge priority.
