Leave it to the Robots: Automating Linux Systems Administration
This article is derived from /etc/hosts, Jon's column in Web Hosting magazine.
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
When you automate system administration, you can have both security and time for the rest of your job. If your machine room is your factory, then good automation lets you replace your assembly-line workers with robots. For better and worse, you can't entirely automate yourself out of a job, but it can't hurt to try.
The most basic sort of security automation tool is the hardening script. It should harden the system by doing the things you should do to tighten security on a system after you install it. You can think of hardening scripts as a sort of automated security checklist.
Unfortunately, there are as many hardening scripts as there are operating systems to lock down. (More, actually.) This is because each OS is just different enough that it's difficult to have one script handle everything. (Because of space limitations, I'll talk about only a few Linux flavors. If you are running something different, then do a search for the words "hardening script" and the name of your favorite Unix flavor in your favorite search engine. But keep reading, because hardening scripts are only half the story.)
SuSE Linux has a hardening script available that will tighten up security on your system with minimal user intervention. This script is available at http://www.suse.de/~marc and is currently at revision 2.4, which can handle any release of SuSE Linux from 5.3 through 6.4. Unfortunately, because it doesn't ask you many questions, you don't have a whole lot of flexibility. It does, however, cover most of the bases for basic security: harden_suse can disable unnecessary system accounts, remove SUID and read bits from files and directories, disable inetd and the services it spawns, improve login and password security, disable world-writable files, display legal disclaimers at login, and provide a strong configuration for SSH.
Mandrake Linux version 7.x offers MSEC (Mandrake SECurity), a tool that provides six user-selectable security levels for your system. It offers many of the same things that SuSE's tool does, as well as a number of other checks, and it can run the tool nightly to ensure that your mistakes never become security holes. (I know, you never make mistakes. Even so, you might want to run this.)
Although Red Hat does not offer a comparable tool, the Bastille Linux Project has produced a thorough hardening script for Red Hat Linux 6.x and Mandrake Linux 6.x that asks you many questions to help you best secure your system. (Disclaimer: I'm lead coordinator for the Bastille Linux Project, but I still think it's one of the best tools available for Linux.) Bastille is available at http://www.bastille-linux.org/.
Bastille Linux does pretty much everything these other tools do and more, and is even more granular: Just about every feature of the script can be turned on or off independent of the others. Furthermore, once you decide what you want, you can run the same set of answers through every one of your machines. But don't take my word for it: Anthony Santoni, administrator at Orisis.net—a Linux-based Maryland Web hosting company—says "If you do not know everything about your services, there is always a chance of leaving a hole open. That's where Bastille Linux comes in. This product is perfect for implementing "best practices" security on your Linux systems."
Hardening scripts are good because, without a written checklist (and even with one), people tend to make mistakes. Robots are brutally consistent—any episode of Star Trek will tell you that. But hardening scripts concentrate on a small subset of system administration. Good security across an enterprise depends upon consistent administration after the machines have been installed.
One solution to ensure consistent administration, including application of security updates, is CFEngine, available online at http://www.iu.hioslo.no/cfengine/.
CFEngine provides a language in which you can write rules that will be automatically applied across all your systems, including different rules based on host names or groups of hosts. This lets you have one central set of configuration files to handle dozens—or even hundreds—of machines. (I know, because I use it at my day job to handle hundreds of machines.) It has advanced text-editing features, so that you can write rules to modify your configuration files in complex and subtle ways, and CFEngine can run any other script you throw at it.
CFEngine isn't the only tool you need after you've installed your boxes. The second dullest job for sysadmins, after acting like a help desk, is reading logs to look for suspicious activity. I take it back: Checking logs for signs of intrusion is even duller than user support. Because even most script kiddies are smart enough to alter logs once they've cracked a system, you have to do this consistently and frequently. Psionic Logcheck helps keep log-reading to a minimum by flagging suspicious entries for you to read. Santoni says, "There is no automation in existence that will make it so you never have to check your logs. What you can do, however, is cut down on the entries that you do have to examine. Logcheck will allow you to create custom filters so that the logs you do end up examining are the ones that are relevant." More work for robots means less work for humans, but reading logs is certainly one job I'd like to be automated out of.
Previous versions of Psionic Logcheck had a funny license that kept people away, but version 1.1.1 is under the GNU GPL, so you can run it without trying to figure out if you're allowed to do so. It's available on the Web at http://www.psionic.com/abacus/logcheck/.
Psionic Software has a number of other useful security tools available on their Web site as part of the abacus project at http://www.psionic.com/abacus/.
I suggest you take a look at PortSentry and HostSentry as well, available at the preceding site. PortSentry identifies "Port scans," used by crackers to determine what software you're running, to both deny them the ability to finish the scan and notify the administrator (that's you). HostSentry tries to identify deviation from users' normal login habits, to warn you if an account has been compromised.
After reading logs, the next least fun task an admin can do is to see if crackers have altered any of your configuration files. A plethora of tools exist that can make checksums of your files and compare them to values stored in a database. Classically, the favored program was Tripwire. Version 1.2 was available for free under a restrictive license; however, Version 2 is available from the Tripwire home page at http://www.tripwire.com/; although it is under a very strict license regarding commercial use, the newest Linux-only version is now available under the GPL as well.
One replacement for Tripwire is AIDE (Advanced Intrusion Detection Environment). It attempts to do everything that Tripwire does and, compared to the no-cost version of Tripwire, AIDE does a good job. (Unlike Tripwire 1.x, AIDE is under active development.) The commercial version of Tripwire, though, has one major advantage: It is able to cryptographically sign the database, so that you need not keep it on write-protected media or otherwise offline. (AIDE and older versions of Tripwire could be defeated by crackers if they re-ran the tool to create the database, and aside from a cryptographic signature, the only way to do that is to keep the data where crackers can't change it.) The AIDE home page is http://www.cs.tut.fi/~rammer/aide.html.
Speaking of cryptographic signatures, many software packages and announcements are signed with PGP. PGP, which is short for "Pretty Good Privacy" (Laugh. It's a joke.), is the de facto standard for digital signatures and cryptography. The best site to start with for PGP is the PGP International home page at http://www.pgpi.org/.
In addition to signing files and documents with PGP, it can also be used to encrypt your password lists safely. Because it's based on public-key cryptography, where any user can encrypt a message but you need a key to decrypt it, you can safely leave a file with all of your passwords world-readable. (Just don't leave your keys sitting around!) PGP files can be encrypted so that multiple keys can decrypt them; that way, every admin can have her or his own key to read the files without having everybody know a password. The security is good enough that you need not worry about crackers getting hold of the list: Even in a relatively insecure mode, it should take several thousand computers several months of coordinated effort to decrypt the file without the key.
Lately, GNU Privacy Guard (GPG), a PGP-compatible tool available under the GPL, has started to supplant PGP in some organizations. The GPG home page is at http://www.gnupg.org/.
These are far from the only useful tools to automate and guarantee system security. There are thousands out there. Go ahead and try all you want, but settle on something and standardize on it. There's nothing worse than having a pile of robots without the ability to swap parts.
About the Author
Jon Lasser is a Unix systems administrator and the author of Think Unix (Que, 2000), an introduction to Linux and Unix for power users, and a columnist for Web Hosting magazine. Jon has been involved with Linux and Unix since 1993 and is a project coordinator for Bastille Linux, a security package for various Linux distributions. He lives in Baltimore with his wife, Kathleen, and their three cats.