Security
The ease with which large monitoring systems can become large root kits makes it imperative that security is considered sooner, rather than later.
Because monitoring systems usually need remote execution rights to the hosts it monitors, it's easy to introduce backdoors and vulnerabilities into otherwise secure systems. Worse, because they're installed as part of a legitimate system, these vulnerabilities may be overlooked by penetration testers and auditing tools. The first, and most important, thing to look for when building secure monitoring systems is how remote execution is accomplished.
Historically, commercial monitoring tools have included huge monolithic agents, which must be installed on every client to enable even basic functionality. These agents usually include remote shell functionality and proprietary byte code interpreters, which allow the monitoring host carte blanche to execute anything on the client, via its agent. This implementation makes it difficult, at best, to adhere to basic security principles, such as least privilege. Anyone with control over the monitoring system has complete control over every box it monitors.
Nagios, by comparison, follows the UNIX adage: "Do one thing and do it well." It is really nothing but a task optimized scheduler and notification framework. It doesn't have an intrinsic ability to connect to other computers and contains no agent software at all. These functions exist as separate, single-purpose programs that Nagios must be configured to use. By outsourcing remote execution to external programs, Nagios maintains an off-by-default policy and doesn't attempt to reinvent things like encryption protocols, which are critically important and difficult to implement. With Nagios, it's simple to limit the monitoring server's access to its clients, but poor security practices on the part of admin can still create insecure systems; so in the end, it's up to you.
The monitoring system should have only the access it needs to remotely execute the specific plugins required. Avoid rexec style plugins that take arbitrary strings and execute them on the remote host. Ideally, every remotely executed plugin should be a single-purpose program, which the monitoring system has specific access to execute. Some useful plugins provide lots of functionality in a single binary. NSCLIENT++ for Windows, for example, can query any perfmon counter. These multipurpose plugins are fine, if they limit access to a small subset of query-only functionality.
The communication channel between the remotely executed plugin and the monitoring system should be encrypted. Though it's a common mistake among commercial-monitoring applications, avoid nonstandard, or proprietary, encryption protocols. Encryption protocols are notoriously difficult to implement, let alone create. The popular remote execution plugins for Nagios use the industry-standard OpenSSL library, which is peer reviewed constantly by smart people. Even if none of the information passed is considered sensitive, the implementation should include encrypted channels from the get-go as an enabling step. If the system is implemented well, it will grow fast, and it's far more difficult to add encrypted channels after the fact than it is to include them in the initial build.
Simple Network Management Protocol (SNMP), a mainstay of systems monitoring that is supported on nearly every computing device in existence today, should not be used on public networks, and avoided, if possible, on private ones. For most purposes involving general-purpose workstations and servers, alternatives to SNMP can be found. If SNMP must be used for network equipment, try to use SNMPv3, which includes encryption, and no matter what version you use, be sure it's configured in a read-only capacity and only accepts connections from specific hosts. For whatever reason, sysadmins seem chronically incapable of changing SNMP community string names. This simple implementation flaw accounts for most of SNMP's bad rap. Look for more information on SNMP in Chapter 6, "Watching."
Many organizations have network segments that are physically separated, or otherwise inaccessible, from the rest of the network. In this case, monitoring hosts on the isolated subnet means adding a Network Interface Card (NIC) to the monitoring server and connecting it to the private segment. Isolated network segments are usually isolated for a reason, so at a minimum, the monitoring system should be configured with strict local firewall rules so that they don't forward traffic from one subnet to the other. Consideration should be paid to building separate monitoring systems for nonaccessible networks.
When holes must be opened in the firewall for the monitoring server to check the status of hosts on a different segment, consider using remote execution to minimize the number of ports required. For example, the Nagios Box in Figure 1.3 must monitor the Web server and SMTP daemon on Server1. Instead of opening three ports on the firewall, the same outcome may be reached by running a service checker plugin remotely on Server1 to check that the apache and qmail daemons are running. By opening only one port instead of three, there is less opportunity for abuse by a malicious party.
Figure 1.3 When used correctly, remote execution can enhance security by minimizing firewall ACLs.
A good monitoring system does its job without creating flaws for intruders to exploit; Nagios makes it simple to build secure monitoring systems if the implementers are committed to building them that way.