Intrusion Prevention: Signatures and Actions

Triggering Mechanism

Triggering mechanisms refer to the conditions that cause an intrusion system to generate a signature action. For example, the triggering mechanism for a burglar alarm might be a motion detector that detects the movement of an individual entering the alarmed room. A Network IPS might trigger a signature action if it detects a packet with a payload containing a specific string going to a specific port. A Host-based IPS might trigger a signature action when a specific function call is invoked. Anything that can reliably signal an intrusion or security policy violation can be used as a triggering mechanism.

Protocol Decodes

Another common triggering mechanism is called protocol decodes. Instead of simply looking for a pattern anywhere in a packet, protocol decodes involve breaking down a packet into the fields of a protocol and then searching for specific patterns in a specific protocol field or some other malformed aspect of the protocol fields. The advantage of protocol decodes is that it enables a more granular inspection of traffic and reduces false positives.

Regular Expression

A regex is a pattern-matching language that enables you to define a flexible search pattern. Using regex, you can easily define complex search patterns. Many different programs use regex to enable you to define custom search strings. In the UNIX world, for example, the grep command is a common program that utilizes regex to search for text inside of files (or other output). For example, to perform a case-insensitive search for the word attack in the file named output.results, you can use the following command:

This command finds various permutations of the word attack, such as ATTAck, attack, AttaCk, and so on.

Besides searching using flexible string patterns, many protocols (such as HTTP) accept multiple encoding mechanisms for input data. Some of the common encoding mechanisms include the following:

• American Standard Code for Information Interchange (ASCII)
• Hexadecimal
• Unicode

Unless your IPS can decode the input data correctly (before performing a pattern search), it will miss valid strings that have been obfuscated. For example, the following patterns each request the same web page:

• http://10.10.10.10/index.html/../etc/password
• http://10.10.10.10/index.html/%2e%2e/etc/password
• http://10.10.10.10/index.html/%c0%ae%c0%ae/etc/password

In each of these requests, the .. is being represented differently. If your IPS correctly deobfuscates these requests (before performing a pattern search), then it can correctly detect the attempt to request a web page outside of the root of the web server's directory tree in each of these web requests (which all request the same web page).

Pattern Matching Considerations

Depending on the fidelity of the patterns that your signatures use, one of the problems with pattern matching can be the rate of false positives generated, especially with atomic signatures.

One of the benefits of pattern matching is that the signature clearly correlates to a specific attack. When the signature triggers, you know which attack it is detecting. This is different from anomaly detection in which the signature indicates only that something outside of the configured normal parameters has been detected.

Host-Based Examples

A Host-based IPS product might have the capability to examine the data inside a file on the hard disk. An example of a pattern-based signature trigger is the word confidential being detected in any data file. A stateful signature with the same triggering mechanism might look for the word confidential in inbound network traffic. The pattern is the same, but can be matched only if multiple packets have been re-assembled.

Network-Based Examples

Most Network-based IPS devices include a robust pattern-matching capability because many of the attack signatures involve searching for patterns in different network protocols. Most of these pattern-matching signatures are also stateful signatures because the information being examined can occur across multiple packets.

The following regex string searches for an attempt to change the working directory to the root directory during an FTP session:

[ \t]*[Cc][Ww][Dd][ \t]+[~]root

Examining traffic with a destination port of 21 (the default FTP server port) detects FTP sessions in which the user attempts to change the working directory to root.

Anomaly-Based Detection

Anomaly-based (also known as profile-based detection) signatures are not based on a specific event. Instead these signatures trigger when a certain activities deviate from what is considered normal. To utilize an anomaly-based signature, you must first determine what normal activity means for your network or host. This is usually accomplished by monitoring your network (or specific applications on your host) for a specific period of time to observe what is considered normal activity. Once you define normal activity, then you can configure your anomaly-based signature to trigger whenever activity on the network or a specific host deviates from your defined normal profile by a certain amount.

Anomaly-Based Detection Considerations

One of the biggest limitations of anomaly-based signatures is that you need to learn (or define) what is considered normal. As your network evolves, your definition of normal might also have to change. Furthermore, you need to guarantee that during your learning phase, your network is free of the attack traffic that you are going to detect (otherwise, this activity will be considered normal traffic). Furthermore, just defining normal can sometimes be difficult because most networks comprise a heterogeneous mixture of systems, devices, and applications that continually change.

Another potential drawback to anomaly-based systems is that when a signature generates an alert, it might be very difficult to correlate that alert back to a specific attack, because the alert indicates only that non-normal traffic has been detected. More analysis is required to determine whether the traffic represents an actual attack and what the attack actually accomplished. Furthermore, if the attack traffic happens to be similar to normal traffic, the attack might go undetected.

Because anomaly-based signatures do not look for specific attacks, they can be used to detect previously unpublished attacks. This is a major advantage for anomaly-based detection. Instead of having to define a large number of signatures for various attack scenarios, you simply define a profile for normal activity. Any activity that deviates from this profile is then abnormal and triggers a signature action. The drawback is that an alert from an anomaly signature does not necessarily indicate an attack. It indicates only a deviation from the defined normal, which can sometimes occur from valid user traffic.

Host-Based Examples

A good example of anomaly detection is using the Profiler utility from the Cisco Security Agent (CSA) software. This tool allows you to monitor an application over a period of time. During this time, it records all of the functions calls that the application uses. You can then use this information to build a profile for the application that can then be used to identify when the application makes anomalous function calls (any function call that is not in the profile).

Sophisticated Host-based IPS tools might be able to apply stateful conditions to function calls recorded during the learning process. If two function calls that are a part of the normal profile occur within 1 second of each other and this has never happened before, this could trigger a signature. This is an example of a stateful signature with an anomaly-based triggering mechanism.

Network-Based Examples

A Network-based IPS can have various anomaly-based signatures. Some simple examples of anomaly signatures with the Cisco IPS solution are its flood signatures. These signatures detect floods of various types of traffic on your network. For example, you might want to monitor the amount of Internet Control Message Protocol (ICMP) traffic on your network. First, you need to measure the amount of ICMP traffic that appears on your network during normal operation. Then the flood signature triggers a signature action whenever the ICMP traffic on the network exceeds the configured maximum threshold for a specific length of time.

Other anomaly-based signatures involve ensuring that traffic to a set of ports matches a defined protocol specification. In this situation, the protocol specification defines what is considered normal. Any traffic that does not conform to this protocol specification is abnormal and triggers a signature action.

Behavior-Based Detection

Behavior-based detection is similar to pattern detection, but instead of trying to define specific patterns, you are defining behaviors that are suspicious based on historical analysis. The behaviors define classes of activity that are known to be suspicious. For example, an e-mail client running shell commands using the Windows cmd.exe program normally indicates something unusual such as a virus attempting to do something to your system.

Behavior-Based Detection Considerations

Similar to pattern matching, behavior-based signatures must be defined before you can use them. It takes a lot of research to determine behaviors that do not occur normally and can accurately indicate suspicious behavior.

The use of behaviors enables a single signature to cover an entire class of activities without having to specify each individual situation. For example, having a signature that triggers a signature action when an e-mail client invokes cmd.exe enables you to apply the signature to any application whose behavior mimics the basic characteristics of an e-mail client without having to apply the signature to each e-mail client application individually. Therefore, if a user installs a new e-mail application, the signature still applies.

Host-Based Examples

Behavior-based signatures are easy to visualize at the host level. A simple example involves the cmd.exe program. This program is frequently used by malicious programs to execute commands and scripts on the system. It is also frequently used by users during their normal day-to-day operations. Using a behavior-based signature, you can detect suspicious invocations of cmd.exe while ignoring the normal uses of this program. For example, an e-mail client that invokes cmd.exe is performing suspicious activity (because the e-mail client does not need to run cmd.exe to operate). Incorporating a signature that triggers a signature action whenever cmd.exe is invoked by any e-mail client application is an excellent example of a behavior-based signature.

Network-Based Examples

At the network level, you might find it a little more difficult to identify the behavior-based signatures. For example, a signature that identifies attempts to directly access RPC applications is behavior-based, because the normal behavior is to first communicate with the PortMapper.