Crafting the Test Approach

This chapter is from the book 

Choosing the Right Test Tools

• "Take these three items—some WD-40, a vise grip, and a roll of duct tape. Any man worth his salt can fix almost any problem with this stuff alone."
• —Clint Eastwood as Walt Kowalski in the movie Gran Torino

In contrast to simple home repair, system testing can be very complicated. In most cases, completing system testing without the assistance of sophisticated test tools is not practical or even feasible. Choosing the right tools and knowing how to use them is extremely important to effectively guide and streamline test execution and reporting of results. There are myriad test tools to choose from, and, as with anything else in networking, no single tool is right for every test or for every tester. While it would not be practical to mention every product or type of test tool in this book, the following sections describe some of the common categories of tools and which type of tests they would commonly be used for.

Stateless Packet Generators (Bit Blasters)

Packet generators are one of the simplest, but most important, types of test tool. Many different vendors sell packet generators, and the products differ from each other by many criteria such as the type of interface or interfaces supported, packets per second (pps) capacity, packet and traffic manipulation capability, results reporting, and automation.

Interfaces

Test tool vendors have historically differentiated themselves competitively by producing multiple interface cards for their products. When the world of low-speed WANs included multiple physical interface specifications (such as RS-232, V.35, and X.21), vendors were at the ready with interfaces, adaptor cables, and the like, to allow direct connection to the WAN circuit. With the growing popularity of Ethernet, including Gigabit Ethernet (GE) and 10-Gigabit Ethernet (10GE) offerings from service providers, many vendors now focus their product offerings on these interfaces, with the assumption that if the network under test includes a WAN link, the packets needed for testing can be forwarded to it via a router or switch over a Gigabit or 10-Gigabit Ethernet interface. Thus, the days of specialized interfaces on packet generators are essentially over, and test tool vendors are focused more specifically on the port density (feeds) and capacity (speeds) of their products.

A packet generator with Gigabit Ethernet interfaces generally has an RJ-45 (Category 5-style) interface, and you need to pay attention to the physical media settings of Gigabit or 100-Mbps operation and half-duplex, full-duplex, or duplex autonegotiation. When using a packet generator with a 10-Gigabit Ethernet interface, you are faced with choices about media adapters (XFPs, SFPs, etc.) and single- or multimode fiber cables.

Tool Power/Capacity

Protocol analyzers, such as Wireshark, which can be installed on a laptop computer, are capable of generating a low-speed stream of packets, which may be adequate for a basic feature test. An example of this usage would be to verify that a packet actually traverses a device or network under test, or that it is blocked, for instance, by an access list. Analyzer-based generators may also have the capability to replay a captured stream of packets back into the network. Analyzer-based generators are especially useful for field technicians, who are dispatched to network trouble sites and who are usually troubleshooting either Layer 1/physical-type problems or device misconfigurations.

For lab-based testing, more powerful devices are generally needed; depending on the vendor, you'll find generators of varying capacity. Packet generators at this level are usually chassis based, and you purchase different types of cards (GE or 10GE, etc.) for the chassis. The cards have varying numbers of generating ports (for instance, ten 1-GE ports or four 10GE ports) and you should pay attention to any caveats regarding capacity that the vendor lists in its documentation. For example, a four-port card may be capable of generating at full line rate on only two ports simultaneously, or a port may be able to support full line rate but is limited as to the number of unique streams it can send; you need to know this as you plan your tool use in the test bed. Generally, these types of generators are also capable of packet capture and inspection (protocol analysis) when they are not being used for generation. Again, check the documentation to assess the product capabilities, preferably before you commit to the purchase!

Packet/Traffic Manipulation

There are many ways in which manipulation of the packets being generated is important to network testing. The address fields (both MAC and IP addresses) need to be easily modifiable so that the packets traverse the network as expected. The Layer 4 header can also be modified to specify a certain TCP or UDP port in order to simulate a certain type of network traffic, or test access lists. The packet payload can also be populated with specific content.

Most packet generators have the capability to send errored packets—for instance, packets that are too long, too short, or have bad checksums. This can be important if the purpose of a test is to verify that the device under test (DUT) does not forward an errored packet and, perhaps more importantly, that it does not experience operational stress, such as a lockup, high CPU, and so on, if it receives a steady stream of errored packets. In addition, the generator may allow the engineer to set the packet size within the range of valid sizes, either as fixed or incremental, or variable in accordance with some particular algorithm.

Other popular packet manipulations include the capability to set QoS bits within a packet, such as those used for Type of Service (ToS) or Differentiated Services Code Point (DSCP). These capabilities make it straightforward to test classification or queuing features in a switch or router.

For manipulating the traffic itself, most packet generators provide the capability to send traffic continuously or to send only a specified number of packets, which can be useful if the focus of the test is to determine if there is any packet loss. Other traffic rate possibilities include

• Sending by percentage of line rate
• Send rate in pps
• Send rate in bps
• Send rate determined by interpacket gap

Almost all packet generators on the market today have these packet- and traffic-manipulation capabilities, although the look and feel of the user interface makes some easier to learn and use than others.

Results

In some test scenarios, test results are collected exclusively from the device or network under test. However, the packet generator provides useful information regarding what it is doing, displaying the traffic being generated in metrics such as

• Transmit rate in bps
• Transmit rate in pps
• Number of packets, in total, transmitted
• Number of bytes, in total, transmitted

Most test tools have the further capability to coordinate information such as packets sent and received, and to use timestamps applied to the traffic being generated, across two or more ports in the same chassis, giving the tester important data such as packets lost, flow latency, and, for convergence tests, packet loss duration.

Traffic generators also generally have the capability to save and/or export these statistics in various formats, the most basic of which is a comma-separated values (CSV) file. The more sophisticated test tools are capable of generating graphs or pie charts within their own software, which can be very useful for producing result reports on the fly as tests are completed.

Automation

Some testing is conducted manually, with single-threaded test cases being executed and observed by the test engineer, and with data collected and conclusions drawn as the tool is in use. Other types of testing, such as regression testing, require a longer duration of tool use, with the expectation that test cases are run sequentially, and device and tool configurations are changed automatically as individual tests within a greater test suite are completed. For automated testing, the test tool must have some type of scripting interface. The most popular scripting language in the world of test engineers is Tool Command Language (TCL). Almost all packet generators available today have the necessary support for scripted operation via TCL. Further, most test tools have their own proprietary automation interface. An advantage to the use of TCL for automation is that it is a global application—for instance, with TCL, you could kick off a router configuration change, and then change the test tool configuration and restart the traffic flows. With the test tool's proprietary automation application, it's not likely that you could reconfigure a router.

When to Use Stateless Packet Generators

Packet generators are useful for many different types of tests. They can be used to create a certain level of background traffic as other application-specific tests are run. They can be used for stress testing, where the desired result is something like a performance chart for a DUT—for instance, pps compared with processor CPU. They are also useful for QoS testing, because the tool allows you to set the ToS or DSCP bits, in addition to manipulating the packets sent in the other ways previously discussed.

Packet Generator Vendors

At press time, the North American market for packet generators is essentially shared by two major corporations—Spirent Communications and Ixia—although there are other "niche market" players:

• Spirent: The "classic" Spirent tool is called SmartBits; as additional software capabilities such as stateful traffic generation (see the next section) were added to the basic packet generator, new product names were rolled out. These new names are used to differentiate the SmartBits tool from the one with the new features, even though all products run on the same physical chassis. Spirent's present flagship test platform is TestCenter.
• Ixia: Like Spirent, the "first generation" Ixia product and the subsequent "feature-enhanced" software packages for IP traffic run on the same underlying chassis. The packet generator engine is called IxExplorer. Subsequent software packages such as stateful traffic generation are, as of this writing, called IxNetwork and IxLoad. Ixia currently supports another popular packet generator, the Agilent N2X, as Ixia acquired Agilent in 2009. Ixia has renamed the N2X product line IxN2X.

Stateful Packet Generators (Application Simulators)

For many years, packet generators verged on the capability of doing stateful traffic. Examples of what might be deemed "semistateful" capabilities include

• Capability to form a routing protocol (BGP, OSPF, etc.) neighbor relationship with a router and inject routes or other routing information such as OSPF link-state advertisements (LSA) into the network
• Capability to inject a multicast stream at one point in the network, and have a different test port send a "Join" for that stream, thus allowing multicast to be tested without a real multicast source in the test bed

What was lacking in these early tools was the capability to respond to network conditions, as a real TCP-enabled network node does. If a "real" network device running an application over TCP experiences packet loss, the normal behavior is to request a resend of the packets that were not received and then "window" the packet rate down, to avoid future packet loss. A stateless packet generator simply cannot do this, which is a reason why it is sometimes referred to as a "packet blaster."

Another benefit of most stateful tools is their capability to interact with devices in the network, as opposed to simply interacting with other test tool ports. Examples of this capability include tools that can send traffic through firewalls, and those that can work with external servers or server load balancers.

Stateful Generation Tool Vendors

The first commercial stateful packet generator was a product called Chariot, and it consisted of endpoint software, installed on Microsoft Windows or UNIX/Linux computers, and console software, installed on a separate machine. The console computer controlled the endpoints (directing them to send application test traffic) and collected results. The Chariot endpoint software was acquired by Ixia, which integrated the endpoint software into the Ixia chassis line cards, with the display console running on a GUI called IxChariot. The current version of this stateful traffic generator is called IxLoad.

Spirent's Avalanche 3100 appliance solution is similar to Ixia's IxLoad product, with the capability to generate stateful traffic at high speeds for application and performance testing. The Avalanche 3100 is a line-rate, 1-Gbps and 10-Gbps Layer 4–7 multiprotocol stateful traffic performance solution with multi-10-Gbps capacity for stateful application traffic generation.

Results Reporting

To be useful for most types of testing, stateful tools must also provide the same basic types of results as packet generators, such as packets sent/received, packet rate sent/received, loss, and latency. However, stateful tools generally have a more sophisticated format for test results. Because stateful tools generally seek to mimic various types of applications, such as a DNS lookup or an HTTP Get, they need to provide measurements that are reflective of the "user experience" for an application. Examples of such metrics include the following:

• Response time (for instance, on an HTTP Get)
• Throughput (for instance, for an FTP Put)
• Transactions per second (for highly interactive applications with small-packet payloads)
• Voice quality (simulations of Voice over IP, with Mean Opinion Scores being derived from delay, jitter, and loss measurements)
• Connection setup rate or maximum concurrent connections for firewall

As with stateless tools, the product must offer some capability to export results data from the tool, and the GUI for transferring these results should not be overly complex.

When to Use Stateful Packet Generators

Stateful generators are required when testing devices or technologies that do not respond correctly to stateless packets, and they add value to many other kinds of tests. They can be used for many of the same types of tests as packet generators—to provide a level of background traffic, or to stress test a device such as a stateful firewall. More information on choosing stateful or stateless traffic is available in the "Understanding the Different Types of Test Traffic" section of Chapter 5.

When used as a routing protocol generator, injecting routes and traffic that follows them, a stateful tool is well suited to testing the scalability and resilience of a particular network design. A stateful generator can also be used for determining the capacity of a given design, particularly if it includes a low-speed link (such as a WAN link), by drawing on user experience results such as response time, or Mean Opinion Score for VoIP. There are many newer technologies, such as Cisco Performance Routing (PfR), that are best tested with stateful flows; for example, with PfR, the PfR Master Controller makes decisions about managing traffic based on delay and throughput metrics that it can collect from real TCP-based transactions. A stateful generator is also useful for QoS testing, where the test is expected to produce user experience results—for instance, where low-priority TCP flows are expected to back off in the face of congestion, allowing preferred service to other types of traffic.

Network Delay and Impairment Tools

Another important tool in the tester's toolkit is a device, sometimes referred to as a "black box," that allows the injection of delay or impairment into a network under test. When it is crucial to understand how a technology, an application, a device, or a design copes with impairment or delay, this kind of tool is invaluable.

Delay

There are many sources for delay in a network, including serialization delay, propagation or transit delay, and queuing delay with network devices. Delay is especially troublesome if a network includes low-speed links, and if a user application is delay-sensitive. Far too often, applications are tested on a single LAN, with no thought given to how they may perform with a client device located miles away from a server. To properly assess the user experience in the face of delay, a test lab should include the capability to simulate network delay. This is commonly achieved using appliance-based impairment tools, or in some cases extremely (several kilometers) long spools of fiber.

Impairment

As with delay, impairment at the physical layer of the network is another stress that should be considered when rolling out a new design, technology, or application. When bit-level losses cause packet corruption, TCP-based applications detect packet loss and require acknowledged retransmission, which slows down application throughput and user response time. In the face of packet loss, UDP-based applications simply lose integrity; the most obvious instance of this being VoIP traffic, where packet loss is noted by the end users in the form of poor voice quality (for instance, "choppiness" or a sense of the voice dropping in and out). An imprecise way to introduce errors at the physical layer is to wiggle connectors or use cables that are known to have bad conductors or internal shorts. However, it is very difficult to do this scientifically; a cable that is simply a little bit bad one day may be completely "shot" the next, making it impossible to produce the same error condition. It is far better to use a tool that introduces a programmable and predictable amount of impairment.

Network Modeling and Emulation Tools

In some instances, it is simply not practical or feasible to send test traffic into a prototype network system built with actual network equipment. Network architects are often asked to evaluate one or more alternatives to a proposed network design, comparing the functionality and estimated performance of various topologies or routing protocol configurations. In these situations, it is often preferable to leverage a modeling tool, which uses software and mathematical models to analyze the behavior of a network, or an emulation tool, which duplicates the functions of a network device by implementing its operating software on another computer.

Network Modeling Tools

Network modeling tools offer the capability to create visual simulations of network topologies by using actual production device configurations as the "seed" files. These tools rely on network device libraries that model the behavior of major networking devices (routers, switches, firewalls) and simulate the interconnection speeds and performance of circuits so that network capacity, resiliency, and application performance can be estimated in a variety of conditions. Network modeling tools are more easily set up than real test topologies and can allow alternatives to be more quickly compared and easily evaluated. Typical applications of network modeling tools include simple tasks such as device configuration audits, to complex sophisticated functions such as capacity planning and application performance analysis. Modeling tools should be used for early decision making on a design. They may point out obvious issues before you build out your test bed, and thus save you valuable time.

Network Modeling Tool Vendors

Cisco Advanced Services teams use several different network modeling tools. They can be invaluable for getting a quick evaluation of the effects a change will have on a large network topology. Three of them are discussed in the following sections.

OPNET Technologies

One of today's market leaders of network modeling tools is OPNET Technologies (www.opnet.com), which provides several products that enable enterprise and service provider planners to analyze how network devices, protocols, applications, and servers operate. OPNET's products model Cisco's and other vendors' device configurations, and they include tools for taking captured data and analyzing which components are the bottlenecks.

Shunra Software

Another one of today's market leaders of modeling tools is Shunra Software (http://www.shunra.com), which offers a suite of products known as Virtual Enterprise Solutions. Like the others, Shunra provides tools that have the capability to simulate network topologies and conditions, allowing users to predict application performance and play "what-if" scenarios prior to deployment. In addition to its software tools, Shunra also offers a hardware-based appliance that offers the capability to plug actual multimedia devices into it so that voice and video can be evaluated subjectively by users, rather than relying solely on numerical presentation. This enables performance to be evaluated against real applications as delay, packet loss, jitter, or reduced bandwidth is introduced.

Analytical Engines

NetRule 7.1, from Analytical Engines (www.analyticalengines.com), is another software modeling tool that allows users to simulate networks and predict application performance. According to the Analytical Engine website, NetRule costs far less than other predictive network applications on the market.

Application Simulation Tools

In the early days of networks, the network infrastructure was managed as its own entity, and the applications were managed separately. Typical IT organizations were often broken into three groups: Network Engineering/Support, Desktop Support, and Application Development/Support. However, the network has evolved to be more application-aware, and the applications have evolved to be more network-aware, creating organizational gray areas on one level, and requiring much more application intelligence in the testing of the network.

As a reflection of the trends toward the blending of applications and infrastructure, testing has evolved from simple single-protocol bit blasting at Layer 2 and Layer 3, to full-fledged stateful multiprotocol, multihost application simulation. Similarly, application testing has evolved from UI-oriented testing, such as that offered by Mercury Interactive and others, to approaches that are much more network-aware.

Application simulation is distinguished from protocol testing because it recognizes that protocols don't exist in a vacuum. Applications are increasingly IP based, complex, and changing at ever-faster rates. Real applications involve vendor proprietary extensions, multiple protocols, and multiple cooperating systems. Testing based on actual network traffic, which includes many not-quite-standard protocol extensions, is much more "real world" than testing based on whatever the standards bodies say should be on the network.

Real application traffic is much different from what is in those standards (after all, standards bodies focus on protocols; very few applications are standardized), so it's essential that application simulation start from reality, not from the ivory tower. Even if there were a standard for applications, the configuration fingerprint of an application in one particular enterprise environment would be unique, and moreover, these configurations change frequently (as components are upgraded, new components are added, new code is rolled out, new users are defined, etc.).

Mu Dynamics (www.mudynamics.com) has taken an approach that is based on actual application traffic, which isolates the application from the transport. The key thing that differentiates application simulation from bit blasting, or from packet replay, is that Mu's approach involves re-creating the same stateful application flow over a dynamically created transport. Application state is maintained by tracking important session variables (cookies, tokens, initial values, sequence numbers, call IDs, etc.) throughout the session, so it is indistinguishable from a session created by a real client device (or server device). In fact, many applications depend on concurrent usage of multiple protocols, involving multiple cooperating systems.

Application simulation testing in these dynamic environments must capture and represent this unique configuration so that the test harness is as close as possible to the behavior of real clients talking to real servers. This is the core reason why Mu Dynamics' Test Suite turns packets into test cases, within an automation framework that magnifies the productivity of test teams by at least an order of magnitude. Application simulation depends on accurately reproducing application flows so that the test traffic is indistinguishable from real application traffic.

Security Testing Tools

Security and penetration test cases are often required components of hardware and software certification test plans. There are several security and penetration test tools available, some of which are marketed commercially, others developed and distributed as freeware by "hackers" on the Internet. In either case, your company's information security team and your code of business conduct should be consulted before any security tool is used on your network.

The best known and most commonly used security tools are referred to as port scanners. Port scanners, such as the open source and free Nmap utility, operate by exploring an address range for active hosts using ping (ICMP ECHO and REPLY) packets. Once active hosts have been identified, they are scanned for open TCP and UDP ports, which can provide indication of the network services operating on that host. A number of these tools support different scanning methods and have different strengths and weaknesses; these are usually explained in the scanner documentation. For example, some are better suited for scans through firewalls, and others are better suited for scans that are internal to the firewall. Some port scanners are capable of gathering additional information, such as the target operating system, using a method known as system fingerprinting. For example, if a host has TCP ports 135 and 139 open, it is most likely a Windows host. Other items such as the TCP packet sequence number generation and responses to ICMP packets (for example, the Time To Live, or TTL, field) also provide a clue to identifying the operating system. Operating system fingerprinting is not foolproof; firewalls can be configured to filter certain ports and types of traffic, and system administrators can configure their systems to respond in nonstandard ways to camouflage the true operating system.

In addition to discovering a target operating system, some scanners are also capable of discovering whether host applications that leverage well-known TCP ports are running. For example, if a scanner identifies that TCP port 80 is open on a host, it can be assumed that the host is running a web service. Even more sophisticated scanners are capable of identifying which vendor's web server product is installed, which can be critical for identifying vulnerabilities that can be exploited. For example, the vulnerabilities associated with Microsoft's IIS server are very different from those associated with Apache web server. This level of application identity can be accomplished by "listening" on the remote port to intercept the "banner" information transmitted by the remote host when a client (a web browser in this example) connects. Banner information is generally not visible to the end user; however, when it is transmitted, it can provide a wealth of information, including the application type, application version, and even operating system type and version. Again, this is not foolproof, because a security-conscious administrator can alter the transmitted banners. The process of capturing banner information is sometimes called banner grabbing.

Many rudimentary security tests on network gear consist of conducting a port scan to check that only the expected ports—those that have services active such as Telnet or SSH—are open. For these basic requirements, a freeware tool such as Nmap should be adequate. When doing more complete or specific security and vulnerability testing, there are several good freeware and commercially available tools on the market. These tend to be much more complex and can require some security and scripting expertise to operate.

Packet-crafting tools such as HPING can be used to assemble and send custom packets to hosts, with the goal of obtaining information from the received replies. These kinds of tools are often used when trying to probe hosts behind a firewall by spoofing a well-known application's TCP port. A tool such as HPING can also be used to create malformed protocol packets in an attempt to launch a denial-of-service (DoS) attack. One common example of such an attack would be to send fragmented BGP packets to a router with the intention of creating a software crash.

Yersinia (www.yersinia.net) is a network tool designed to take advantage of some weaknesses in different network protocols. Attacks for the following network protocols are implemented:

• Spanning Tree Protocol (STP)
• Cisco Discovery Protocol (CDP)
• Dynamic Trunking Protocol (DTP)
• Dynamic Host Configuration Protocol (DHCP)
• Hot Standby Router Protocol (HSRP)
• IEEE 802.1Q
• IEEE 802.1X
• Inter-Switch Link (ISL) Protocol
• VLAN Trunking Protocol (VTP)

You can use a tool like Yersinia to harden your Layer 2 design and implementation.

Other security tools that can be used during testing are common password crackers, wireless scanners, and DoS tools, just to name a few.

Network Protocol Analysis Tools

These tools are often referred to as "sniffers." The best-known one is a free tool called Wireshark (formerly known as Ethereal, found at www.wireshark.org/), which can decode hundreds of protocols, including OSPF, BGP, SNMP, Telnet, and just about any other protocol your production network may have running. Sniffers used with a spanned port on a Cisco switch are invaluable tools for troubleshooting.