This chapter is from the book
This chapter is from the book
This chapter is from the book
Test Scoping
Two questions you can expect from your clients when discussing potential test engagements are
- "How long will this testing take?"
- "How much will it cost?"
You would be wise to refrain from giving an answer to either question until you have a good understanding of the scope of what you will be testing. Defining the test scope will help you estimate the extensiveness of the test process and help you forecast costs. For example, a bug fix verification test is a test with a narrow scope and would not normally require a complicated lab topology to be built or an extended set of test cases to complete. The test scope would broaden as more devices, features, and test cases are added to the requirements.
It is sometimes difficult to define the scope of a test when you do not have a technical understanding of the solution or design you are being asked to verify. For this reason, consider involving your most senior technical people during the scoping exercise, despite their protestations to avoid it. Whenever possible, you should spend time with the network architects to review the proposed network design prior to scoping a test so that you can accurately estimate the necessary tools, people, and equipment.
The sections that follow describe some considerations when scoping a test engagement, the steps for which are as follows:
- Step 1. Categorize the type of test to be completed.
- Step 2. Identify project stakeholders.
- Step 3. Identify indicators of test success.
- Step 4. Estimate the resources required to complete the test.
- Step 5. Identify risks.
- Step 6. Identify the timeline for completion.
Step 1: Categorize the Type of Test to Be Completed
Identify the reasons for testing and try to determine whether the type of test fits into one of the categories described earlier in the chapter. Try to clearly articulate the trigger and objectives in one sentence, such as:
- "Proof of concept test, with the goal of gaining confidence and experience with the new voice gateway platform."
- "Network ready for use test to certify the new data center in Syracuse is operationally ready to be cut over into production and carry live customer data."
- "Design verification test to ensure that the next-generation WAN design will work and support company business needs for the next 3 to 5 years."
Categorizing a test as shown in the previous examples will allow you to consider and compare a potential test with similar engagements you may have completed in the past.
Step 2: Identify Project Stakeholders
Stakeholders are people or groups who have a vested interest in the outcome of the test initiative. They may include the company leadership, representatives from the business units, application developers, network architects, network operations staff, and contracted professional services firms. Early in the project, work with your project sponsor to create a list of all possible stakeholders so that you can develop an effective communications plan. The goal is to gain and sustain commitment to the test initiative, and to avoid negative behaviors such as stalling or undermining from people who demand to be "in the loop." It is a good idea to solicit input to the test plan from a wide variety of stakeholders so that a comprehensive approach can be taken.
For very large test efforts requiring input from multiple stakeholders, it may be helpful to assign and designate them using a well-known method from organizational design known as RACI (Responsible, Accountable, Consulted, Informed):
- Responsible: This person is responsible for completing a task.
- Accountable: This person will be called to account if the task is not completed and may manage the person who is responsible for completing the task. Project managers often have this role.
- Consulted: Though not accountable or responsible for completion, this person is consulted about aspects of the task.
- Informed: The holder of this passive role is kept informed but isn't accountable or responsible for tasks.
Step 3: Identify Indicators of Test Success
It is critical to get agreement from the stakeholders on what constitutes testing success; otherwise, exiting the test may become difficult. You don't need to identify success criteria for each and every test case during the Scoping Phase—that will come later during test planning. What you need to understand at this point is what constitutes overall test success, so that you have a good idea of which elements are important to build test cases around. Here are some examples of success criteria for various types of tests.
Network Design Verification Test
- Test Pass Criteria:
- 100 percent of test cases have been executed.
- Network design meets customer requirements with respect to feature functionality, performance, and convergence around failures.
- No Severity 1 or 2 defects encountered.
- All Severity 3 defects (if any) have been documented/filed and workarounds have been provided.
- Test Fail Criteria:
- Severity 1 defects are found with no workaround in an area critical to the solution.
- Severity 2 defects are found that can put in jeopardy the on-time deployments of the solution.
- One or more "key" features are missing, are not operational, or don't meet customer-specific requirements.
Network Ready for Use Test
- Test Pass Criteria:
- 100 percent of test cases have been executed.
- All device hardware and line cards power up properly and successfully pass self-test diagnostics.
- All devices configured with the hardware and Cisco IOS Software specified in the LLD document.
- No device crashes observed during testing.
- All circuits passing traffic in an error-free state.
- Test traffic reroutes around failures within the constraints of the SLA specified in the LLD.
- All devices can be monitored and managed by the NMS platforms in the NOC.
- Test Fail Criteria:
- Device crashes observed during testing that cannot be correlated to faulty hardware.
- Circuit errors incrementing.
- Excessive errors seen in device logs that cannot be explained.
- Test traffic does not converge around failures within the constraints of the SLA specified in the LLD.
- Devices unreachable by the NMS platforms in the NOC.
Step 4: Estimate the Resources Required to Complete the Test
An estimation of the people, lab equipment, and test tools needed to complete the testing activities will be necessary to develop a reasonably accurate project timeline, and to provide pricing for the test if procurement is necessary. When estimating the people required, be sure to account for the skill level and number of people required for each of the necessary tasks; for example:
- Test plan development (high skill level)
- Equipment cabling and lab assembly (low skill level)
- Test plan execution (medium to high skill level, depending on technology and scale of test)
- Test results development (medium to high skill level, depending on technology and scale of the test)
It will be difficult to accurately estimate equipment resources needed for testing unless fairly complete design documentation is available for review. At a minimum, you should request a high-level network topology diagram so that you can identify any major components lacking in your lab inventory that will need to be procured. This will have an impact on the price and timeline for test execution. Be sure to consider that special software licenses may be needed if you are conducting applications testing. If new equipment will need to be installed, consider the space, power, and cooling impact to existing facilities. When scoping test engagements where unfamiliar platforms or new technology is involved, it may be necessary to allocate additional time for staff training or even account for assistance from outside consultants.
Test tools are also lab assets and need to be considered when estimating resources. Choosing the proper tool for a particular test is an art in itself, as explained in a later section, aptly titled "Choosing the Right Test Tools." Work with your test tool vendors to help you understand whether their products have the capabilities to perform the types of tests you will be executing. They are an invaluable resource, and many of the larger vendors often lease test tools for the duration of a test engagement if necessary.
Step 5: Identify Risks
A risk is an event or condition that, if it occurs, could negatively impact your test project. Sometimes risks are outside your control, such as when an equipment or software manufacturer cannot ship critical hardware or tools on time. This type of risk can be mapped to external dependencies. In other cases, risks can be mapped to internal dependencies, as in the case of having to hire staff with the appropriate skill set to complete a test. Tests that do not have clear objectives or well-documented designs present a risk of going over budget as the test team wastes time testing the wrong features or design scenarios. A few of the most common risk factors to consider when planning test schedules include the following:
- Shipping issues with equipment
- Third-party equipment requiring special skills from an outside vendor
- Lack of responsiveness from customer technical leaders or decision makers
- Contention for equipment due to a critical outage in another test bed or production area of the company
- Personnel without appropriate skills
- Personnel leaving the group/company
Step 6: Identify the Timeline for Completion
Large testing projects often require contributions from multiple people across different departments and even companies. Good teamwork, clear communications, and dedication to the project are necessary so that testing of a particular design or solution does not last forever. Because network testing is commonly triggered by a proposed network deployment, deadlines are often readily available and documented in an overall project plan. A challenge you will often face is developing a realistic test timeline that allows you to execute your testing thoroughly, while at the same time meeting the deadlines of the project. It is often helpful to allocate time to each of the various test phases when constructing your timeline; for example:
- Assessment of Objectives: 1 week
- Test Case Planning: 2 weeks
- Test Lab Setup: 2 weeks
- Test Execution: 2 weeks
- Test Documentation and Analysis: 1 week
Understand that there are dependencies that may affect your timeline, causing it to slip past the eight calendar weeks given in the preceding example. For example, you will not be able to move into the test execution phase if you are waiting for critical equipment or test tools to be procured, as identified in the assessment phase. Also, it is extremely important to obtain stakeholder feedback and signoff on your test plan prior to moving into your lab setup. Otherwise, you risk the chance of last-minute test changes or additions being requested by the stakeholders. A clear and continuous communications plan during this process is necessary to maintain an accurate test timeline.
Once the goals, objectives, and test scenarios are clearly stated and acknowledged by the stakeholders, you should be able to define success criteria, execute the tests, deliver the results, and exit the engagement.