This chapter is from the book
This chapter is from the book
This chapter is from the book
Measure of Success
How can you measure whether the IRT is successful? Executives always like to know whether the budget given to the IRT is well spent and whether the organization is more secure now than it was before. There is no universal answer to these questions. Instead of trying to provide partial answers, it is better to describe a framework on how to create metrics that will be used to measure the team’s success.
At the start, it must be said that, by itself, counting the number of incidents the team has handled in a given time period is not a good measure of how the team is doing. It can certainly be a component of the measure, but that number by itself is not informative, and there are good reasons why. After the team starts operating, it will initially see only a few incidents. Quickly that number will start to rise rapidly, and the more the team is working on them, the more incidents will come to light—and the number of incidents will just keep on growing. From that perspective, it might appear that the team is not doing things right because before it started working, there were only a few incidents, and now they never stop. In reality, the reason for seeing an increased number of incidents is because the IRT is actively looking for them while before nobody took notice of them, even when the signs were obvious.
The way to approach creating the metrics to measure the team’s success is to start from who is the team’s constituency and what is the team’s goal, and what it tries to do for the constituency. That will provide the starting point of defining what can be measured. Additionally, you can try to measure changes in the risk the organization faces from a compromise. Part of that risk assessment is the speed of recovery and limiting the damage after the incident. The final part of the metrics is the team’s influence and standing with the community. A good guide on how to define what to measure, how, and why is the ISO 27004 standard. Let’s now look at some examples of how metrics for measuring the team’s success can be defined.
One of the goals for most of the IRTs is to increase security awareness within the constituency. This goal can be aligned with specific policies such as “All users will receive basic security training” or “All users’ passwords will be longer than six characters.” Data on a number of users receiving security training and the results of checking users’ password can be easily obtained, so you can calculate where you are in meeting the policy goals. This then directly feeds into one of the measures of the team’s success.
Assessing changes in the risk the organization faces from computer attacks is harder to accomplish. You cannot directly measure the attacker’s willingness to attack your organization, but you can use the fact that attackers are mostly opportunistic creatures to your advantage. If you are a hard target, attackers will go after others who are easier targets. What you can measure here is what is happening to your organization relative to your peers and the industry. Reliable data on attacks is hard to come by. CSI and BERR surveys (mentioned in Chapter 1) can serve as guides, but the numbers must be taken with caution. Attacks do not have to be targeted; you can also compare the number and severity of virus outbreaks within the organization versus the industry. One example that illustrates this very well was an outbreak of a particular worm a few years ago. Most of the other organizations were infected, but Cisco was not because of the measures the Cisco InfoSec team implemented.
Being a leader in the field is also a sign of the team’s success. This can be measured by looking at the number of talks the team was invited to give, the number of interviews the IRT members gave, and how many of the team’s ideas were incorporated into best practices and international standards.