- What is Disaster Recovery Planning?
- Purpose of This book
- A Working Definition of Disaster
- The Time Factor in Disaster Recovery
- The Need for Disaster Recovery Planning
- The Auditor's View
- An Imperfect Legal Mandate
- Building Management Consensus for Disaster Recovery Planning
- Who Should Write the Plan?
- A Straightforward, Project-Oriented Approach
- A Note on Methodology
This chapter is from the book
This chapter is from the book
This chapter is from the book
Who Should Write the Plan?
Once the decision has been made to undertake disaster recovery planning, the information manager must first determine the method to be used to develop the plan. One option is to hire a consultant to perform this task. Another is to develop the plan in-house. Valid arguments exist to support each option.
At first glance, hiring a consultant with X years of experience in developing this type of project may seem the best choice. Indeed, this approach has several distinct advantages.
First, the disaster recovery planning project is just as complicated as a major system development project and, in fact, parallels the systems development life cycle (SDLC). (Figure 1–1 depicts the similarity.)
)
Figure 1–1 Comparison of systems development life cycle (SDLC) to
disaster recovery planning project.
Like a system development project, a disaster recovery planning project begins with analysis. A risk analysis process is undertaken to identify potential threats and vulnerabilities, while business impact analyses and application impact analyses are undertaken to identify critical business processes and their IT infrastructure supports and to discern recovery priorities, objectives, and requirements.
Recovery strategies are then outlined and tasks are prioritized much in the same way that an analyst would set forth a general system design. This general design is subjected to user review and, if it is approved, a detailed system description is articulated. At this point, development costs are specified and a project time-and-money budget is developed.
In systems development, the project would be approved by management, and coding would begin. Similarly, the disaster recovery planning budget is presented to senior management and, if approved, vendors are contacted, products and services purchased, and recovery procedures developed and documented.
Plan testing and user training follow, just as comparable activities would follow the conclusion of coding. Finally, when the system is released or the plan is placed into effect, it is integrated into a change management system to provide for periodic review, revision, and maintenance.
An IT manager, realizing the scope and complexity of the planning project, may decide that a consultant is needed to manage it. The IT manager either cannot reassign an employee to manage the project or feels that no employees are equal to the assignment. There may also be other factors that favor the consultant option:17
Consultants bring specialized knowledge to the planning project that may facilitate the speedy development of an effective plan. An experienced consultant knows how a disaster recovery plan is constructed, knows the right questions to ask, and typically knows who's who in the disaster recovery products and services industry. Consultants who work within a specific industry may combine an understanding of the industry with a methodology for disaster recovery planning. This reduced learning curve, in turn, can help to speed plan development.
Consultants can bring a fresh eye to the project, noticing recovery requirements that may be overlooked by someone who is too close to the data center he or she is seeking to protect. One consultant relayed a story about a client who had hired her to perform a risk analysis of a data center. In conferences, the client confidentially reassured her that all vital processing equipment had been identified. Then, during a preliminary visit to the data center, the consultant nearly tripped over an ancient time card reader. She asked what it was and learned that no payroll checks could be generated without its use. A vital piece of equipment, yet it had not been mentioned anywhere in the lists that otherwise documented completely the state-of-the-art hardware installed in the shop.
Consultants are expensive. While this may be viewed as a drawback of the consultant option (and will be discussed later in this chapter), it may actually favor plan development in certain cases. Disaster recovery planning requires the interaction of users and information systems and network technical personnel. Within a large information systems shop, where rivalries frequently exist among applications support personnel, systems administrators, and operations, disaster recovery planning will also require the interaction of these groups. Often the only way to get all of the relevant parties to sit down as a group and discuss critical issues is to make it clear that a great deal of money is being spent for the consultant's time. Similarly, senior management, having invested a considerable sum of money for a consultant-developed plan, may be less inclined to withdraw support for the implementation of the plan.
Consultant-driven plans are similar to computer hardware acquisitions: They are usually available with a maintenance agreement. For a fee, the consultant will return on a semiannual basis to aid in the testing and updating of the original plan. Furthermore, since the plan usually reflects the "favored" (read "proprietary") methodology of the consultant, many consulting firms offer a training service to educate personnel in the client company who will maintain or use the plan.
Good consultants usually produce good plans and provide competent maintenance and training services. Unfortunately, not all consultants are good consultants. As of this writing, the disaster recovery planning consultancy is an unpoliced field. In the late 1960s, there were only handfuls of disaster recovery consulting firms. Since that time, the number has increased exponentially. It is not uncommon for consulting firms to open and close their doors within the same year. This bodes ill for the industry as a whole.
Consultants may attempt to demonstrate their competence by referencing a certification from a DR planning certification body. Several certification organizations have evolved over the past decade with the stated objective of training novice planners and, for a fee, "certifying" the skills of those who have learned their trade "on the job." Contrary to the view of many who have obtained certifications, the kindest thing that this author has to say about certification programs is that, at present, they provide little assurance about the capabilities of those who hold them.
This assertion is likely to draw fire from several quarters, so it merits further discussion. In the early 1990s, the originator of one of the first certification programs for DR planning contacted this author to solicit his participation in promoting a "marketing concept": disaster recovery certification. The proposal consisted of selling certifications for a fee to anyone who could pass a multiple-choice test consisting of easy-to-answer questions such as the meanings of familiar acronyms and the definitions of DR-specific concepts such as "dial backup" and "hot site." Not to be exclusionary, DR practitioners would also be invited to participate. They would be "grandfathered in"—that is, provided the certification without testing in exchange for dues payment. The fellow was clearly delighted with the concept, which he viewed as a "sure money-maker" on three grounds:
Many individuals responsible for disaster recovery planning for their organizations suffered from a lack of confidence. They feared that they were not as professional as consultants who developed DR plans for many companies. The certification program would not make them better planners, but it would give them the appearance of professionalism when they interacted with other professionals and with management. Basically, the certification program was a confidence game.
The certification program would become a discriminator between consultants in a highly competitive and totally unregulated disaster recovery consulting industry. The revenue potential for the program based on "grandfathering" fees alone was enormous, even if the certification program was meaningless.
The certification program would be a great way to amass the world's best database of DR planner names and addresses, which could subsequently be resold to vendors of DR products and services, producing another revenue stream.
Demonstrating a lack of business acumen, the author declined to participate. Nevertheless, the program was launched and became one of the most successful certification programs today.
Recently, an acquaintance of many years, who had retired as the chief disaster recovery planner for a major financial institution, complained that he was required to obtain a certification before he could join the consulting group of a major systems integrator. The fellow was told that in spite of his extensive experience in DR planning, his intimate familiarity with planning methods and tools, his thorough knowledge of vendors and their offerings, his former senior role within a major disaster recovery planning user group, and his numerous references, he was unmarketable without the letters of a certification program following his name on his business card. The situation has achieved the status of a mind-boggling absurdity.
While numerous organizations, including the well-respected National Fire Protection Association, are working to develop objective DR planning standards, effective DR planning remains at this writing a mixture of art and science. Effective planners require a broad base of knowledge across a variety of technologies and business practices. For this reason, disaster recovery planning is not a skill set that is easily tested or certified.
Thus, this book contends that, despite the fact that a consultant's business card contains an acronym for a disaster recovery certification body, this alone is insufficient evidence that the consultant is competent to do an acceptable job for the business client. Some of the best consultants in the field do not have letters following their names on a business card or brochure.
Speaking of credentials, over the past decade, many "Big Five" accounting firms entered the contingency planning business. In other words, the same firm that performs the company's annual audit probably offers a disaster recovery planning service as well.
Despite claims by these organizations that their audit and planning organizations are entirely separate, it is not uncommon, following an audit that discovers a missing or inadequate disaster recovery plan, for a representative of the planning services arm of the firm to pay an impromptu courtesy call on the IT manager, CIO, or other business manager. The accounting firms argue that there is nothing incestuous about this practice, but concerned observers have asked how an auditor can objectively assess a DR plan bearing the label of his or her own firm.
Undoubtedly, there are good and bad consultants in "Big Five" accounting firms just as there are in the "pure" disaster recovery consultancies. The business or IT manager should use the same criteria when evaluating either type of consulting service. The following guidelines may be useful when considering the hiring of consultants to develop the disaster recovery plan.
Check the qualifications of the consultant. It is important to know the name and background of the consultant who will be providing services. Find out how many and which companies the consultant has served and check directly with the clients for recommendations and criticisms. Be wary of using an inexperienced consultant, even if he or she reputedly has access to more experienced hands. Ideally, the consultant will be able to demonstrate a knowledge of the IT and network technology used at the prospective client's company, will understand the specific requirements within a prospective client's industry, and will have developed satisfactory disaster recovery plans for at least two other businesses within the same industry.
Ask for a project roadmap. Ask for a proposal that shows the phases and tasks of the planning project. The consultant should not view this as an illegitimate request. Over the past few years, with the increasing availability of excellent DR planning project models and improved information on the techniques and methods of recovery planning, consultants have been hard pressed to portray what they do as secret, mysterious, or otherwise beyond the reach of nonconsultants. Most consultants have planning methodologies that they adapt to accommodate specific client requirements. All the manager needs is enough information about techniques and methods to evaluate the validity of the methodology. (For this reason, even if a manager elects to use a consultant, this book will help the manager to evaluate the consultant's planning methodology.)
Check and validate proposed time and cost estimates. Read consultant proposals carefully and note, first, whether time and dollar cost estimates have been assigned to parts of the project. Unless consulting services are packaged as fixed-price contracts, there is no way that a consultant can develop meaningful time and cost estimates. The manager should be especially wary if the consultant quotes exact prices or times before knowing anything about the particular requirements of the company.
Estimates provided by the consultant can be of value to the information manager in other ways. For example, valid time and cost estimates can provide a useful benchmark for comparing various consultant proposals, especially if each consultant states that he or she is basing estimates on similar projects performed for similar businesses. This is about the only way "comparison shopping" can be performed for this type of service.
To ensure that the data being collected from each candidate is not skewed by anything other than unknown factors, ask whether all predictable costs, including the consultant's travel and lodging, are reflected in the estimated cost.
IT managers should be aware that some consultants tend to push their premium service initially, and offer less-expensive shared responsibility approaches only if they sense that they may be pricing themselves out of a contract. Faced with the prospect of losing a potential client, some consultants can become very creative in finding cost-saving measures. One manager reported that he cut the cost of consultant-aided plan development in half by offering to provide "administrative assistance" (someone to do word processing, etc.) to the consultant, and by allocating one of his employees to work with the consultant on a full-time basis, replacing the assistant to be provided by the consulting firm. Other managers have discovered that they could purchase the consultant's PC-based disaster recovery planning tool and utilize the consultant's personal services only in the up-front analysis and data collection phases of the project. Substantial cost reductions resulted in each case.
Another manager reported that the business ethics of the consultant could be discerned from the way in which he reacted to the manager's reluctance about costs. In one instance, a consultant offered to reduce costs by dropping the final two phases of the proposed project. These phases consisted of training personnel who would play key roles in the plan and maintenance of the plan document itself. Implied in this offer was the consultant's willingness to develop a paper plan that would sit on a shelf and satisfy a casual audit but provide no meaningful recovery capability!
Cases such as the one described above are certainly the exception rather than the rule. No stereotyping of disaster recovery consultants is intended—some of the author's best friends are disaster recovery planners.
Ask about the consultant's relationships with vendors of disaster recovery products and services. Managers who are considering the use of consultants also need to be aware that many consulting firms have formal or informal relationships with vendors of disaster recovery products and services. These relationships can profit the consultant's client in some cases. Using a particular consultant, for example, may qualify the client for discount rates on fire protection systems, off-site storage, or hot sites (subscription-based system backup facilities).
There is, however, a potential for misuse of these relationships. An unethical consultant may be willing to sacrifice the objective analysis of client requirements in favor of recommending a product or service from which the consultant receives a kickback. It is valuable to know whether and with whom the consultant has marketing agreements, and how these agreements may result in price advantages for the client. Most vendors will openly admit to any special arrangements, particularly when they may profit the client and improve the marketability of their service. Some consultants argue that it is partly their extensive knowledge of the disaster recovery industry that qualifies them for the rates they command.
Should the manager decide to use a consultant, whether or not the consultant admits having special marketing arrangements with vendors, he or she should pay particular attention to soliciting competitive bids for any product or service that the consultant recommends.
For many managers, the cost of a consultant-driven disaster recovery plan is the major drawback. Plans can range from $20,000 to upwards of $120,000. This is generally perceived as a cost over and above the cost for in-house plan development. Consultants respond that their price is reasonable from many perspectives.
A company electing to use in-house personnel to develop a plan must patiently wait for the novice disaster recovery coordinator to acquire knowledge that the consultant already possesses and finance the coordinator's education and pay his or her salary while doing so. Plan development is a slower process when performed by a novice in the field. In the meantime, the company's vital information asset remains exposed. Consultants also point to the fact that most plans begun by in-house personnel are never completed.
Despite these arguments, many companies elect to use in-house personnel. Even consultant plans ultimately require that in-house skills and knowledge be developed. Someone must coordinate plan revisions and maintain the plan between visits by the consultant. In addition, much of the consultant's work must be overseen by in-house personnel since the consultant is essentially an outsider who does not participate in day-to-day business operations. Also, in-house personnel must perform all evaluations of products and services to be used in the plan, partly to ensure the honesty of the consultant.
Finally, in-house personnel now have access to information about disaster recovery planning techniques and methods through special training courses, published articles and books, the Internet and World Wide Web, and by participating in "sharing" groups. So, the learning curve for the in-house planner is drastically reduced.
Generic PC-based planning tools are also now available, and several consulting firms market their own software package containing their proprietary planning tool. These tools provide a structured approach to planning for common equipment configurations. They need to be modified by the purchaser to account for specific applications, networks, decentralized processors, and other characteristics peculiar to the customer site.
Although the PC-based planning tool does not provide comprehensive answers for the novice planner, it can offer valuable models that the planner can imitate when customizing the plan to meet his or her requirements.
Another change that is supporting the development of disaster recovery plans by in-house personnel is the improvement of project management skills across all industries and business activities. The development of a disaster recovery capability is essentially a project with discrete tasks, milestones, resources, and budget. Once the principles peculiar to disaster recovery planning are understood, any person skilled in the techniques of project management can develop a competent disaster recovery plan. Many, including this author, have found that the only tools they require are old-fashioned research and communication skills; email and web browser; a word processor; and a generic, off-the-shelf, spreadsheet, database, or PC-based project management software package.
A final word on the consultant versus in-house development strategy is suggested by consultant Philip Jan Rothstein, who notes that there are other roles for consultants than performing or managing the plan development process. Consultants can be used in connection with in-house planning efforts "to perform or support certain planning phases" (such as analysis or testing) with specialized methods or techniques, "or to serve as a true consultant—meaning, a knowledge base or coach."18
Given the right consultant, such an approach has the potential to deliver the best of both the in-house and the consultant-driven planning project models.