Hurricane Sandy and Widespread Telecom Outages: How Poor Disaster Recovery Measures Made the Aftermath Even Worse
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
The lowly copper cable telephone landline has been the Rodney Dangerfield of technology lately; it doesn’t “get any respect.” This was true at least until Hurricane Sandy struck.
The U.S. population has converted en masse from landlines to wireless services over the past ten years. Every day more and more people dump their landlines in favor of modern, convenient wireless services. The percentage of cell phone-only households has risen from 18 percent in 2008 to 34 percent today, according to the National Center for Health Statistics. As communications have moved from the security of the wired environment to a wireless environment, the traditional standards designed to safeguard phone services have taken a back seat. Despite the urging of regulatory authorities to think ahead and plan for disasters, these watchdogs have been paper tigers, and largely powerless to influence an industry that considers itself “unregulated.” As a result, people and business suffer the consequences of the wireless industry’s inaction through widespread communications outages both during and after storms and other disasters. According to The Federal Communications Commission, at one point, 25% of cell towers in Hurricane Sandy’s path were affected by the storm.
Could something have been done about this issue in advance? The answer is a resounding yes, but it’s probably going to take activist regulatory action. After Hurricane Katrina knocked out communications along the Gulf Coast, federal regulators proposed that wireless companies have backup power at all cell towers. Some wireless industry providers sued to block the requirement, saying it would be a financial burden. The carriers also argued regulators did not have authority to impose such rules.¹ The Cellular Telephone Industries Association (CITA), was at the forefront of this effort.²
The widespread failures experienced by all providers—and in particular the wireless providers—has resurfaced questions of whether carriers should be required by Federal and State regulators to make their networks more fault tolerant. Harold Feld, senior vice president at Public Knowledge, a public interest group, commented that “The biggest issue is they have not wanted to invest the money in hardening their networks sufficiently against a catastrophic event.”
Things were not always this way. Less than a generation ago, the central offices that served traditional telephone service were engineered to allow for only an hour of down time every twenty years. That big black Clark Kent rotary dial phone on your kitchen wall rarely failed, regardless of weather. First, it was not powered inside your home. It received its power from the central office. The rigid engineering criteria for the central office meant that backup power was not only available but also plentiful. Three days of station battery in a central office was the minimum. Many serving central offices had a backup diesel generator, stored diesel fuel, and rotated their in-ground stock of fuel by periodic generator testing.
The Recent Landscape of Telecom Outages
So what happened in the wake of hurricane Sandy?
Four years ago almost to the day, my co-author and I wrote about a similar situation here in Texas when Hurricane Ike scored a direct hit on the city of Houston. The eye of the storm passed right over downtown. Ike was a strong Category 2 storm when it made landfall, leaving Galveston a ghost town. Like Sandy, mobility became a problem and getting food, water, and other essentials into the area was difficult. The 4.5 million residents of Houston fared slightly better, but roughly 90% lost power. Despite all of this, one forward-thinking telecommunications company, Cytel, stayed on the air throughout. Read their story here.
The difference is one of philosophy. Mission critical data centers must employ adequate physical and technology safeguards. These include not only backup equipment and technology but also written Operating and Security Standards on how and why to employ them. Can you imagine the data center for your bank not having backup power? Neither can anyone else. That’s why they have had these systems for decades.
When mission critical applications move out from the relative security of a large data center to an alternate arrangement—for example to distributed LAN platforms—the protective systems or their analog must still move out with them. Absent these precautions, “windows of vulnerability” inevitably develop when technology changes, applications migrate, and standards do not keep up. This is precisely what has happened when telecommunications went wireless. Over the last decade, communication has migrated from a regulated, well organized, and backed up system to one that in many cases does not even provide backup power. Many of those communications applications are mission critical.
Additional Complications
You may find that many Competitive Local Exchange Carriers (CLECs) and long distance companies are congregated in just a few “carrier hotels” or that they derive all of their connections to the Bell network in the same access tandem downtown. Indications are that this physical congestion of carriers contributed to the havoc following Sandy.
This concentration of facilities is not always the competitor’s fault. Oftentimes the monopoly incumbent makes it difficult for competitors by making switch diversity, “ROTN” (Route Other Than Normal) cable routing, fiber optic ring technology or “dark” fiber cost prohibitive or difficult to get.
For all carriers, downed power lines, trees, and other debris complicated mobility in the company’s response. At the time of this writing some services are still out and another nor’easter storm is further complicating restoration of services.
Even in one carrier’s headquarters in lower Manhattan (a critical node of its network infrastructure) flooding hampered efforts for it to repair crippled network equipment and restore service both to its own end users as well as other carriers that use their wholesale services. These carriers include one client of mine who just completed a mad scramble to find another service provider on November 5, many days after the storm. This client is a wholesale customer with millions of calls that were blocked due to what amounts to an error by the carrier. To view this issue another way, consider that any would-be Manhattan company looking to move calls off of this carrier and on to this alternate provider due to Sandy would have been unsuccessful. Both the carrier and this alternate provider were using same underlying facilities. Clearly telecommunications is still an incestuous business.
The issues were even more pronounced for the wireless carriers. AT&T, T-Mobile, Verizon and Sprint wireless customers were all impacted. Wireless providers said that the sustained power outages outlasted their back-up battery power at some cell towers. Remember the “old days” when a three-day supply of station battery in a central office was the minimum? Remember the backup generators that these offices employed? Obviously, this was not the case here. The outages in Sandy exposed other weaknesses in wireless networks during disasters. In addition, all wireless carriers employ “backhaul” facilities largely owned by Verizon to carry calls from cell towers to their switching equipment. Even a wireless carrier who is not a Verizon customer is therefore easily impacted by a flood or other failure that affects Verizon. I have often advocated microwave radio as a backup system, and it is perfect for wireless backhaul—after all, one end of the service already has a tower, right? The trouble is it is often difficult to clear new microwave frequencies in heavily congested areas like New York City.
How to Make it Better
So what can we do about it? For starters, take a look at your local facility. Most local serving offices owned by major carriers like AT&T or Verizon are sound structures constructed of reinforced concrete. Because designs vary, you should drive by the serving Central Office (CO) for your area. Look for the obvious. Does the structure have a large portion of the surface area covered by windows? Does it appear to be in an area prone to flooding? Is there major construction activity planned or occurring in the immediate area around the CO? Ask some questions of your telephone company’s account representative. Is the CO manned 24 hours a day? What type of fire prevention systems does it employ? How old is the structure? Is the CO a tandem or end office? End offices serve only end customers. Tandems switch traffic from one end office to another. Today, many COs perform both functions. This could have a bearing on which systems get recovered first in a disaster as well as how many people might be affected in a metropolitan area. For example, loss of a Tandem all the way across town could still affect you by contributing to network congestion and blocking your calls. What plans exist for restoration in the event of fire and flood? What services would be affected?
Finally, it’s time to hold wireless providers to the same standards that were second nature to landline central office providers. These standards have been published for years and are still available to anyone inclined to look for them.
The carriers currently have a veritable birds nest on the ground in wireless services. People want these services, and demand for them is insane. Wireless service is arguably not as capital intensive as digging up streets and laying cable or fiber. The present providers have a virtual monopoly due to limited frequency allocations. Now consider the fact that the wireless analog of a residential land line that cost $9.00 a month a generation ago (in a regulated environment) today brings Wireless carriers $50 a month or more. And there is virtually no regulation on profits! Yet when people need service it is not there. I think it’s time the wireless providers stopped simply pocketing all of the money and instead invest a small portion in network resiliency. Absent the ability for the providers to act of their own accord, federal and state regulators should step in and require it.
In summary, mission critical applications are migrating out to the wireless world. Basic human safety is also mission critical. The reliability of wireless service in a disaster has become increasingly critical as more Americans ditch their landlines in favor of convenient, modern wireless devices of all types. Until carriers bring the wireless infrastructure to the same level of protection as they employed in the wireline world, we can simply expect these kinds of issues again and again. This will not only cost money; it will cost lives when another Sandy, Katrina or Ike inevitably returns.
About the Author
Leo A. Wrobel has 25 years of experience with a host of firms engaged in banking, brokerage, heavy manufacturing, telecommunications services and government. A noted author and technical futurist, Leo is responsible for many technological firsts, including the first microwave "bypass" shot in Dallas (1985). He was also the first person in Dallas to run T1 telephone traffic over a cable television system in an agreement he pioneered in 1985. Leo also served ten years as an elected Mayor and City Councilman of a Dallas suburb (but says he is "better now").
A sought-after speaker, Wrobel has lectured throughout the United States and overseas in Israel, South America, and other locations, as well as appearing on several television news programs, including KLRU's "Austin at Issue." A knowledgeable and effective communicator, he has combined his political and technical savvy in repeated engagements in order to get things done, even at the highest policy levels.
Leo Wrobel served as President and CEO of Dallas-based Premiere Network Services Inc. from 1986 until 2005. Prior to this he was Director, Network Planning and Engineering for Lomas and Nettleton and held technical positions at AT&T in its former Long Lines subsidiary. Leo is a Vietnam Era Veteran (Sgt. U.S.A.F.) and holds degrees in Electronic Systems Technology and Telecommunications Systems Technology and from Los Angeles City College as well as in Business and Public Policy from the University of Texas at Dallas. Leo is a member of the I.E.E.E., Independent Telephone Pioneers of America, Southwestern Bell Pioneers, and other noteworthy organizations.
An active author and technical futurist, he has published ten books and over 300 trade articles on a wide variety of technical subjects, including: Understanding Emerging Network Services, Pricing, and Regulation (© Artech House Books), Managing Emerging Technologies for Competitive Advantage (© 1995 Computer Economics), the Definitive Guide to Business Resumption Planning, (© Artech House Books) and The MIS and LAN Managers Guide to Advanced Telecommunications (© I.E.E.E. Books)
Find out more about Leo’s books at http://www.4ci.us.
1. Last year, the Federal Communications Commission sought public comments on how wireless providers should strengthen their systems, citing "the inadequacy of backup power" as a key reason for wireless failures during emergencies. In response, CTIA, which represents the wireless industry, urged regulators not to adopt regulations requiring back-up power at cell towers, citing in part “the unnecessarily burden wireless carriers and potentially undermine the investments and network planning that have made their networks so successful."
2. CTIA-The Wireless Association® is an international nonprofit membership organization that has represented the wireless communications industry since 1984. Membership in the association includes wireless carriers and their suppliers, as well as providers and manufacturers of wireless data services and products. The association advocates on behalf of its members at all levels of government.