Disasters come in all sizes

Burst pipe damages three businesses

Three businesses in the Millyard Technology Park were damaged by water Saturday when a pipe burst.

A restaurant and two technology companies were affected, according to Nashua Fire Rescue Lt. Byron Breda.

The pipe was in the third-floor restaurant, and water was spilling out for about an hour before the soaked fixtures triggered the fire alarm, Breda said. The water leaked from the restaurant to the second and first floors, he said.

Breda said a computer company sustained the worst damage because its mainframe got soaked, Breda said.

After arriving on scene, fire crews shut off the water, tried to recover as much property as possible and pumped out the water, Breda said. The dollar value of the damage is unclear, Breda said.


--Published: Sunday, August 10, 2008 in the Nashua Telegraph

What good is a Business Continuity Plan if the hotsite is also under 20-feet of water?

The other day, someone asked me an interesting question: “What good is a Business Continuity Plan if the client’s hotsite is [also] under 20-feet of water?”

First, at a very high level, a good Business Continuity Plan is more business and process centric rather than data and IT centric. A BCP does not replace a functioning Disaster Recovery Plan, but rather builds upon it or includes it (assuming that the DRP is viable and tested) as one of its components.
The lines between a DRP and BCP are slightly blurred as each installation tends to implement things just a little bit differently.

A good DR plan is one that when exercised (tested) or executed for real, recovers the data and IT infrastructure (including circuits, etc.) and provides a useable environment at a pre-determined level of service. In many cases the DRP may not attempt to replicate the entire production environment, but will provide lesser, yet agreed-upon level of service until a return-to-normal is possible or the production environment can be rebuilt.

In the situation where “the client’s hotsite is [also] under 20-feet of water” I have a couple of different observations, but unfortunately, no real solution to offer.

We can infer from the fact that the client did establish a hotsite that there was some sort of a Disaster Recovery Plan. Perhaps not as complete or effective as it could have been, but still a plan none the less. But was the plan successful? The most obvious answer is “NO” since the recovery site was also under water.

But, is that really true? What if the DR planning process had correctly identified the risk of the hotsite suffering a concurrent failure AND management either actively accepted the risk or simply decided not to fund a solution?

In this case, the DR plan did not fail. I’d be hard pressed to call it a “success,” but one could honestly say that it worked as expected given the situation. Now I know that this is very much like saying “The operation was a success but the patient died.” However, this does underscore the idea that simply having a DR plan is insufficient to protect the enterprise. The DR plan must be frequently tested and also be designed to support a wide range of possible situations.

If we find that the DRP didn’t fail, the Business Continuity Plan, BCP (or lack of) failed miserably. No Business Continuity is possible for this client and the possibility of eventual recovery is dependant upon luck and super human efforts.

If however, the risk associated with the hotsite flooding had not been identified to management, then the Disaster Recovery Planning failed as well as the Business Continuity Plan.

Disaster Recovery and Business Continuity are both similar to an insurance policy: It is difficult to quantify a financial return on investment and by the time any of the (obvious) benefits become tangible it’s way too late to implement!

That’s one of the reasons why governing bodies have been forced to mandate recovery. One of the best set of guidelines that I’ve seen are the draft regulations for financial institutions that were published after 9/11: “Interagency Concept Release: Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U. S. Financial System”. [Links to this and similar documents can be found by visiting NASRP.]

The requirements were something like “each financial institution should have [at least] two data centers [at least] 200-300 miles apart. Both locations should be staffed and either location must be capable of supporting the entire workload.” Unfortunately, these regulations were never signed into law. I suspect this may be due in part to the realization by some elected officials that the mileage limit would move part of the tax revenue out of their local area!

Still, the guidelines were sound – at least as a starting point. Would that have prevented the submersed hotsite example? Maybe / maybe not. It depends on distance and many other factors. Even following these draft regulations, the possibility of multiple concurrent failures exist. There simply isn’t a guarantee.

This is precisely why some companies that are very serious about business continuity have gone beyond having a single hotsite and instead have moved into a three-site configuration. As you might imagine, there are several variations of a three-site BCDR configuration. One of my favorites is where there is a “local” (say, less than 100 miles distant) active mirror site that can instantly assume the workload (Recovery Time Objective = zero) with zero data loss (Recovery Point Objective = zero). This can be achieved by synchronous mirroring of the data, but does tend to limit the distance between the two sites.

The third site is much farther away – say 1000 to 2000 miles away. Data is propagated using an asynchronous data mirroring process. Because asynchronous mirroring is used to propagate data into this third site, there can be some data lost. This will equate to a Recovery Point Objective > zero. The amount of data that can be lost is anywhere from a “few seconds” to an hour or so based on several factors including distance, available bandwidth and the particular mirroring methodology implemented. Generally this tertiary site will have a Recovery Time Objective > zero as well, as some additional processing or recovery steps may be needed before full services can be restored.

Is this a “belt & suspenders” approach? Yes it is. Is it expensive? Yes it is. Is it necessary for every client environment? Certainly not. But it is appropriate for some environments. Is this solution sufficient to guarantee recovery in all situations? No, there is still no 100% guarantee even with this solution.

With these issues in mind, I try to approach each Business Continuity Planning (or Disaster Recovery Planning) effort with the following two thoughts:

• The recovery planning process is essentially additive: Once you have a process that works up to it’s limits, then it’s time to evaluate solutions that address issues not covered by the current plan. In this fashion, there is always another recovery solution that can be added to the current BCDR environment, but each additional solution brings with it additional costs and provides marginally less benefit.
• At some point, the cost of the possible additive solutions will exceed what the company is able (or willing) to fund. Both the costs and alternatives must be clearly understood for management to make a decision that is appropriate for that company and situation.

In Summary, no BCDR solution can provide a 100% guarantee. It is very important that the limits and risks of the existing plans are correctly identified before disaster strikes.

ATT Study confirms DRJ and Forrester results

The following study announcements were found clogging my inbox on Friday...

Survey finds most businesses prepared for disasters Bizjournals.com - Charlotte, NC, USA The survey found that 77 percent of Seattle and Portland executives indicate their companies have a business continuity plan.

AT&T 2008 Business Continuity Study Continuity Central (press release) - Huddersfield, UK AT&T has published the results of its latest annual survey of business continuity practices in US organizations. The 2008 survey is the seventh such survey...

AT&T Study: One in Five US Businesses Does Not Have a Business ...Converge Network Digest - USA For the seventh consecutive year, AT&T's Business Continuity Study surveyed IT executives from companies throughout the United States that have at least $25...

Reading just these excerpts, it took me a moment or two before I realized that they were all describing the same AT&T study.

These particular reports are a little light as far as presenting the specific results from the AT&T survey, but still very timely considering the topic of my last post. The consensus of these interpretations of this AT&T study seems to be that 80% of companies have a Business Continuity plan. 59% of the respondents have updated their plan within the last 12 months, but fewer (46 percent) have had the plans fully tested during the same time period.

Using these numbers in place of the percentages from the DRJ/Forrester study referenced in the previous post, we end up with a set of calculations that look something like this:

80% of the companies have a Business Continuity Plan.
59% of the companies update their plan at least once a year. This means that (.80 * .59 = 47%) 47% of the companies have a plan that is updated at least once a year.
46% of the companies actually test their recovery plans at least once a year. This indicates that (.46 * .47 = 22%) 22% of the companies have a plan, update and test it at least once a year.

This result isn't too bad, I guess, but it doesn't incorporate the result from the Gartner study that only 28% of the planned tests actually are successful and meet all of their objectives. If we apply this calculation to the results of the AT&T study, we find that:

(.28 * .22 = 6%) Only six percent of the surveyed companies can be expected to have successful Business Continuity exercises that meet all of their business requirements.

This is discouraging news indeed.

Are we prepared?

I attended a Disaster Recovery seminar last week aimed at building a better disaster recovery plan. Some of the statistics presented sparked my interest but rather than taking the presentation at face value, I thought that some additional discussion and analysis about some of these findings would be useful.

It is fortunate that I did take a closer look as some of the statistics that were presented as fact did not bear up to close scrutiny. In fact, upon verifying the presenters’ source information it became apparent that one of the statistics I had chosen as the starting point for my analysis had been misinterpreted and used out of context of the original source document.

However, I was still interested in where this information might have led with the proper analysis, so I discarded the seminar materials and went looking for similar – but more accurate and verifiable data that that could stand up to analysis.

These are the results.

The current state of DR/BC

79%The percentage of enterprises that report having a formal and documented recovery plan in place. [Source: DRJ/Forrester article] This is a very strong showing and represents the significant progress that the industry as a whole has achieved. While no one can argue that the percentage should be anything less than 100%, 19% of the respondents indicated that they expect to have a plan in place within the next 6-12 months, leaving only 2% of the respondents with no plan whatsoever

81% Of those with a DR plan, 81% responded that their plans are updated at least once a year. 26% of the respondents indicated that their plans are updated in an ongoing fashion as part of the change and configuration management processes. Kudos to these folks! 14% update their plans quarterly, 18% update their plans twice a year, and 23% update their plans once a year. [Source: DRJ/Forrester article]

82%
82% of those that responded perform a full exercise of their disaster recovery plans at least once a year. 50% test once a year, 22% test twice a year while 10% test more than twice a year. [Source: DRJ/Forrester article]

At this point, the numbers look really encouraging. As a DR/BC professional myself I can look at these numbers and say “Wow! 80% is really good. We’re doing a great job!” On a personal level, this causes warm and fuzzy feelings as my GQ (Goodness Quotient) is set to 80.

However…

Simple numbers, such as these, can be deceiving. In fact, much better business decisions can be made if additional understanding and analysis of the overall results can be achieved.

First of all, even though each of the numbers presented so far are very close to 80%, it is important to realize that each additional statistic represents but a portion of the prior sample: There is a subsequent reduction in the effective end-product success at each iteration.

In other words, the numbers should be understood in this context:

79 out of 100 enterprises have a disaster recovery plan. (GQ=79)

81% of the enterprises with a plan update them at least once a year: 100*.79 = 79 * .81 = 64 (GQ = 64)

82% actually test their recovery plans at least once a year: 100*.79 = 79 * .81 = 64 * .82= 52% (GQ = 52)

Hmmm. So this means that only about 52% of all enterprises actually have a DR plan, update it and test it at least once a year. While that doesn’t make me as warm and fuzzy as the 80% number did, it’s still pretty good, right?

Well, maybe not. While this still appears to be a relatively positive indication, it doesn’t yet include any indication of how many of these DR plans are successful and actually meet or exceed the client requirements.

In order to proceed with this next analytical step, it is necessary to reference the results of an additional study, this one – a recent Gartner study that found: “Twenty-eight percent of organizations reported that their last disaster recovery exercise went well and met all their service targets. However, 61 percent of survey participants reported that they had problems with the exercise.” So,

28%“Twenty-eight percent of organizations reported that their last disaster recovery exercise went well and met all their service targets. So, a 28% “success” rate. [Source: Gartner article]

However, we must remember that this percentage only applies to those that actually have a plan and update and test their plan. So in reality, this is a 28% success rate of only 52% of the total. (52% * .28 = 15%)

The complete breakdown of this analysis is shown graphically here:

This indicates that only 15% of the total companies will actually recovery from a disaster as they have planned. In other words, 85% of all companies will either fail following a disaster or will experience difficulties that will cause them to exceed either their RTO or RPO or both.

It would make a great - although completely irresponsible - headline if we were to state that "85% of all organizations will fail following a disaster". Tempting to some perhaps, but no.

To do so would totally ignore the 61% (as reported by Gartner) who reported that they had "some problems with their last exercise". Since we do not have the detailed information regarding this statistic, we cannot
ascertain the severity or number of the problems they encountered. It is, however safe to assume that many of these problems have been corrected and that a portion of these enterprises will enjoy greater success the next time they exercise their DR validation program.

The areas of Disaster Recovery and Business Continuity are ones that capitalize on the benefits of a Continual Service Improvement methodology. Maintaining the plan and keeping it current and validating the plan via frequent test executions are two of the cornerstones necessary for a compliant and Resilient enterprise.

Source Information
Website: Disaster Recovery Journal, "The State Of DR Preparedness", http://www.drj.com/index.php?option=com_content&task=view&id=794&Itemid=159&ed=10, with references from the Forrester/Disaster Recovery Journal October 2007 Global disaster Preparedness Online Survey, verified 06/01/2008

Website: Gartner, "Gartner Says Most Organizations Are Not Prepared For a Business Outage Lasting Longer Than Seven Days", http://www.gartner.com/it/page.jsp?id=579708, verified 06/01/2008

Disaster Recovery is…. Boring!

Let’s face it. Disaster Recovery – at least the portion that IT is involved with – is boring. There’s no dramatic TV footage (we hope!), no flashing lights, no daring helicopter rescues and no one shouting “Clear!” as the patient is shocked back to life… In other words, there is nothing about an IT recovery that is very interesting at all to a majority of the general population.

Unfortunately, this does not help prepare the client or end-user to connect a disaster event with a subsequent IT service outage. “Sure there was a class 5 hurricane, but why doesn’t the damn ATM work?” may be the prevalent attitude. Now, that is not to say that those who experienced the disaster first hand and can actually see damaged infrastructure will share in this perception; but for individuals who reside in a different state and did not experience the event first hand – there is no intuitive reason to link the effect – the IT service outage, with the disaster itself.

So, ‘why doesn’t the damn ATM work?’ The only truly acceptable answer is “it should.”

In general, IT services are perceived as a utility function. And just like any other utility – they are supposed to work. Just as when you ‘throw the switch’ there is an expectation that the light will come on, the ATM, email server, or airline reservation system is just supposed to work. Period.

It is the industry’s realization of this that has been leading the paradigm shift from Disaster Recovery to Business Continuity.

There is simply no such thing as an instantaneous Disaster Recovery event. Business Continuity, on the other hand, is implemented to continue the delivery of critical IT services when the normal IT infrastructure has suffered a catastrophic failure.