Have a Disaster Recovery plan? How confident are you in implementing it?

The year 2021 witnessed several outages that significantly affected users across the globe. Remember the network outage on Facebook along with its associated services WhatsApp and Instagram that lasted for six hours? This was notably the second major outage affecting the social media giant in the year, with the first occurring on 19th March for 45 minutes, affecting the same services.

As per ITIC’s 12th annual 2021 Hourly Cost of Downtime Report, 44% of the surveyed firms indicated that hourly downtime costs exceed $1 million to over $5 million, exclusive of legal fees, fines, or penalties. Additionally, 91% of organizations said a single hour of downtime takes mission-critical server hardware and applications offline, averages over $300,000 due to lost business, productivity disruptions, and remediation efforts. Meanwhile, only 1% of organizations—mainly very small businesses with 50 or fewer employees—estimate that hourly downtime costs less than $100,000.

Looking at the examples of outages and their impact, we can deduce that disasters are unexpected and extraordinary. Which is why you require a well-defined and documented Disaster Recovery plan that helps in recovering lost data and accelerating a company’s return to normal business operations.

What is a Disaster Recovery plan? Why do you need it?

Disaster recovery (DR) is the practice of recovering an organization’s IT operations after a disruptive event or disaster hits it. The DR plan comprises policies and procedures to respond to the major outage to restore the infrastructure and applications online as soon as possible.

A DR plan and procedure includes the following steps:

• Recognizing different ways that systems can fail unexpectedly by identifying potential threats to IT operations
• Having a well-planned process to respond to and resolve outages
• Mitigating the outage’s risk and impact on the operations

Basically, as part of a business continuity plan (BCP) that focuses on restoring an organization's essential functions as determined by a business impact analysis (BIA), disaster recovery focuses on restoring IT operations.

So, is your system prepared to bounce back when the unexpected happens?

If not, read on to find out how.

But first, when is the right time to initiate a DR plan?

It’s crucial to be ready and able to respond in an emergency. Consider a disaster recovery plan just like an insurance policy—you need it, but you hope you never need it. Here too, you need to have a well-defined ‘plan activation.’

The decision to invoke DR is largely based on three categories of assessment:

• Technical
• Business
• Operations

So, from the time when an incident is notified, the Recovery Time Objective (RTO) clock starts ticking. After that, the maximum time at hand is:

D = RTO – MTTR (Mean Time to Recover)

Assuming a decision is taken on these bases, DR must be invoked before D is reached.

While in crisis, it’s tempting to wait a little bit more to avoid the DR invocation; this could be due to a lack of confidence in the outcome of the DR process. Having regular DR mock drills prepares for confident decision-making whenever the time comes.

How can chaos engineering help in DR planning?

Chaos engineering is about finding weaknesses in a system through controlled experiments to improve the system’s reliability. It helps identify and fix failure modes before they can cause any real damage to the system. This is done by running chaos experiments to inject harm into a system, application, or service. For instance, adding latency to network calls, shutting down a server, etc.

Just like Site Reliability Engineers (SREs) have runbooks, DRPs also contain instructions and steps to restore service after an incident. These are used to validate runbooks by performing fire drills that are intentionally created outages. GameDays (days dedicated to chaos experiments) are also conducted to test the resilience of the systems.

We can use chaos engineering to recreate or simulate a black swan event for disaster recovery. This gives us the opportunity to test the DRP and response procedures in a controlled scenario instead of recreating disaster-like conditions manually or waiting for a real disaster.

In one of my previous blogs on resilience, I discussed the top 5 pitfalls of chaos engineering and how to avoid them.

How do we inject faults to simulate a DR scenario?

You can start by defining specific DR scenarios related to a set of business functionalities. Based on the RTOs and RPOs of associated services and their dependencies, teams must also identify the what, where, and when of fault injection. This involves what type of faults are to be injected, where exactly these need to be injected, and when. The sequence of injections, severity of the faults, etc., are additional levers to stage the disaster scenario being tested.

There are several types of faults and ways to inject them, as mentioned below:

Depending upon the systems, platforms in use, etc. using one or the other of these low-level tools might not always be feasible and/or practical. The fault injection and orchestration tools also package most of the discussed tools and/or an alternative of them in an easy-to-use manner. These include:

• Chaos Toolkit
• Litmus
• Gremlin

Best practices to run a DR plan for highly complex scenarios

How can you be confident in running a DR plan while reducing the risk of an experiment that causes harm to your systems? How should you handle scenarios of scale? For instance, when triggering the DR, if it requires spinning up 500 servers, how can you optimize the operations to achieve maximum efficiency rather than operating sequentially?

Here are the key considerations for optimizing the recovery process:

• Identification of relative criticality of functions (typically having different RTOs/RPOs)
• Identification of inter-dependencies of functions, including those common functions which are omnipresent

Having a clearly defined dependency tree, you can initiate the recovery of independent nodes based on the following two factors:

• All common systems which are needed by one or more independent functions must first be recovered
• All independent functions can be initiated for recovery in-parallel thereafter

In this blog, I and my fellow experts list down the best practices of chaos engineering for successful implementation.

How Nagarro can help

In a 24/7, digital world, where disaster recovery is more important than ever, we, at Nagarro, can help you leverage chaos engineering to be better prepared for any disaster and minimize disruptions.

Connect with us today! For more on our offerings in this space, check out our Business Resilience page.