Once you are faced with the reality that your business operations have been compromised - time is of the essence. Having a thorough Disaster Recovery Plan will save you substantial amounts of time, money, and resources. A Disaster Recovery Plan (or runbook) is a working, living document that is unique to every organization - the business' specific blueprint on how to recover quickly and efficiently from a DR event. However, a lot of DR plans are not thorough enough to serve as any help during an actual disaster event.
A Disaster Recovery Plan explains not only the necessary steps to recover from a DR event, but it also lays out the current infrastructure, people involved and their specific roles, the failover environment, the failover process and the failback process. It's a set of organized, Standard Operating Procedures (SOPs) specific to a DR event. The plan should provide consistency that when done repeatedly, there is little to no deviation or failure to execute.
Having a thorough DR Plan is the difference between a successful and an unsuccessful recovery. Below you will find the seven key elements of a successful Disaster Recovery Plan:
- Communication: This is essential during a disaster event. Ensure that other lines of communication such as landlines are open during the event of a disaster. You should not solely rely on emails and cell phones. If all else fails - determine a centralized meeting location that your team can meet at in person.
- Roles and Responsibilities: In the event of a disaster, stress and chaos can take over your team - so having delegated roles and responsibilities can ease the stress. Every individual on your team should know what their role is during an event. You should have who does what, who contacts who, etc., all written out in your plan. In the case your key decision makers are unavailable, someone should be assigned as the backup key decision maker.
- Logins: The login to initiate the recovery process is vital information and should not be available to everyone on the team, to mitigate the risk of the login being compromised. Additionally, not only one person should hold the login credentials, in the occasion they are not present when disaster strikes.
- Remote Access: Today, workplace location looks different for every business; in office, fully remote, and hybrid are all different models that businesses operate under. In the case you are not at the single location where the recovery process needs to be initiated, remote access should be available. With remote access, you should have the capability to monitor and manage business operations efficiently and effectively.
- Documentation: Your Disaster Recovery Plan should be clear, comprehensive, and thorough. Individuals on your team should be able to read the plan and know exactly what action to take, when to take the action, along with a step-by-step guide to instruct them on how to complete the tasks.
- Test, Test, Test: Disaster Recovery Plans should not be written with hopes it will automatically work in the event of a disaster. Your plan should be tested routinely throughout the year, at least twice. By testing your plan, you will be able to fully understand the capabilities of your solution. Additionally, if there are any missteps during the test, your team will be able to address them directly without the stress surrounding an actual disaster event. With routine testing, your plan can be changed or updated to reflect the most current business requirements for complete business continuity.
- Revise Your Plan: Disaster Recovery Plan's are meant to serve as a living document that encompasses roles, actions, and processes for your team. If your DR plan is years old, it needs to be updated - roles could be out of order, the DR solution could have changed, logins may have been altered, etc. Your DR plan during an actual disaster is only as useful as the effort and time you put into it when there is NOT an actual disaster.
Review your current Disaster Recovery Plan and assess if it contains all of these elements of success. It cannot be overstated that everyone involved should be familiar with the plan, every detail of the plan is documented, and it is up-to-date to reflect all aspects of the current IT environment along with roles assigned to team members. Along with the key elements of a successful DR Plan, Net3 Technology's Director of Sales Engineering, Devon Stephens, has created "Do's and Don'ts of Disaster Recovery" to help minimize the pain, inconvenience and consequences of a disaster - check out the webinar.
The Do's of Disaster Recovery
- DO Get management/ownership on board: Preparing for a disaster event requires significant resources that span departments. It is critical that your management team is onboard and understands the impact of what the team is trying to accomplish.
- DO Have a DR team: This has to be a team effort, there is no way a single person can coordinate and document all the requirements of a business - the team needs to have both technical and non-technical members.
- DO Document your DR plan: No plan is a plan unless you write it down, putting it down on paper gives you and your team the ability to make the knowledge needed during an event accessible to everyone.
- DO Implement DR toolsets according to your plan: The tools and processes you are going to put in place to provide a disaster response need to fit the plan. If you state in your plan that certain parts of the infrastructure need to be running within 30 minutes, a recovery from tape backup isn't going to fit the bill.
- DO Automate as much as possible of the plan: Disasters, no matter what kind, are emotionally, mentally and physically stressful. In moments of stress, our memories go, responses can be haphazard, and people can make big mistakes. Eliminate as much of the detail work as possible by automating with scripts and configurations that can be done ahead of time.
- DO Test frequently against different scenarios: Testing is the single most important thing in this list. Unless you test and ensure that your plan is encompassing all the elements needed for your business to run - you don't have a plan.
The Don'ts of Disaster Recovery
- DON'T Limit your plan to IT services: In the best cases we have seen, IT was seen as only a part of the larger DR plan. The company as a whole needs to be prepared for how they are going to react during a disaster event. Where are people going to work? How are checks going to be mailed to vendors? How do we provide assistance to employees whose homes are affected by the disaster? The business as a whole has a responsibility, not just the IT department.
- DON'T Let your plan stagnate: Business needs change. Whether a new software goes into use, or a remote policy is put in place, review the DR plan on a regular basis to include these changes.
- DON'T Let the tools fall into disrepair: Backup and recovery toolsets can be tough to manage. They require maintenance and attention to alerts. We have seen far too many cases where a system has failed, and because backups were failing or not enabled - that system was not recoverable.
- DON'T Assume you will be the one enacting the plan: During a disaster event, there is no guarantee that the people who wrote the plan will be the ones acting on the plan. When people move in on their careers or are affected by the current disaster in a way that makes them unavailable, the plan needs to be written carefully enough that anyone can follow along and make it happen.
- DON'T Depend on HA or redundancy: Highly available systems and redundant systems are critical to a stable network infrastructure. But since the data needs to be in two places at the same time, most of them rely on synchronous replication which is not appropriate for a DR scenario. Asynchronous, journaled replication is key to successful DR.
- DON'T Forget to ask for help: Disasters can be overwhelming for an IT team. While you're working to get infrastructure up, there are people on the sidelines just watching. If you have a service provider like Net3 Technology, contacting a reliable team will be helpful, allowing them to assist in getting machines running and making sure networking is good.
If you don't currently have a thorough Disaster Recovery plan, a good place to start would be to complete our Business Impact Analysis - it will help determine your IT readiness for a disaster event. If you want to see the true impact a disaster event could have on your business, fill out this downtime calculator.
At Net3 Technology, we strive to help customers become Cloud Confident, which includes being confident in their Disaster Recovery Plan. Because of this, we assist each customer with their customized DR Runbook - here is our FREE Disaster Recovery Runbook Template to get you started.
