Blogs

Many firms invest significant amounts of time and money each year to host and manage their critical IT systems and services on the most advanced, highly available, infrastructure technology available today. It’s an absolute business requirement. We leverage highly redundant server clustering, redundant network connectivity, highly resilient storage technology, redundant Internet and Wide Area Network circuits with diverse carriers connected to centralized, industrial strength, data centers with highly robust remote access technology.  These are all important to making sure services are available to our lawyers and clients 24X7X365. 

However, when it comes to formal DR planning, readiness and maintaining a truly comprehensive "DR plan", many organization fall short. The reason for this is pretty clear. It just has not been high on the priority radar for many firms.  Most have focused the majority of their time, effort and budget dollars over the past decade on establishing high availability within the IT infrastructure, mitigating the higher risk of smaller, more localized, hardware or service failures that occur fairly regularly. Likewise, many firms have spent far less time, effort and budget resources on mitigating the risk of a large scale disaster and disaster recovery readiness and planning, which is far less likely to ever happen to the fortunate majority of us.  

Having said all this, true disaster recovery readiness and maintaining a formal comprehensive DR plan is now becoming a higher priority for law firms because it’s becoming increasingly more important to the clients we serve. As a result, many firms have added DR readiness and planning to their list of project priorities for this year.  This has some IT professionals asking, “Where do I start?”...

While what I’ve laid out here is certainly not intended to be an all-encompassing DR planning guide by any stretch, I’ll simply touch on some of the key areas to consider when working to develop a more comprehensive DR plan that you should ultimately be able to confidently share with your clients when asked. Be no mistake about it, you will be asked at some point. Depending on where you are in each of the areas highlighted below and how complex your environment might be, developing the kind of comprehensive DR plan your firm should have and your clients are now demanding can easily take a year or more to complete.  And I use the word “complete” a bit loosely here as it will continually change and evolve as your environment changes and evolves.  

People, Process and Technology

First things first. The key to kicking off a successful DR planning project is to identify all the right people, processes and technologies required to map out your plan at a high level first.  Once you have these important prerequisites identified, you’re be well positioned to move on to the rest of the DR planning stages.

It’s typically easier to kick off an initiative like this by first meeting with a smaller group of key stakeholders who can assist in the process of identifying some of the key business systems, services and technology that will need to be incorporated into your DR plan. Once you’ve established your initial punch list, you can then identify all of the SMEs associated with each. They’ll most certainly need to be active participants in each important stage of the project.  They’ll know the technology, will likely have created at least a portion of the documentation you’ll need (or will help create what is required), and they'll play an important role in the overall DR plan testing and validation process once the initiative moves out of project mode and into ongoing operations mode.

Disaster Recovery Prioritization

This part of the planning process is quite often the most challenging. When asked, most application or service owners will categorize their applications and services as most critical (tier-1).  So when it comes to DR prioritization planning, I like to use the following VERY simplistic analogy to help level set expectations relative to DR prioritization (all negotiable of course – including recovery time objectives):

Tier-1 Recovery Priority (recovery time of 24 hours or less) – Analogy for Law Firms

Tier-2 Recovery Priority (recovery time of 48 hours or less) - Analogy for Law Firms

Tier-3 Recovery Priority (recovery time of 72 hours or less) - Analogy for Law Firms

Collecting, Creating and Centralizing All DR Documentation

Many organizations already have most of the technology in place required to facilitate disaster recovery and even already have some of the tools and processes identified and documented.  So, you may find yourself pleasantly surprised to learn that you’re actually in much better shape than you may have originally thought relative to having things already documented. The key to this important stage of the project effort is to collect, update, create and then centralize all of the documentation electronically in one master repository that is maintained on-site and at least one other that is maintained off-site. Consider something as simple as providing key individuals with encrypted thumb drives that they can add to their keychains. Bottom line here is that there’s an endless list of options when it comes to being able to make an electronic copy of your DR plan available to team members today. Just make sure you track where these copies are maintained so that they can be kept up-to-date as changes are made to the master copy. At a minimum, this documentation should include each of the following:

• An environment diagram that identifies each of the components required to make each system/service work.
• A comprehensive list of servers (including server names, locations, specs and configurations)
• A comprehensive list of network and storage equipment requirements and configurations
• Application, data and configuration file locations along with backup schema, schedules and retention policy details
• Any pertinent data replication tools used along with all relevant data replication details
• A comprehensive list of all Telco, Internet and WAN circuits, configurations
• All equipment, software and service provider vendor contact information (sales and support)

Validating Your DR Plan and Establishing a Testing Schedule

This particular part of your DR readiness and planning project can often be just as challenging as getting everyone on the same page  for recovery priorities. While the objective in all of this planning is to always be ready to recover critical business services as quickly as possible in a true disaster type scenario, completing regular testing of your DR plan and technology (usually at least once annually) can be quite disruptive to the business.  Some systems recovered in alternate data centers may not provide the same level of performance that is otherwise expected when delivered out of its normal production location. This obviously needs to be thoroughly vetted and assessed as part of the overall project and then service level expectations must be clearly communicated to key firm management personnel and DR planning and management stakeholders. DR tests are often scheduled over weekends to help minimize business impact.

In addition to completing full DR plan validation testing on production systems and services, some firms also conduct “table top” DR drills that target process and procedure readiness by "talking and walking through" disaster recovery scenarios without actually initiating any DR action against production systems.  These drills are not intended to replace true DR testing and validation but rather to augment the live testing process and schedule.

Making DR Planning, Documentation and Testing an Important Part of Every New System and Service Deployment Project Moving Forward

Last but certainly not least, once you’ve successfully tackled each of the above for your current production environment, it’s critical that DR planning, documentation and testing become an important requirement for each new critical business system or service you add to your environment moving forward. 

.......

While true, comprehensive, DR planning and testing may have been able to elude the top of your priority list over the past couple years, the time has come to put it back on the list. While most of us will be fortunate enough to never have to use it, not having one that you can confidently execute and share with your top clients could soon make the difference between keeping or losing their business.