So what does having a comprehensive IT recovery plan mean? Does it mean that you have a Business Continuity Plan? Maybe it means that you have a real chance of recovering IT systems that support employees and systems serving members?
The NCUA wants an action plan and actionable progress being made toward implementing a DR site, but what about back-ups? For clarity’s sake let’s compare a Business Continuity Plan (BCP) versus an IT Disaster Recovery Plan. Everyone has a definition for both, and pretty much everyone agrees on the importance of both so here’s my definition:
BCP Plan – A plan that recovers credit union business processes.
IT DR Plan – The technical reality of recovering processes with underlying IT systems.
I strongly believe that both are needed. For now, however, I’m going to focus on the IT part since this is where I see most of my credit union clients having difficulties, regardless of size.
What is happening now with credit unions is interesting. Here are two recent stories that highlight challenges I often come across.
EXAMPLE 1: A Small Credit Union [$68 million in assets] – When I asked him why he wasn’t backing-up his new Microsoft Systems and Imaging systems, the senior executive in charge of technology stated: “We just installed our network and since our core processor is backed-up we didn’t think that it was important to back-up the new systems right away. Our imaging system might be backed-up but our Exchange and File systems are not……” Needless to say, I was stunned; I just couldn’t believe what I was hearing. This decision, by the way, went all the way up to the board of directors.
EXAMPLE #2: Medium Credit Union [$200 million in assets] – When asked what systems are backed-up on tapeless back-up solutions, the IT manager replied: “Everything is backed up to the tapeless backup solution including the core system, Microsoft Systems, and Imaging.” The IT Manager left the credit union shortly thereafter and the CFO engaged us to help their staff do a recovery test to a reputable recovery facility. We found the following during the test:
1) Most of the backup agents were not configured properly;
2) There was no encryption between headquarters and the DR site, leaving all backup data “in the clear” when transmitting to the recovery facility;
3) They had no local restore and no corruption protection with their tapeless solution. They had decided to go with only 1 device at the DR site and forego the device at HQ.
In summary, they were unrecoverable, which is almost unbelievable! Ironically, since his old IT manager left, the CFO has “rolled his sleeves up” and now embraces network IT strategy. His comment: “if I can’t even get clean backups of my enterprise, what does it matter if I have a fancy DR site?”
My point with these short stories is to point out that credit unions are faced with interesting challenges when it comes to the basics of simple backups. Yes, the NCUA wants a DR site but what about backups?
Trust, but verify.
Here are some questions that you can ask yourself, or even better, your network support personnel – regarding your “non-core” systems like all Microsoft Systems which will be systems like File, Email, and Imaging.
1) Are the non-core credit union systems recoverable in the event of a system outage caused by hardware failure, virus, water spill, or flood?
2) Are these systems recoverable in a mini-disaster or outage?
3) Would they be offended if you asked this question?
4) What is needed to demonstrate proof? You might also consider how often you get this proof.
5) Is a non-core system going down during the middle of the day a disaster or just a problem?
6) Has your IT manager proved to you recently that Active Directory is not corrupt? What would it mean to you if it were?
7) Has your IT manager proved that the imaging systems can be recovered?
8) Can the IT Manager prove to you that all Microsoft systems can be recovered?
9) Can the IT Manager prove to you that the Imaging system can be recovered?
10) Have you asked how much time it is taking to backup all systems?
11) If non-core system nightly backups fail, do you know why? Are you notified?
12) Can you complete all system backups during the night? How tight is this window?
My next blog entry (Part 2) will focus on IT Disaster Recovery issues at a large credit union.