When to Consider Managed Services

September 15, 2008


I am often amazed at the lack of qualified technology staff at credit unions with less than $200 million in assets. In firms between $200 and $400 million, I do start to see more qualified staff across the necessary disciplines, but there are often talent holes.


Credit unions need to think creatively about how to staff for success. I have found that the best methods of staffing aren’t necessarily behind the company’s four walls, especially in the technology/IT arena. This is where Managed Services Providers are an option to consider.


Should you consider partnering with Managed Service Provider for your non-core technology needs? Here are some questions to help you with that answer:


  • Can you afford the personnel costs of managing and supporting your IT investments?
  • Does change in technology and the rate of that change negatively impact your staffing efforts?
  • Would you like your IT people to spend more time focused on core systems and member facing applications? Could you do this if the basic, everyday IT “plumbing” were handled?
  • Can you afford the raw hardware and software costs for IT today? Does this part of the budget frustrate you?
  • Does compliance risk associated with DR, Security, and infrastructure keep you up at night?
  • Are you keeping pace with requirements when it comes to compliance and IT?
  • Have you developed a multi-year approach to planning technology compliance?
  • How good is your reporting in tough areas of the network related to logging and auditing?


Working with a Managed Service Provider who is a credit union specialist will mitigate many of your every day IT concerns. When you have a trusted IT partner who understands and keeps up with compliance and the technical aspects of Disaster Recovery, IT Security, Infrastructure, and IT operations, you will free up valuable internal technology resources (hardware, software, and people) that can focus on more strategic, member-facing initiatives that directly impact your bottom line.


Red Flag Identity Theft Alert – Not Such a Big Deal to Solve

July 1, 2008

The Deadline for compliance of 12 CFR part 717 is November 1, 2008.

A very good client of mine brought this to my attention recently. I see very little forward momentum within my clients in terms of implementing solutions that satisfy compliance requirements. The following is a passage taken from www.thefederalregister.com:

Section .90(b)(9) Red Flag. The proposed regulations defined “Red Flag” as a pattern, practice, or specific activity that indicates the possible risk of identity theft. The preamble to the proposed rules explained that indicators of a “possible risk” of identity theft would include precursors to identity theft such as phishing,\21\ and security breaches involving the theft of personal information, which often are a means to acquire the information of another person for use in committing identity theft. The preamble explained that the Agencies included such precursors to identity theft as “Red Flags” to better position financial institutions and creditors to stop identity theft at its inception.

As I mentioned in a previous posting, e-mail encryption solutions will suffice. The solution needs to incorporate the following components:

  • Be located at the network’s choke point
  • Be able to integrate with identity sources within the credit union via a tap 
  • Have a financial lexicon engine that automatically blocks and/or encrypts sensitive personally identifiable information.
  • Have the ability to discriminate between a social security number in context and a random set of numbers.


Believe me, all solutions are not created equally. I have seen several solutions, and I love one of them, but I have also seen others fail. 

If there is enough interest I will delve into product specifics but for now I will let my comments above stand.

Backup and Corruption Protection: “I want to go tapeless”

May 15, 2008

I constantly hear:

1) “I want to virtualize my environment.”

2) “I want to eliminate tapes and go tapeless.”

3) “I want buy a SAN.”

4) “I want to replicate between my headquarters (HQ) and DR facility.”

The key here is appropriately mixing technologies. By design, and based on the laws of physics, virtualization and SANs need speed and high performance. iSCSI is surpassing Fiber Channel this year in shipments and is the only way for credit unions to go. The cost of supporting FC and the training uplift needed for staff and personnel is not worth it. My point is that your SAN and virtual hosts need to be screaming demons from a performance perspective. 

Back-up is another story. Back-ups can be slow and can be scheduled. When it comes to back-up ask yourself what you want to do. For example do you wish to:

1) Use traditional back-up software?

2) Leverage VCB and a proxy server to back-up your virtual hosts to tape?

3) Implement an ‘in the cloud’ backup solution?

4) Use a disc-to-disc approach for backup (cool method)?

You should also ask yourself the following questions:

1) What is the method for corruption protection? This is key to the design and architecture of your system.

2) What is the present cost to back-up? Costs include Iron Mountain, tapes, and maintenance fees.

3) What is the cost to lose back-ups now? Costs include communications with members, and fines.

4) Do you want to plan for archiving now? Do you want to use a SAN approach of integrate with cheaper back-up discs when archiving? What about your core system?

5) Can you integrate all aspects of the CU? This includes Microsoft, Imaging, and Unix. Can you integrate your core system into the back-up solution? What about the imaging system? What about your Microsoft environment? In summary, can we embrace a comprehensive solution with the back-up system? 

6) Can you truly “go tapeless”? What about the O/S tape from the core system?

I hope that this gives you a general impression of what questions to ask when discussing backup, DR, SAN and virtualization.

Implementing an IT Disaster Recovery Plan That Works (Part 2)

May 3, 2008

Here’s another story of a larger credit union client of mine.

The credit union said to me one day: “Bill, we have all this stuff and my staff if good. How do we pull all this software and hardware together into a comprehensive Disaster Recovery program?” Over the past two years, the credit union had acquired the following:

  • Fatpipe load balancers;
  • VMware;
  • Backup software (Issue: tapeless versus traditional? They were partial to tapeless);
  • Doubletake;
  • Platespin;
  • HP NAS appliance;
  • iSCSI SAN from Lefthand;
  • DR Site;
  • Connectivity;

The executive summarized his predicament to me like this: “the products all appear to be good.” I agreed, but based on his current network problems, tight back-up windows, huge WAN latency and more, it appeared that several of these products had overlapping functionality causing them to argue and step on each other. “There is no way I can roll this out into production without being sure” he said.

The credit union asked me to come up with solutions in several areas, in particular the executive wanted answers to these questions:

  • How can they repurpose the HP NAS so that the investment is not wasted?
  • What is the best way to use the iSCSI SAN from a block level replication perspective?
  • How is back-up and corruption protection going to be handled?
  • What function will Doubletake play in the new design?
  • How will the WAN network respond to the new design? The credit union had a combined MPLS and point-to-point architecture.
  • Why are backups barely being completed overnight? This could be indicative of bigger issues that need to be solved first.

For larger credit unions, one must approach back-up differently, often through production uses of a SAN.

The following summary of questions will help you understand Disaster Recovery as two necessary categories including 1) backup and correction protections and 2) production SAN and virtualization.

These two categories are what I explore with clients in the process of determining the exact level of customization required for their DR environment.

Backup and Corruption Protections

1. Do you want to keep managing backup tapes for Microsoft systems, core systems, imaging systems? You should know the costs for this process (for example, Iron Mountain)

2. What is the risk for this process? You might consider losing tapes, theft, or other eventualities.

3. If you move to a tapeless system what is the local restore capability?

4. What about “in the cloud” solutions? There are some cool ones that work really well.

5. Does your backup solution provide local corruption protection?

6. If I make a call to Microsoft for support will they take my call?

7. Will the tapeless solution recover the O/S of the core processor? The only answer is no, so are you really tapeless?

Production SAN and Virtualization

1. Why do I need virtualization and an iSCSI SAN?

2. Why is virtualization perfect for my day-to-day operations?

3. Why is virtualization a ‘magic bullet’ for my DR initiatives?

4. What are the downsides of virtualization?

5. How do I know if I can virtualize a system?

6. How do I assess the readiness of my virtualization?

7. What about Microsoft systems that are iSCSI aware? SQL 2005 and Exchange 2007 are both options. You should consider how this might change your iSCSI SAN and virtualization objectives.

That’s enough information to get you started on the path to implementing an IT Disaster Recovery Plan that works. (See my previous post for questions related to a smaller credit union.)

Implementing an IT Disaster Recovery Plan That Works (Part 1)

May 3, 2008

So what does having a comprehensive IT recovery plan mean? Does it mean that you have a Business Continuity Plan? Maybe it means that you have a real chance of recovering IT systems that support employees and systems serving members?

The NCUA wants an action plan and actionable progress being made toward implementing a DR site, but what about back-ups? For clarity’s sake let’s compare a Business Continuity Plan (BCP) versus an IT Disaster Recovery Plan. Everyone has a definition for both, and pretty much everyone agrees on the importance of both so here’s my definition:

BCP Plan – A plan that recovers credit union business processes.

IT DR Plan – The technical reality of recovering processes with underlying IT systems.

I strongly believe that both are needed. For now, however, I’m going to focus on the IT part since this is where I see most of my credit union clients having difficulties, regardless of size.

What is happening now with credit unions is interesting. Here are two recent stories that highlight challenges I often come across.

EXAMPLE 1: A Small Credit Union [$68 million in assets] – When I asked him why he wasn’t backing-up his new Microsoft Systems and Imaging systems, the senior executive in charge of technology stated: “We just installed our network and since our core processor is backed-up we didn’t think that it was important to back-up the new systems right away. Our imaging system might be backed-up but our Exchange and File systems are not……” Needless to say, I was stunned; I just couldn’t believe what I was hearing. This decision, by the way, went all the way up to the board of directors.

EXAMPLE #2: Medium Credit Union [$200 million in assets] – When asked what systems are backed-up on tapeless back-up solutions, the IT manager replied: “Everything is backed up to the tapeless backup solution including the core system, Microsoft Systems, and Imaging.” The IT Manager left the credit union shortly thereafter and the CFO engaged us to help their staff do a recovery test to a reputable recovery facility. We found the following during the test:

1) Most of the backup agents were not configured properly;

2) There was no encryption between headquarters and the DR site, leaving all backup data “in the clear” when transmitting to the recovery facility;

3) They had no local restore and no corruption protection with their tapeless solution. They had decided to go with only 1 device at the DR site and forego the device at HQ.

In summary, they were unrecoverable, which is almost unbelievable! Ironically, since his old IT manager left, the CFO has “rolled his sleeves up” and now embraces network IT strategy. His comment: “if I can’t even get clean backups of my enterprise, what does it matter if I have a fancy DR site?”

So What?

My point with these short stories is to point out that credit unions are faced with interesting challenges when it comes to the basics of simple backups. Yes, the NCUA wants a DR site but what about backups?

Trust, but verify.

Here are some questions that you can ask yourself, or even better, your network support personnel – regarding your “non-core” systems like all Microsoft Systems which will be systems like File, Email, and Imaging.

1) Are the non-core credit union systems recoverable in the event of a system outage caused by hardware failure, virus, water spill, or flood?

2) Are these systems recoverable in a mini-disaster or outage?

3) Would they be offended if you asked this question?

4) What is needed to demonstrate proof? You might also consider how often you get this proof.

5) Is a non-core system going down during the middle of the day a disaster or just a problem?

6) Has your IT manager proved to you recently that Active Directory is not corrupt? What would it mean to you if it were?

7) Has your IT manager proved that the imaging systems can be recovered?

8) Can the IT Manager prove to you that all Microsoft systems can be recovered?

9) Can the IT Manager prove to you that the Imaging system can be recovered?

10) Have you asked how much time it is taking to backup all systems?

11) If non-core system nightly backups fail, do you know why? Are you notified?

12) Can you complete all system backups during the night? How tight is this window?

My next blog entry (Part 2) will focus on IT Disaster Recovery issues at a large credit union.