Credit Union Mergers – Mitigate Technical Risk

March 18, 2009

From my point of view a credit union merger is a ‘non-trivial’ event, however I am excited about the opportunity that this provides both entities from a technology perspective. A small credit union can come out of a merger stronger, leaner, and more efficient than before. This is an opportunity to streamline, achieve economies of scale, and combine the best of each entity while discarding the unworkable elements. The following are a list of good questions to ponder with your teams. Here is a short list of what I would consider to be the ‘tough stuff’ from a voting and discussion perspective between teams.

Technical Systems Integration Planning Steps

  • What is the plan for the coexistence of two (separate) LAN and WAN networks and what is the end state goal?
    •  IP Scheme – bridged/ routed network
    • Are the Credit Unions using disparate core systems? Different Versions? Do they have conflicting IP Scheme requirement?
  • What are the deadlines that need to be hit so that they details can be coordinated?
    •   IP Scheme
    •  Printing (Sharing, Services, Drivers)
    • Bandwidth
    •  Routers
  • What is the plan for the convergence of credit union peripheral hardware convergence?
    • Signature pads
    • Receipt Printers
    • Scanners
    • Check printers 
  • How will imaging be merged, including the old that may need to be kept for 7 years? What is the plan for current and historical images? What is the final imaging goal?
  • How will old core system records be kept (Core historicals, etc.)?

Internal Questions

  • What is the end network design?
    • WAN Architecture
    • Integration
    • POP Diversity
    • Redundancy
    • Encryption
    • QoS – quality of services to protect VoIP integrity
  • Can the credit unions use each other for DR?
  • What services will be shared?
    • Active Directory
    • Email
    • Files
    • SQL Databases
    • Domain Controller Authority
    • Imaging
    • Domain Trust
      • Is Microsoft SBS involved? If yes, there are important trust planning considerations.
  • Will Microsoft licensing be audited to take advantage of consolidation? Use a merger to negotiate and consolidate licensing.
  • What is the plan for enterprise back-ups long term?
  • How will the phone system be consolidated and converged?

So here is the summary of my merger material. I have collaborated with a couple of team mates to put this 2 part series together for everyone. I hope you like it and that it was useful to you.  


Implementing an IT Disaster Recovery Plan That Works (Part 1)

May 3, 2008

So what does having a comprehensive IT recovery plan mean? Does it mean that you have a Business Continuity Plan? Maybe it means that you have a real chance of recovering IT systems that support employees and systems serving members?

The NCUA wants an action plan and actionable progress being made toward implementing a DR site, but what about back-ups? For clarity’s sake let’s compare a Business Continuity Plan (BCP) versus an IT Disaster Recovery Plan. Everyone has a definition for both, and pretty much everyone agrees on the importance of both so here’s my definition:

BCP Plan – A plan that recovers credit union business processes.

IT DR Plan – The technical reality of recovering processes with underlying IT systems.

I strongly believe that both are needed. For now, however, I’m going to focus on the IT part since this is where I see most of my credit union clients having difficulties, regardless of size.

What is happening now with credit unions is interesting. Here are two recent stories that highlight challenges I often come across.

EXAMPLE 1: A Small Credit Union [$68 million in assets] – When I asked him why he wasn’t backing-up his new Microsoft Systems and Imaging systems, the senior executive in charge of technology stated: “We just installed our network and since our core processor is backed-up we didn’t think that it was important to back-up the new systems right away. Our imaging system might be backed-up but our Exchange and File systems are not……” Needless to say, I was stunned; I just couldn’t believe what I was hearing. This decision, by the way, went all the way up to the board of directors.

EXAMPLE #2: Medium Credit Union [$200 million in assets] – When asked what systems are backed-up on tapeless back-up solutions, the IT manager replied: “Everything is backed up to the tapeless backup solution including the core system, Microsoft Systems, and Imaging.” The IT Manager left the credit union shortly thereafter and the CFO engaged us to help their staff do a recovery test to a reputable recovery facility. We found the following during the test:

1) Most of the backup agents were not configured properly;

2) There was no encryption between headquarters and the DR site, leaving all backup data “in the clear” when transmitting to the recovery facility;

3) They had no local restore and no corruption protection with their tapeless solution. They had decided to go with only 1 device at the DR site and forego the device at HQ.

In summary, they were unrecoverable, which is almost unbelievable! Ironically, since his old IT manager left, the CFO has “rolled his sleeves up” and now embraces network IT strategy. His comment: “if I can’t even get clean backups of my enterprise, what does it matter if I have a fancy DR site?”

So What?

My point with these short stories is to point out that credit unions are faced with interesting challenges when it comes to the basics of simple backups. Yes, the NCUA wants a DR site but what about backups?

Trust, but verify.

Here are some questions that you can ask yourself, or even better, your network support personnel – regarding your “non-core” systems like all Microsoft Systems which will be systems like File, Email, and Imaging.

1) Are the non-core credit union systems recoverable in the event of a system outage caused by hardware failure, virus, water spill, or flood?

2) Are these systems recoverable in a mini-disaster or outage?

3) Would they be offended if you asked this question?

4) What is needed to demonstrate proof? You might also consider how often you get this proof.

5) Is a non-core system going down during the middle of the day a disaster or just a problem?

6) Has your IT manager proved to you recently that Active Directory is not corrupt? What would it mean to you if it were?

7) Has your IT manager proved that the imaging systems can be recovered?

8) Can the IT Manager prove to you that all Microsoft systems can be recovered?

9) Can the IT Manager prove to you that the Imaging system can be recovered?

10) Have you asked how much time it is taking to backup all systems?

11) If non-core system nightly backups fail, do you know why? Are you notified?

12) Can you complete all system backups during the night? How tight is this window?

My next blog entry (Part 2) will focus on IT Disaster Recovery issues at a large credit union.

Remote Access, Compliance, Security, and Disaster Recovery all in ONE!

April 21, 2008

This week was interesting. In the latter half of 2007, I worked with a $400 million credit union to develop a 2-year road map for budgeting and IT strategy. I’d worked closely with the credit union EVP to formulate and solidify his thoughts in strategic areas for the executive team and board. Business optimization was the goal for each and every IT decision.

Together we had made the shift from direct integration support to blending integration with a more proactive planning approach, including an IT roadmap to development for the next two years.

What made this remote access decision interesting was to observe how the credit union turned a seeming tactical/technical decision into a brilliant strategic decision. Here’s a summary of the situation.

Overview of the problem:

* 25 remote access users need remote access to the credit union network.
* SSL certificates are expiring.
* Simplicity and security are paramount at the network edge.
* The edge device has to be an intelligent perimeter to aid the inspection engine.
* Avian Flu remote access support is needed.
* There is a DR bump license requirement.
* Legacy Citrix remote access technologies are in place (including CSG, NFuse, and cert server) and there is no desire to move to a newer weak Citrix remote access product.
* 3 quotes are needed from 3 quality vendors
* Integration with two-factor authentication is needed
* DR site integration.
* Tight Citrix integration.
* Ease of management.
* Full client integrity and security policy enforcement is needed at the end points.

So I arranged for 2 new SSL VPN product demonstrations. The credit union IT team reviewed product demos from Citrix, F5 Firepass and Sonicwall/Aventail.

How can an SSL VPN be strategic?

In a previous blog entry I spoke about the importance of client integrity for credit unions as they develop their security strategy. My company and I have been recommending and integrating SSL VPNs for about 6 years now and have seen the client integrity aspects of these products morph and change quite a lot. Read my previous blog on SIP and client integrity to get a list of questions that you need to ask when reviewing solution sets in this security category.

When it comes to strategy, however, make sure you look at SSL VPNs from the “end game” perspective. You need to ask yourself:

1. Do I need to have a remote access plan in the event of a disaster?

2. How do these devices communicate with one another from HQ to the DR site, and how do they keep their access policies in sync?

3. Which one is most maintainable and supportable by your internal staff?

4. Do they have a virtual appliance strategy?

5. Can I incorporate this technology into my virtualization strategy?

6. How far can you stretch $1.00 of spending in regards to your roadmap for Security, Disaster Recovery, and Infrastructure?

7. What does the management interface look like?

8. How do you set up a security policy? For example, what will be the security policy for remote access users at a conference, on a trusted laptop versus un-trusted device like a kiosk?

9. How will you enforce security hitting Exchange OWA versus Exchange using Citrix?

I hope that that helps. My point as always is to never, ever, make an IT decision on technology alone. Always make the business a partner.

CASE STUDY: How to Recession Proof Your Credit Union with Strategic IT Spending

April 21, 2008

Here are a few “what-if” scenarios for your IT vision:

*What if your IT infrastructure were “plug-n-play”?

*What if your IT infrastructure were “portable”?

*What if your countless data bases were “portable” and “consolidated”?

*How would your DR and infrastructure spending change if you knew that $1.00 in spending could gain you critical compliance momentum regarding DR, infrastructure, and security?

*What if these databases were replicated to your DR facility on the same platform as your Exchange/email and file systems?

*What if you were able to buy less hardware?

*What if you had a two-year roadmap of technical and reality-based spending to support this vision?

In today’s lean business climate, credit union executives must ensure that every dollar of day-to-day operational and compliance IT spending is doing triple duty within DR, infrastructure, and security. So what is your vision?

How would you get started? A story about a client of mine comes to mind that you might think of as a real-world case study direct from the trenches. I had worked closely with this client in late 2007 and early 2008 to formalize their strategy with Infrastructure, DR, WAN, Security, Citrix, VoIP and databases.

It is important to note that this credit union created the technical roadmap, design, and architecture that they wanted to pursue. They submitted their objectives to the SAN vendor community based on what they wanted to accomplish rather than doing the rounds and gathering opinions.

Seeking a Desired Solution

After deciding on a lowest common denominator for what their needs were, the credit union determined that they had a two-part goal. The first phase was to implement an iSCSI Storage Area Network – SAN at their Headquarters and DR facility according to the criteria listed below. The second phase was to create a “portable” and ‘plug-n-play’ infrastructure using VMware virtualization.

Overview of Current Environment Problem

The credit union was experiencing storage growth issues in the following areas:

1. Email;
2. Knowledge Base;
3. VoIP System recording data;
4. Imaging system data base;
5. IT Software (media storage);
6. Instant messaging;
7. Voice mail;
8. Fax;
9. File Servers;
10. Security (IDS logs, FW logs, internet logs);
11. Databases (Member Account Statements, Visa Statements, check copies, financial reports, statements);
12. Scalability regarding how to deal with the growing storage;
13. Unconsolidated storage (SQL data base proliferation);
14. Compliance with future regulations (future GLBA), retention, and archiving;
15. Cost to maintain multiple backup solutions;
16. Tape encryption;

The Requirements

The credit union wanted to purchase an iSCSI SAN in order to accomplish the following:

Primary Goal 1

Eliminate Iron Mountain tape costs
Eliminate isolated tape back-up solutions for:

i. Microsoft systems;
ii. The Synergy system;
iii. Core system;

Eliminate recurring maintenance costs for backup software
Ease staff storage management
Accommodate growth
Enable data encryption
Data storage consolidation for:

i. Imaging data base;
ii. Microsoft databases;

Consolidate file storage
Operations replication of live data between Headquarters and the DR facility
Backup replication between headquarters and the DR facility
Comply with current and future IS&T NCUA regulations
The system should be able to support future email archiving functionality

Primary Goal 2

Create a “portable” infrastructure accomplished using virtualization with VMware technology.

In a future entry, I will examine the questions that this very visionary credit union CIO asked himself — with a little help from yours truly — as he went through the budgeting and planning process for technology platform changes. I will review the overall roadmap and execution plan that was developed as we worked together to present an overall IT strategy for his credit union. Questions like the following were asked:

What do I do with my Citrix environment when using VMware?
When do I move to Exchange 2007? How does this impact my SAN and virtualization?
What impact does this SAN have on my switching infrastructure?
What good is my new plan without an MPLS network?
How will an MPLS network react to replication traffic?
How will my VoIP traffic be impacted?

In this entry I have reviewed strategy as it relates to the SAN and virtualization. The critical concern is to maximize the value of your investments. A planned approach results in an optimized situation for IT as well as for the entire business.