Managed Services for Credit Unions: The Difference between Surviving and Thriving

December 15, 2008
When I search the internet, I see virtually no mention of managed services in the credit union space. This surprises me. 

I hear too many credit union CEOs wonder how to survive. I offer an alternative vision for what is possible for your credit union: Thriving. 

And I assert that any credit union from $20 million to $200 million in assets must be using a Managed Services Provider in order to thrive.  

Think this is a bold statement? Perhaps it is. Let’s look at the facts before you decide. 

Small to small/mid credit unions are faced with managing a level of IT complexity that no other business of the same size must manage (other than, perhaps, health care). The complexity is created because of five requirements: 

  1. Compliance
  2. Security
  3. Third party relationships (e.g., ATMs, Shared Branching, Home Banking, Core Processing, Fedline)
  4. Disaster Recovery
  5. Infrastructure Operations 

No small to small/mid credit union can effectively manage IT through in-house staff alone.  Staying focused on driving member value is critical. Diverting the IT department to review  and maintain “plumbing systems” when they could be reviewing, implementing, and evaluating systems that enhance the value of the credit union in the eyes of the members–that is where IT has to be focused. Resource coverage in the areas mentioned above is too challenging, and too risky to try with only in-house staff. 

In the IT space, a step that supports thriving is outsourcing your IT operations. Hire a Managed Service Provider (MSP) who can handle all the blocking and tackling of the five items I listed above. 

I have noticed that a credit union that has reached over $100 million in assets typically has one person on the IT staff who is smart and capable. Without an MSP in place, this person invariably ends up trying to do everything. Rather than tying up this valuable resource on housekeeping chores, have your MSP report monthly to this person. Require that the reporting be compliance based in nature and not all technical; if this is not required then you are still saddling your key IT Manager with the burden of producing the proof needed each month. 

I can’t stress this point enough: Shift your key manager’s focus to member-facing projects and have the MSP deliver the rest. This will put the company on the road to thriving and, from a professional growth perspective, it places your “shining star” IT employee in a position of managing the plumbing versus doing the plumbing, which should be a welcome step up for any bright, ambitious manager.



What’s Wrong with This Picture? (and How to Put It Right)

November 15, 2008


“I’m the CFO, it’s not my job to worry about IT.”


I have noticed an interesting trend over the past several months that I find exciting. This is the heavy involvement of Finance (Controller and CFO) in IT, not just in decision-making and approvals of IT investment, but in the strategic planning process. I am very encouraged by this.


If your senior financial management is not involved in the IT function of your company, I strongly suggest that you consider fixing this situation. Here is a cautionary story that illustrates the problems that a company can face when it doesn’t involve non-IT decision makers in the IT planning process. It illustrates why the CFO must care.


We had a non-credit union client recently who was experiencing tremendous pain around complaints from a user community of about 350 users distributed over 14 sites. They had just had a turnover of IT management at the highest level, and this is where I got involved.


The user community complaints were actually a symptom of a much deeper and more serious issue.  In the course of our engagement with senior management, we uncovered eight years of executive management neglect of the IT function. It wasn’t malicious neglect; it was unintentional neglect that arose from a lack of a vision, strategy, and long term IT roadmap upon which to base financial and management decisions. There had been no involvement of non-IT executives; as such, IT was not aligned with business vision or strategic objectives.


How did this happen? How did they get themselves into this predicament? Here are two examples among several:


  1. Their WAN was creaky and old (one of the oldest I have ever seen), but there was no attention on uplifting the infrastructure as part of an iterative and ongoing strategy. A major core business application was rolled out to all sites across , and since no attention was paid to shoring up the infrastructure before application installation, infrastructure performance took a steep (and problematic) drop.
  2. The company was encouraged by their VoIP vendor to purchase a brand new VoIP system. Three integrators later, they were left with the most complicated VoIP routing and switching installation I have ever seen. To make matters worse, they have never received the expected value from the investment.


The good news is that we are working with management to fix things. The company must now allocate significant spending to IT in order to make up for the years of little to no investment in infrastructure, disaster recovery, compliance, and other key program components. Though this is a somewhat bitter pill to swallow, it has had the good result of gaining the CFO’s attention and interest.


The new IT goal set collaboratively by the IT manager, the CFO, and the Controller is stable, simple, and maintainable systems that produce happy users. They wanted a high quality ‘austere’ network—not “cheap,” but “no frills.”


This company also made the decision to go with a Managed Services Provider (MSP) as part of a strategic move to focus their limited but talented IT resources on core business activities. They determined that as far as third-party relationships, they didn’t want a tactical IT partner—that is, a provider that only manages a device or set of devices. They wanted a partner that would participate in strategic planning, design, and architecture, as well as a partner who could assist them in day-to-day management of sophisticated devices from Tier 1-Tier 3 support.


Areas that we recommended they turn over to an MSP encompassed much of the security infrastructure, including the DMZ, firewalls, SPAM filters, SSL VPN, Load balancers, QoS devices, AD, Servers, and Consolidated Event Management. (The caveat, of which they are cognizant, is that an MSP can only be brought in after their infrastructure has been assessed and remediated.) Hiring and managing the in-house talent to effectively support all the equipment listed above would run $80-110k per year; the MSP we recommended performs the same services for $48k per year.


One of their primary goals, right after end user happiness, is network stability for the VoIP system. We encouraged them to focus on simplicity in order to make the network able and supportable. Since they had determined that they did not want their core IT staff supporting a non-business value add system then this system also had to be simplified so that the MSP taking over the VoIP management wasn’t saddled with the same issues.


We continue to work with senior management on effective IT strategy. As far as next steps, the CFO wants an IT roadmap, that is, a doable plan that is sized right for the company. Immediate action items include:


  1. Data Center power distribution and re-cabling.
  2. Replaced the 10-year-old ATT WAN with a new Sprint MPLS WAN.
  3. Virtualization (there is no more server rack space left)
  4. Disaster recovery site implementation
  5. Employing a different back up method from the tape backups currently being used.
  6. A comprehensive Microsoft licensing strategy that includes an audit of current licenses.


My reason for providing a high level of detail in this story is to give you clear examples of IT issues that may track with your own. If any of the problems or strategies that this client is dealing with ring any bells for you, it may be time to examine your own IT function and how your financial management relates to it. If your senior financial manager is not getting involved with IT strategy or decision making, you may want to better align the two. If you don’t, there may be trouble brewing behind the scenes.

Managed Services for Credit Unions – What a Great Idea!

October 15, 2008

Over the past few months, I have been hearing a similar complaint from a number of my clients. One after another, they have observed that “just can’t keep up the pace.” IT overload, in a sense. 

Frankly, I was amazed to hear this, especially since I kept hearing it over and over. These are very smart managers, and many are moving up the ranks in the credit unions they work for. I could understand “I can’t keep up” stress from a business person trying to manage enterprise technology, but I was shocked to hear these words from seasoned and more than capable technologists.  

I thought of Managed Services Programs (MSP) as a solution for my frazzled clients. Managed services could free up my clients to focus on member-facing value add and other strategic items. If credit union technology professionals could stay focused on increasing value to the member and less on the IT plumbing systems, credit unions and their members would be immensely better off. 

MSP for credit unions can cost half of the cost of the same in-house services. This gives credit unions a great advantage; they can obtain the plumbing expertise through contractors and invest their W2 resources in core systems and member facing applications. 

The path to MSP success starts with finding a Managed Services and Compliance Program Vendor who focuses specifically on credit unions. Credit unions are more complex than many similar sized companies in other sectors, and it is very important that your IT business partner knows this and has credit union expertise in their business.  

Here are other key items to check out before closing a deal with an MSP provider: 

  1. Do their programs match to an Information Security Program, NCUA, or FFIEC?
  2. Are compliance and IT operations the focus of their service (as opposed to devices and products)?
  3. Have you reviewed samples of monthly reports?
  4. Have they been questioned by NCUA auditors before? How did they do?
  5. Is access to Tier 1-Tier 3 talent included in the monthly fee?
  6. Is there a complete 3rd party due diligence package for: insurance coverage, financials, security controls, background checks, NDAs, etc.?

Finally, something to consider from your side: Will bringing this provider on board really enable you to focus your valuable in-house IT and business personnel on core systems and member facing activities? 

If the answers to these questions are positive, you stand an excellent chance of reaping big benefits from partnering with an MSP provider.

When to Consider Managed Services

September 15, 2008


I am often amazed at the lack of qualified technology staff at credit unions with less than $200 million in assets. In firms between $200 and $400 million, I do start to see more qualified staff across the necessary disciplines, but there are often talent holes.


Credit unions need to think creatively about how to staff for success. I have found that the best methods of staffing aren’t necessarily behind the company’s four walls, especially in the technology/IT arena. This is where Managed Services Providers are an option to consider.


Should you consider partnering with Managed Service Provider for your non-core technology needs? Here are some questions to help you with that answer:


  • Can you afford the personnel costs of managing and supporting your IT investments?
  • Does change in technology and the rate of that change negatively impact your staffing efforts?
  • Would you like your IT people to spend more time focused on core systems and member facing applications? Could you do this if the basic, everyday IT “plumbing” were handled?
  • Can you afford the raw hardware and software costs for IT today? Does this part of the budget frustrate you?
  • Does compliance risk associated with DR, Security, and infrastructure keep you up at night?
  • Are you keeping pace with requirements when it comes to compliance and IT?
  • Have you developed a multi-year approach to planning technology compliance?
  • How good is your reporting in tough areas of the network related to logging and auditing?


Working with a Managed Service Provider who is a credit union specialist will mitigate many of your every day IT concerns. When you have a trusted IT partner who understands and keeps up with compliance and the technical aspects of Disaster Recovery, IT Security, Infrastructure, and IT operations, you will free up valuable internal technology resources (hardware, software, and people) that can focus on more strategic, member-facing initiatives that directly impact your bottom line.

A Buyers Guide for MPLS Wide Area Networks Questions to Ask (Part 2)

August 15, 2008

The following is a continuation of a previous posting where I review common questions and compliance issues that credit unions frequently ask. I hope that this helps in your planning and decision-making process. 

1. Flexibility In Case Of Disaster: Recovery and Avian Flu Remote Working. This topic is big from a business resumption perspective and it is important to have options during uncertain or chaotic times. Has the carrier explained how it will handle the redirection of branches to the DR site in the event of a disaster? What is the process for such a redirection? Can instant changes be made if the CU owns the routers? What about instant changes to a “managed service” carrier?  

2. Flexibility When Faced With Day-To-Day Operational Issues. This issue relates to finding and determining problems on your POP diverse sites, or “Big Branches”, including routing issues on two routers or Class of Service (CoS) issues between two separate routers.

Another situation might be whether or not a new subnet or manned subnet can have encryption, Quality of Service (QoS), and routing all happen seamlessly and immediately after having zero calls to the service provider. This situation might occur if the credit union or a 3rd party local integration vendor needs to perform troubleshooting. If your answer is no, then what is the process? 

3.IMPORTANT: MPLS Project Rollout. You should know how an MPLS project rollout is handled if the CU chooses either a managed or unmanaged product. There is a massive difference here between each carrier’s approach. Some carriers are so cumbersome that it adds a significant burden to the project rollout. To bid on and manage the rollout communication process between last mile carriers (Verizon) and long haul carriers (Qwest, AT&T, Spring), you might consider using a broker.

In a recent conference call I participated in, two credit union clients of mine were discussing MPLS. One had successfully rolled out their MPLS to 14 sites last year, and one was in the final stages of their selection process. One of the important suggestions my client made to the CU was to clean up cabling prior to rollout. This means extending DMARCS and ensuring there is necessary documentation and cabling in place for the last mile carrier technician. I couldn’t agree more: nothing has delayed MPLS rollouts for me more than this issue.

4.Security. Understanding the recommended security architecture is essential. There are enormous differences in carrier options, and the nuances will make your head spin. Carrier marketing department security will not suffice for a credit union. You should know the overall MPLS encryption architecture and understand HQ to DR site encryption. 

If the carrier is recommending a “hosted” call center, then you should know how the network encryption will be addressed, as well as laptop security access into the credit union network. Security issues revolving around working at home, traveling, and avian flu, and DR remote site workers all need to be taken into account.

 5.Disaster Recovery and Wireless Access. It is necessary to know how remote workers can access the CU network in the event of a disaster. Would users be accessing the Internet and hitting the CU network via the credit union’s own security infrastructure, or would the employee be leveraging an “in the cloud” solution for remote access? A client of mine uses an “in the cloud” firewall service, and it makes me very nervous….with a Tier 2/3 provider as well.

6.Credit Union Network Remote Access For Home Worker or Travelers. You will definitely need to describe how home workers and travelers will remotely access the credit union network using the new MPLS network. Will users be utilizing the CU security infrastructure (a private approach), or a carrier network via another method (semi-private approach)? You might also be asked whether “end-point” security is offered for remote access users, and to explain how it works.

I recommend a CU use its own security infrastructure in most cases to enable remote access for the network and to provide end-point security.

7.Quality of Services (QOS) or (COS). This issue relates to disaster recovery QoS protection from HQ to the DR site. You need to know how the carrier will protect replication traffic and prevent ripples that will affect the entire network. In my experience, carrier QoS/CoS will not work for a credit union. Whatever they might call it (four buckets of protection, color coded etc.,), it is all marketing hype. Credit unions need granularity in QoS. I will write about QoS in more detail as requested.

You might also be asked what would happen if you needed to examine Citrix traffic and manage applications within ICA (independent component analysis) or print traffic within ICA. Also, how fine grained and differentiated can you get with traffic types? You should understand the following types of traffic:

I) AD (replication versus authentication)
II) Exchange
III) Citrix ICA
IV) Terminal Services RDP
VI) Citrix
VIII) VDI – other
IX) DR Replication

Other differentiations you will need to make are those between CIFS (Common Internet File System) traffic and IPC (Inter Process Communication) pipe traffic as well as authentication traffic versus AD replication. How are RTM (Round Trip Measurements) for servers and clients met and achieved, and what is the sampling rate? How are they doing this at a layer 7 level? You will need to know how many buckets of CoS/QoS you get (carriers give you between 4 and 6), and whether or not it is layer 7-aware. Finally, issues regarding how bandwidth and queuing is controlled should be understood, and how you achieve ZERO dropped packets.  

8.The SLA (Service Level Agreement) . For this type of question, you are going to need to know the SLA process, including how the service is set up, including how to get a troubleshooting-engineer. If you have a POP diverse routing problem, would the same engineer be likely to fix problems with encryption and VoIP jitter issues?

Other issues might include how to find a noisy broadcasting NIC (Network Interface Card), including which escalation path should be chosen and how long it should be. You should also know how many mission critical calls versus troubleshooting calls you get monthly, and who determines the SLA. Do you determine it or do they? Do they have an SLA example that has been set up for a client?

I hope that this guide helps you during the implementing of your MPLS network. You should now have a good idea of what questions might be asked, and what to look for in establishing solutions for your credit union.

A Buyers Guide for MPLS Wide Area Networks Questions to Ask (Part 1)

August 7, 2008


This guide has been put together to help credit union buyers quickly understand the nuances of MPLS networks. After being personally involved in the implementation of all major carriers, I have developed considerable knowledge and gained valuable insights into the details of network implementation. I think this guide will be useful for any credit union, regardless of size, to help wrap one’s mind around the scope of these projects. Hopefully this list will spur your thoughts to deeper questions about carriers, integrators, and your own staff.

MPLS migrations have become very common for credit unions over the past few years due to the fact that MPLS (WANs) offer credit unions tremendous flexibility, especially router flexibility in the case of a disaster. An additional impetus is that carriers (Sprint, AT&T, Verizon, Qwest) are trying to move clients away from older technology and toward their newer, cheaper to maintain, platforms. But just because carriers say in a slide presentation that they offer a particular functionality, credit union buyers should be aware that this DOES NOT mean that a particular functionality is good for the credit union. I have witnessed disasters first hand stemming from marketing miss-information regarding what the big telecoms sell. What you should remember is that the slides that you are observing in sales presentations are marketing slides. They are not “how to” guides nor are they necessarily accurate representations of current technology or products offered by the carrier.


To develop a good branch strategy, the following items should be kept in mind.

1. Bigger branches need High Availability, meaning that despite there being no disaster at the HQ Data Center, branches cannot communicate with core systems.

2. The CU must be able to seamlessly connect to the core system resources through the proper handling of routing paths. It is important to understand which routing method (EIGRP, BGP, etc.,) will be used for multiple access points from the CUs biggest branches. The devil is in the details with this.

3. MPLS network encryption will be an important item for the NCUA. It’s important to be sure about how this will be handled, where the encryption endpoints are, and who is doing the encrypting.

4. QoS (Quality of Service). See details in part two.

5. Managed and Unmanaged MPLS networks.

6. A credit union might be easily fooled into believing there is a multi-billion dollar organization protecting them when a carrier is managing their network. I have not found this to be the case. A managed network can work quite well when one has simple requirements, but credit unions do not have simple requirements. In fact they are quite complicated, and the decision to choose a managed or unmanaged network needs to be examined carefully based on the flexibility given to the credit union.

An unmanaged network is highly flexible, much more so than a managed network. CUs require maximum flexibility, and by flexibility I mean “owning, acquiring and managing” all of your own equipment and having the ability to self-manage the network. Of course, the usual fear is “do I have enough staff to do this?” In reality, if a network is set up and integrated properly, a CU can operate with higher degrees of user satisfaction and flexibility than through a carrier. If staff competency and training is an issue, there are other viable management options to consider that can bridge this gap.

In summary, MLPS network items that require the most attention include the following:

  • 1. Branch connectivity, including high availability and disaster recoverability.
  • 2. ATM connectivity.
  • 3. Disaster Recovery preparedness including redirection to a DR site of branches, ATMs, shared branching, and 3rd-party connectivity. Avian flue remote workers should also be considered.
  • 4. Encryption.

5. QoS

6. SLA (service level agreements) including knowing how, in the above categories, problems with the carrier network are addressed. It is important to know what role your broker-of-choice has, and what recourse the CU has when carrier mistakes occur.

7. From a pre-project planning perspective you should make sure that all DMARCS extensions are completed prior to the last mile carrier going on-site to turn up the lines.

Red Flag Identity Theft Alert – Not Such a Big Deal to Solve

July 1, 2008

The Deadline for compliance of 12 CFR part 717 is November 1, 2008.

A very good client of mine brought this to my attention recently. I see very little forward momentum within my clients in terms of implementing solutions that satisfy compliance requirements. The following is a passage taken from

Section .90(b)(9) Red Flag. The proposed regulations defined “Red Flag” as a pattern, practice, or specific activity that indicates the possible risk of identity theft. The preamble to the proposed rules explained that indicators of a “possible risk” of identity theft would include precursors to identity theft such as phishing,\21\ and security breaches involving the theft of personal information, which often are a means to acquire the information of another person for use in committing identity theft. The preamble explained that the Agencies included such precursors to identity theft as “Red Flags” to better position financial institutions and creditors to stop identity theft at its inception.

As I mentioned in a previous posting, e-mail encryption solutions will suffice. The solution needs to incorporate the following components:

  • Be located at the network’s choke point
  • Be able to integrate with identity sources within the credit union via a tap 
  • Have a financial lexicon engine that automatically blocks and/or encrypts sensitive personally identifiable information.
  • Have the ability to discriminate between a social security number in context and a random set of numbers.


Believe me, all solutions are not created equally. I have seen several solutions, and I love one of them, but I have also seen others fail. 

If there is enough interest I will delve into product specifics but for now I will let my comments above stand.