Roundtable to Discuss – Reducing Costs and Recession Proofing Your IT Spending

April 14, 2009

It was very exciting last week. Last week, on the 8th and the 9th, the ITCUstrategy group and the CUCTO groups met at Apple Federal Credit Union and NIHFCU to discuss the following agenda topics. The discussions were engaging and many of the attendees left with actionable steps for themselves moving forward.


Topic/ Agenda

  • Roundtable Discussion Topic – What are local credit union IT Leaders in the area doing in today’s economic environment to help drive costs down or increase value to members?
  • RedZone Presentation – Recession Proof your Credit Union’s IT Spending Using Fixed Cost MSP Programs for IT Plumbing and Day to Day Compliance Needs
  • – Discuss the May 2009 launch and functionality of the local Forum, Blog, and user Library

 The following are ideas, comments, and views that flowed from the meeting. Enjoy!


The following is a composite list of the items that were discussed as a part of Cost Savings Measures that we currently underway and planned for many of the attendees.


  • On a case by case basis credit lines are being revoked by attendees for those members demonstrating higher than acceptable risk.
  • Some have used this opportunity to eliminate risky members as an opportunity to reach out to members versus the obvious initial approach of just cancelling the card and terminating the member. In one case a family member offered to use their cash as collateral to get the other family member caught up thus averting a negative situation for the CU and the family member with the obligation.
  • Hiring freezes are in place and if an employee leaves they are not being replaced.
  • There is a move toward more of a web based approach to member service versus brick and mortar.
  • Modifying member behavior is the goal to lighten and reducing lobby traffic which helps with labor savings.
  • There is an increasing interest in ‘tellerless’ and remote kiosk approaches to member services.
  • There was an interesting Call Center and workflow strategy being pursued by Arlington FCU that was interesting from the perspective of combining Call Center and Cross Selling opportunities.
  • Reviewing e-statement approaches and incentives for members via electronic delivery has been a big costs savings for some. Apple FCU in particular had a great story here.             
  • There is certainly a more vigorous approach to vendor negotiation to obtain better pricing
  • Mergers as a general rule were seen as not being successful, however there are models emerging in which smaller credit unions can combine forces and purchasing power while keeping there own unique identity in the process (emerging examples of this are Partnership FCU).
  • Server virtualization was discussed and several members have taken aggressive steps to embrace server virtualization and create a portable infrastructure that drives down costs in the following areas: Power management reduction, power consumption reduction, reduction of server needs, smaller rack space footprint. (NIHFCU and Tower FCU) shared stories here. Also NIHFCU is pushing the envelope in effort to deploy more of a ‘branch office in a box strategy’ using Virtual Desktops VDI. John Szeglin, the IT Director is actively testing the approach and integrating peripherals into the solution so that he can drive down costs at the periphery of his network.
  • Some even expressed that they are not following regular upgrade schedules on PCs that were normally an every 3 year 33% of PCs followed a refresh cycle. Instead they were experimenting with throwing more RAM into machines to breathe more life into them. Since most of the core systems have a “Fred Flinstone” approach to building internet friendly web based applications, this may be a simple way to put a band aid on problems caused by incompetent coding development. It never ceases to amaze me in days of the ‘thin client’ how core processors still want a heavy footprint on the desktop.
  • SaaS – Software as a Service approaches were discussed. This is definitely an approach that needs to be explored further in further group meetings since alternative ways of buying and deploying software would reduce costs and deployment lifecycles.
  • Managed Services (MSP) approaches were discussed as well. The question of how can a credit union can purchase specialized talent to complement existing staff levels was the focus of the Managed Services discussion. As a strategy to do more with less it was a definite approach to consider.
  • Although many attendees are using Monitoring Services to provide monitoring services, there was a lively conversation discussing the differences between Management Services MSP versus Monitoring Services.  

 Thanks for everyone’s time and energy of these two meetings combining ITCUstrategy and the CUCTO groups of the MidAtlantic.


Credit Union Mergers – Mitigate Technical Risk

March 18, 2009

From my point of view a credit union merger is a ‘non-trivial’ event, however I am excited about the opportunity that this provides both entities from a technology perspective. A small credit union can come out of a merger stronger, leaner, and more efficient than before. This is an opportunity to streamline, achieve economies of scale, and combine the best of each entity while discarding the unworkable elements. The following are a list of good questions to ponder with your teams. Here is a short list of what I would consider to be the ‘tough stuff’ from a voting and discussion perspective between teams.

Technical Systems Integration Planning Steps

  • What is the plan for the coexistence of two (separate) LAN and WAN networks and what is the end state goal?
    •  IP Scheme – bridged/ routed network
    • Are the Credit Unions using disparate core systems? Different Versions? Do they have conflicting IP Scheme requirement?
  • What are the deadlines that need to be hit so that they details can be coordinated?
    •   IP Scheme
    •  Printing (Sharing, Services, Drivers)
    • Bandwidth
    •  Routers
  • What is the plan for the convergence of credit union peripheral hardware convergence?
    • Signature pads
    • Receipt Printers
    • Scanners
    • Check printers 
  • How will imaging be merged, including the old that may need to be kept for 7 years? What is the plan for current and historical images? What is the final imaging goal?
  • How will old core system records be kept (Core historicals, etc.)?

Internal Questions

  • What is the end network design?
    • WAN Architecture
    • Integration
    • POP Diversity
    • Redundancy
    • Encryption
    • QoS – quality of services to protect VoIP integrity
  • Can the credit unions use each other for DR?
  • What services will be shared?
    • Active Directory
    • Email
    • Files
    • SQL Databases
    • Domain Controller Authority
    • Imaging
    • Domain Trust
      • Is Microsoft SBS involved? If yes, there are important trust planning considerations.
  • Will Microsoft licensing be audited to take advantage of consolidation? Use a merger to negotiate and consolidate licensing.
  • What is the plan for enterprise back-ups long term?
  • How will the phone system be consolidated and converged?

So here is the summary of my merger material. I have collaborated with a couple of team mates to put this 2 part series together for everyone. I hope you like it and that it was useful to you.  


Considering a Merger? Is the Time Finally Right?

March 5, 2009

With the economy shifting south, coupled with the NCUA assessment fee to bail out the Corporate Credit Unions, small credit unions can combine forces to compete better and provide more value to their membership. I am observing a trend toward small credit unions merging on a much more rapid scale than I have seen in the past. The merging of credit unions is not noteworthy in and of itself, however I do believe that mergers that combine to reach the $100 million plus range are going to increase.


When considering a merger, it is critical to establish relationships with experts you can turn to if you go forward. These experts should span all operations in the credit union, and be able to weigh in on questions such as:


· What are the best practices in merging a credit union?

· How do you merge IT departments without adding risk?

·  How to plan for and cut waste during a merger?

·  How can risk be mitigated?

·  What is the best way to leverage the opportunity to build in efficiencies?

· What functions can be strategically outsourced?

· What processes can be integrated?

· How should IT integration be handled?


Credit Union Merger Questionnaire – Information Technology

The following questionnaire pertains to the last point, and represents the starting point for planning and implementing effective IT integration for a credit union merger. These questions are intended to bring up important issues that must be planned for in the IT space, and to start discussions that will lead to effective decision making.


High Level Objectives/ Co-Existence Plan

  • Is the objective for the merger:  To attain one united front or identity with the leveraged strength of a partnership……
  • or is the goal of the merger to  maintain dual identities with the leveraged strength of a partnership?  
  • What is the plan for the existing domain names and the new domain name? Is there a timeline set for the sites to disappear and one to replace them, or will the old sites remain in place?
  • What is the plan for the email utility in the new entity? What is the timeline for implementation? Will there be coexistence of emails between domains?
  • How will home banking be presented to the members? What is the timeline for the change?
  • What SSL Certificates can be merged, deleted and/or re-used (web sites, ssl vpns, etc.)?
  • Is there a common encryption policy for sending information to third parties ( e.g. credit card processing via PGP, or does one of the entities have ZIx email encryption)?
  • What is the encryption goal? Are there any vendors that require specific encryption technology?
  • What is the end goal for the phone system and call center/ member services? Is there a timeline set for the convergence of the systems?

o        PRI analysis – what is the call routing plan?

o        Are you launching with core phone system functionality first and then integrating Call Center functionality after the merger?


  • What is the goal for integration and collapse of the networks (WAN – MPLS)? Applications  (like imaging, etc.)? Data bases? Other elements?

o        Has a cost analysis been completed for the infrastructure WAN collapse of the two entitities? Data, Voice (long distance/local)

o        What questions does one need to ask when integrating carriers – Sprint, ATT, Qwest, Verizon, and Paetech for example? (This blog link is an overview of questions to ask.


  • How are third parties (PSCU, FedLine, DI, etc. ) being addressed? Which third parties will remain? Are there redundancies? Which ones are going away? 


On my next post I will examine most technical questions that I have to ask myself when helping a credit union during a merger. 



What’s Wrong with This Picture? (and How to Put It Right)

November 15, 2008


“I’m the CFO, it’s not my job to worry about IT.”


I have noticed an interesting trend over the past several months that I find exciting. This is the heavy involvement of Finance (Controller and CFO) in IT, not just in decision-making and approvals of IT investment, but in the strategic planning process. I am very encouraged by this.


If your senior financial management is not involved in the IT function of your company, I strongly suggest that you consider fixing this situation. Here is a cautionary story that illustrates the problems that a company can face when it doesn’t involve non-IT decision makers in the IT planning process. It illustrates why the CFO must care.


We had a non-credit union client recently who was experiencing tremendous pain around complaints from a user community of about 350 users distributed over 14 sites. They had just had a turnover of IT management at the highest level, and this is where I got involved.


The user community complaints were actually a symptom of a much deeper and more serious issue.  In the course of our engagement with senior management, we uncovered eight years of executive management neglect of the IT function. It wasn’t malicious neglect; it was unintentional neglect that arose from a lack of a vision, strategy, and long term IT roadmap upon which to base financial and management decisions. There had been no involvement of non-IT executives; as such, IT was not aligned with business vision or strategic objectives.


How did this happen? How did they get themselves into this predicament? Here are two examples among several:


  1. Their WAN was creaky and old (one of the oldest I have ever seen), but there was no attention on uplifting the infrastructure as part of an iterative and ongoing strategy. A major core business application was rolled out to all sites across , and since no attention was paid to shoring up the infrastructure before application installation, infrastructure performance took a steep (and problematic) drop.
  2. The company was encouraged by their VoIP vendor to purchase a brand new VoIP system. Three integrators later, they were left with the most complicated VoIP routing and switching installation I have ever seen. To make matters worse, they have never received the expected value from the investment.


The good news is that we are working with management to fix things. The company must now allocate significant spending to IT in order to make up for the years of little to no investment in infrastructure, disaster recovery, compliance, and other key program components. Though this is a somewhat bitter pill to swallow, it has had the good result of gaining the CFO’s attention and interest.


The new IT goal set collaboratively by the IT manager, the CFO, and the Controller is stable, simple, and maintainable systems that produce happy users. They wanted a high quality ‘austere’ network—not “cheap,” but “no frills.”


This company also made the decision to go with a Managed Services Provider (MSP) as part of a strategic move to focus their limited but talented IT resources on core business activities. They determined that as far as third-party relationships, they didn’t want a tactical IT partner—that is, a provider that only manages a device or set of devices. They wanted a partner that would participate in strategic planning, design, and architecture, as well as a partner who could assist them in day-to-day management of sophisticated devices from Tier 1-Tier 3 support.


Areas that we recommended they turn over to an MSP encompassed much of the security infrastructure, including the DMZ, firewalls, SPAM filters, SSL VPN, Load balancers, QoS devices, AD, Servers, and Consolidated Event Management. (The caveat, of which they are cognizant, is that an MSP can only be brought in after their infrastructure has been assessed and remediated.) Hiring and managing the in-house talent to effectively support all the equipment listed above would run $80-110k per year; the MSP we recommended performs the same services for $48k per year.


One of their primary goals, right after end user happiness, is network stability for the VoIP system. We encouraged them to focus on simplicity in order to make the network able and supportable. Since they had determined that they did not want their core IT staff supporting a non-business value add system then this system also had to be simplified so that the MSP taking over the VoIP management wasn’t saddled with the same issues.


We continue to work with senior management on effective IT strategy. As far as next steps, the CFO wants an IT roadmap, that is, a doable plan that is sized right for the company. Immediate action items include:


  1. Data Center power distribution and re-cabling.
  2. Replaced the 10-year-old ATT WAN with a new Sprint MPLS WAN.
  3. Virtualization (there is no more server rack space left)
  4. Disaster recovery site implementation
  5. Employing a different back up method from the tape backups currently being used.
  6. A comprehensive Microsoft licensing strategy that includes an audit of current licenses.


My reason for providing a high level of detail in this story is to give you clear examples of IT issues that may track with your own. If any of the problems or strategies that this client is dealing with ring any bells for you, it may be time to examine your own IT function and how your financial management relates to it. If your senior financial manager is not getting involved with IT strategy or decision making, you may want to better align the two. If you don’t, there may be trouble brewing behind the scenes.

CASE STUDY: How to Recession Proof Your Credit Union with Strategic IT Spending

April 21, 2008

Here are a few “what-if” scenarios for your IT vision:

*What if your IT infrastructure were “plug-n-play”?

*What if your IT infrastructure were “portable”?

*What if your countless data bases were “portable” and “consolidated”?

*How would your DR and infrastructure spending change if you knew that $1.00 in spending could gain you critical compliance momentum regarding DR, infrastructure, and security?

*What if these databases were replicated to your DR facility on the same platform as your Exchange/email and file systems?

*What if you were able to buy less hardware?

*What if you had a two-year roadmap of technical and reality-based spending to support this vision?

In today’s lean business climate, credit union executives must ensure that every dollar of day-to-day operational and compliance IT spending is doing triple duty within DR, infrastructure, and security. So what is your vision?

How would you get started? A story about a client of mine comes to mind that you might think of as a real-world case study direct from the trenches. I had worked closely with this client in late 2007 and early 2008 to formalize their strategy with Infrastructure, DR, WAN, Security, Citrix, VoIP and databases.

It is important to note that this credit union created the technical roadmap, design, and architecture that they wanted to pursue. They submitted their objectives to the SAN vendor community based on what they wanted to accomplish rather than doing the rounds and gathering opinions.

Seeking a Desired Solution

After deciding on a lowest common denominator for what their needs were, the credit union determined that they had a two-part goal. The first phase was to implement an iSCSI Storage Area Network – SAN at their Headquarters and DR facility according to the criteria listed below. The second phase was to create a “portable” and ‘plug-n-play’ infrastructure using VMware virtualization.

Overview of Current Environment Problem

The credit union was experiencing storage growth issues in the following areas:

1. Email;
2. Knowledge Base;
3. VoIP System recording data;
4. Imaging system data base;
5. IT Software (media storage);
6. Instant messaging;
7. Voice mail;
8. Fax;
9. File Servers;
10. Security (IDS logs, FW logs, internet logs);
11. Databases (Member Account Statements, Visa Statements, check copies, financial reports, statements);
12. Scalability regarding how to deal with the growing storage;
13. Unconsolidated storage (SQL data base proliferation);
14. Compliance with future regulations (future GLBA), retention, and archiving;
15. Cost to maintain multiple backup solutions;
16. Tape encryption;

The Requirements

The credit union wanted to purchase an iSCSI SAN in order to accomplish the following:

Primary Goal 1

Eliminate Iron Mountain tape costs
Eliminate isolated tape back-up solutions for:

i. Microsoft systems;
ii. The Synergy system;
iii. Core system;

Eliminate recurring maintenance costs for backup software
Ease staff storage management
Accommodate growth
Enable data encryption
Data storage consolidation for:

i. Imaging data base;
ii. Microsoft databases;

Consolidate file storage
Operations replication of live data between Headquarters and the DR facility
Backup replication between headquarters and the DR facility
Comply with current and future IS&T NCUA regulations
The system should be able to support future email archiving functionality

Primary Goal 2

Create a “portable” infrastructure accomplished using virtualization with VMware technology.

In a future entry, I will examine the questions that this very visionary credit union CIO asked himself — with a little help from yours truly — as he went through the budgeting and planning process for technology platform changes. I will review the overall roadmap and execution plan that was developed as we worked together to present an overall IT strategy for his credit union. Questions like the following were asked:

What do I do with my Citrix environment when using VMware?
When do I move to Exchange 2007? How does this impact my SAN and virtualization?
What impact does this SAN have on my switching infrastructure?
What good is my new plan without an MPLS network?
How will an MPLS network react to replication traffic?
How will my VoIP traffic be impacted?

In this entry I have reviewed strategy as it relates to the SAN and virtualization. The critical concern is to maximize the value of your investments. A planned approach results in an optimized situation for IT as well as for the entire business.

Blueprint for a Multi-Year Security, Identity, and Privacy (SIP) Strategy for your Credit Union

April 2, 2008
When I visit credit unions one of the biggest challenges is that the current IT diagrams don’t match reality. Why would this happen? I’ve observed that most credit unions have inherited their IT foundations from years and years ago, and through the years many hands have touched, moved, re-routed, and re-booted products and systems so many times that there is little confidence left in the network.
                                                                                                                     So here, as promised, is a blueprint for developing a multi-year Security, Identity, and Privacy (SIP) strategy for your credit union.
SIPS Blueprint
SIPS Strategy
The first question is the obvious one: can you adhere to the IT security policy of the NCUA without breaking your budget? What if you don’t have an IT strategist on staff? Can you ensure IT security alignment is strategic and planned rather than reactive? Remember that security is about design and architecture not product selection. A CEO and the board of directors should be able to see that decisions are not merely based on the latest media threat but are being made based on a strategic security framework.

It is my belief that you can use architecture alone to accomplish 75% of NCUA regulatory requirements. How is this possible?

A lot can be accomplished through the proper positioning and correct usage of current networking and security devices. Oftentimes proper redeployment of pre-existing software and hardware can accomplish the job quite easily.

A credit union must consider all aspects of security architecture before acquiring products. Simple questions asked up-front can drive product decisions for years (see August’s postings on “Questions to Ask”). Similar to building a home, an executive should expect an organization to apply principles of architecture and design, especially in regards to security architecture, when developing a multi-year security program.

My intent is that you will gain the following wisdom from the blueprint shown above:

1. Require your IT staff to communicate business alignment purchasing decisions.                                                                                                               2. Develop a multi-year plan.
3. Follow the multi-year plan.
4. Set budget milestones.
5. Communicate the vision for what you want to accomplish.
6. Get out of fire-fighting mode and into strategic management.
7. Purchase products as part of your strategy, not as a reaction to a problem.

In my next post, I’ll cover the topic of client integrity – the first item in the blueprint diagram above.

Security Strategy that a CFO can Understand

March 28, 2008

I recently had lunch with the CFO of a medium-sized credit union in the Mid-Atlantic region. 

The CFO had joined the credit union 6 months ago, and the account manager for my company was giving him an update on the progress of several IT projects that were being handled by my company. We had been working with this credit union for about 3 years, so we were educating him about decisions made before his arrival.

Since he had just lost his IT manager, he wanted to know why he had four devices acting as firewalls on his network. I responded that we had noted this fact as a risk item two years ago, but the former IT manager had disregarded our warnings. In explaining the history of the credit union, we explained that the IT manager was not concerned with security, but with how he was going to articulate the problems to his bosses (the CEO, IT review committee, and the board of directors).

In 2004 this credit union had passed a security review, though it was in actuality only a remote “penetration test”. The IT manager was not willing to face the challenges we described in our findings.

Fast forward to 2008 at our lunch meeting, and the IT manager is gone and the CFO is running the show. My company’s consultants tell the CFO that the four firewalls are doing absolutely nothing, and are in fact acting as a “screen door” for security.  The CFO shakes his head in amazement.

I explain that security can be quite straightforward. In fact if a business person can’t understand the security strategy and the tactics employed then it is too complex — complexity is a death sentence for credit union security. The more needless complexity you build into your infrastructure, the higher your costs.

I took out a pen and drew pictures of a firewall and 3rd parties (e-funds, shared branch, home banking, FedLine) on a restaurant napkin to show him how a firewall manages external business relationships. I drew a representation of his current situation with four firewalls and then sketched out an optimized (and affordable) future state. [See diagrams below]

A de-militarized zone (DMZ) must be a key part of a credit union’s security strategy. It is not the number one aspect of security, but it is close. The DMZ of a network is simply the drawbridge, moat, and exterior sentries of a castle’s defenses.

At its core, a DMZ must manage 3rd party access to a credit union network. Everyone thinks they are doing this, but I have yet to find a successfully managed and installed DMZ. My experience has been that DMZs are a real challenge for credit unions whether they are $30 million or $1 billion outfits.

I provided the CFO with a multi-year Security, Identity, and Privacy (SIP) strategy blueprint, and he promised to keep his IT strategy front-and-center for the next two years.

I’ll talk about the SIP strategy blueprint in future posts on this blog.