Roundtable to Discuss – Reducing Costs and Recession Proofing Your IT Spending

April 14, 2009

It was very exciting last week. Last week, on the 8th and the 9th, the ITCUstrategy group and the CUCTO groups met at Apple Federal Credit Union and NIHFCU to discuss the following agenda topics. The discussions were engaging and many of the attendees left with actionable steps for themselves moving forward.

 

Topic/ Agenda

  • Roundtable Discussion Topic – What are local credit union IT Leaders in the area doing in today’s economic environment to help drive costs down or increase value to members?
  • RedZone Presentation – Recession Proof your Credit Union’s IT Spending Using Fixed Cost MSP Programs for IT Plumbing and Day to Day Compliance Needs
  • ITCUStrategy.com – Discuss the May 2009 launch and functionality of the local Forum, Blog, and user Library

 The following are ideas, comments, and views that flowed from the meeting. Enjoy!

 

The following is a composite list of the items that were discussed as a part of Cost Savings Measures that we currently underway and planned for many of the attendees.

 

  • On a case by case basis credit lines are being revoked by attendees for those members demonstrating higher than acceptable risk.
  • Some have used this opportunity to eliminate risky members as an opportunity to reach out to members versus the obvious initial approach of just cancelling the card and terminating the member. In one case a family member offered to use their cash as collateral to get the other family member caught up thus averting a negative situation for the CU and the family member with the obligation.
  • Hiring freezes are in place and if an employee leaves they are not being replaced.
  • There is a move toward more of a web based approach to member service versus brick and mortar.
  • Modifying member behavior is the goal to lighten and reducing lobby traffic which helps with labor savings.
  • There is an increasing interest in ‘tellerless’ and remote kiosk approaches to member services.
  • There was an interesting Call Center and workflow strategy being pursued by Arlington FCU that was interesting from the perspective of combining Call Center and Cross Selling opportunities.
  • Reviewing e-statement approaches and incentives for members via electronic delivery has been a big costs savings for some. Apple FCU in particular had a great story here.             
  • There is certainly a more vigorous approach to vendor negotiation to obtain better pricing
  • Mergers as a general rule were seen as not being successful, however there are models emerging in which smaller credit unions can combine forces and purchasing power while keeping there own unique identity in the process (emerging examples of this are Partnership FCU).
  • Server virtualization was discussed and several members have taken aggressive steps to embrace server virtualization and create a portable infrastructure that drives down costs in the following areas: Power management reduction, power consumption reduction, reduction of server needs, smaller rack space footprint. (NIHFCU and Tower FCU) shared stories here. Also NIHFCU is pushing the envelope in effort to deploy more of a ‘branch office in a box strategy’ using Virtual Desktops VDI. John Szeglin, the IT Director is actively testing the approach and integrating peripherals into the solution so that he can drive down costs at the periphery of his network.
  • Some even expressed that they are not following regular upgrade schedules on PCs that were normally an every 3 year 33% of PCs followed a refresh cycle. Instead they were experimenting with throwing more RAM into machines to breathe more life into them. Since most of the core systems have a “Fred Flinstone” approach to building internet friendly web based applications, this may be a simple way to put a band aid on problems caused by incompetent coding development. It never ceases to amaze me in days of the ‘thin client’ how core processors still want a heavy footprint on the desktop.
  • SaaS – Software as a Service approaches were discussed. This is definitely an approach that needs to be explored further in further group meetings since alternative ways of buying and deploying software would reduce costs and deployment lifecycles.
  • Managed Services (MSP) approaches were discussed as well. The question of how can a credit union can purchase specialized talent to complement existing staff levels was the focus of the Managed Services discussion. As a strategy to do more with less it was a definite approach to consider.
  • Although many attendees are using Monitoring Services to provide monitoring services, there was a lively conversation discussing the differences between Management Services MSP versus Monitoring Services.  

 Thanks for everyone’s time and energy of these two meetings combining ITCUstrategy and the CUCTO groups of the MidAtlantic.


Drive IT Costs Down with Server and Desktop Virtualization

February 13, 2009

The more I visit with my clients I see the effect that server virtualization is having with credit unions. 4 years ago when I started recommending virtualization strategies all I heard was crickets. Now the tide has changed and literally the only question to ask oneself is, “Why am I not virtualizing?” In today’s environment the reality is, that questions are being asked, “can I do more with less?” VMware and Microsoft Hyper V give credit unions this flexibility.  I have a few items that I list below which I believe need to be brought to the table prior to implementing virtualization. I am going to be writing more on this subject over the next few weeks. Virtualization is analogous to a carpenter buying a ¼ inch drill bit. A carpenter doesn’t want a ¼ inch drill bit; what he wants is a ¼ inch hole. No one wants virtualization. What people want the benefits of virtualization. Here are a few of them and some planning questions to ask.

  • What actual hard costs are you looking to save? For example, I have a client who was adamant about not doing virtualization until midway througha data center upgrade project shesaid to me, “how canIsave onsome rack space? I don’t want to buy another rack ifI don’tneed to.” I replied by pointing to 4 servers and saying, “those 4 servers can all be virtualized into one.” These were very old Microsoft 2000 and 2003 servers that she has no intention of upgrading on the short term. Walla – she saves on rack space, cooling costs, server hardware costs, etc.
  • Snapshot VMs – consider if this is one of your goals.
  • If HA (high availability) and snapshot’ing of VMs is important you will need to look at iSCSI SANs as a backend to the Virtual Servers in order to enable this functionality.
  • Consider HA with virtualization on your LAN. I have several credit unions that love this. They have long term goals of growing the environment to support FR needs,etc., but on the short term all they want to be able to do is server consolidation and have a more stable and reliable server infrastructure.
  • They want to be able to do server upgrades and patches and have instant recovery to application failures

I have worked with Joe Fletcher with Johns Hopkins Federal Credit Union for several years and I think one of the ways to review the value of a technology and a strategic path chosen is to see it in action. Joe is a strong IT credit union leader in the mid atlantic area and has experienced the following benefits from VMware virtualization and iSCSI SANs. Here is a listing of his successes:

  • Budget, he has been able to cut costs significantly because of the recent increase in web servers. His core processor is requiring the CU’s to provide more hardware for web based applications than ever before. My client has been able to virtualize all of these applications. He also upgraded many of his older servers to new version of Windows without the expense of buying physical hardware. Finally, he was able to build this environment in one budget cycle by adding the hardware costs for the core required web servers and reaching the same total as purchasing two LeftHand SAN’s, VMware enterprise licensing and HP Procurve Gig Switches.
  • He is thrilled with performance of the machines and the ease of setup, both on the VM and LeftHand side of the fence. VM to VM performance has been great; and he is upgrading core switching to increase the other performance. He has also been working with his core providers to virtualize other applications to improve performance.
  • Consolidation. With old core systems having to remain up for 6 years and a lack of space in general the ability to get rid of 15 servers has opened up the datacenter, making airflow better, making the DC cleaner and giving them room manage the servers without stepping over machines.
  • Disaster Recovery. This meets his overall goal of portability and recoverability. Eventually the branch office will have a replicated SAN and ESX hosts for recovery in the event of a disaster or more likely a small issue like power failure or hardware issues.

Drive IT Costs Down with Desktop Virtualization VDI

February 7, 2009

I almost forgot to mention that I have been excited about this concept of desktop virtualization for a long time. I have only 2 customers out of 40 credit unions who are fully deployed on thin clients using Citrix. The client side requirements for Citrix have usually made the technology a show stopper due to the raw horsepower needed from internal credit union IT staff and consultants to get all the mapped drives, printers, scanners, etc to work properly. Don’t get me wrong, I love Citrix. I have been a huge supporter of Citrix since back in the Winframe days. However, credit unions integration with Citrix has been definately flagged with a warning sign that says warning ‘jump dont dive’. 

The Vision has always been ooo sooo sweeet…..to have all clients cenrally managed like a Unix host. The concept of running a Credit Union on a Windows mainframe has always been appealing. Since most credit union have limited IT budgets and staffing possibilities; then the possibility of centrally managing workstation session has never ending appeal.

Recently I have personally witnessed the power of VDI – Microsoft Virtual Desktop Infrastructure with a CUSO client. I have had high hopes for VDI since I know that the ability to control client side devices is a world apart from Citrix and Terminal Services. This client of mine has integrated Symitar and VDI over a VPN connection to a remote office working multiple session and printing worked like a charm.

In addition to desktop virtualization mentioned above I have another client who is running Citrix servers in a full virtual server environment! This is exciting as well because most people will snuff at this idea. It is real and they are running 150 users in this environment on virtual Citrix servers with a LeftHand iSCSCI SAN.


What’s Wrong with This Picture? (and How to Put It Right)

November 15, 2008

 

“I’m the CFO, it’s not my job to worry about IT.”

 

I have noticed an interesting trend over the past several months that I find exciting. This is the heavy involvement of Finance (Controller and CFO) in IT, not just in decision-making and approvals of IT investment, but in the strategic planning process. I am very encouraged by this.

 

If your senior financial management is not involved in the IT function of your company, I strongly suggest that you consider fixing this situation. Here is a cautionary story that illustrates the problems that a company can face when it doesn’t involve non-IT decision makers in the IT planning process. It illustrates why the CFO must care.

 

We had a non-credit union client recently who was experiencing tremendous pain around complaints from a user community of about 350 users distributed over 14 sites. They had just had a turnover of IT management at the highest level, and this is where I got involved.

 

The user community complaints were actually a symptom of a much deeper and more serious issue.  In the course of our engagement with senior management, we uncovered eight years of executive management neglect of the IT function. It wasn’t malicious neglect; it was unintentional neglect that arose from a lack of a vision, strategy, and long term IT roadmap upon which to base financial and management decisions. There had been no involvement of non-IT executives; as such, IT was not aligned with business vision or strategic objectives.

 

How did this happen? How did they get themselves into this predicament? Here are two examples among several:

 

  1. Their WAN was creaky and old (one of the oldest I have ever seen), but there was no attention on uplifting the infrastructure as part of an iterative and ongoing strategy. A major core business application was rolled out to all sites across , and since no attention was paid to shoring up the infrastructure before application installation, infrastructure performance took a steep (and problematic) drop.
  2. The company was encouraged by their VoIP vendor to purchase a brand new VoIP system. Three integrators later, they were left with the most complicated VoIP routing and switching installation I have ever seen. To make matters worse, they have never received the expected value from the investment.

 

The good news is that we are working with management to fix things. The company must now allocate significant spending to IT in order to make up for the years of little to no investment in infrastructure, disaster recovery, compliance, and other key program components. Though this is a somewhat bitter pill to swallow, it has had the good result of gaining the CFO’s attention and interest.

 

The new IT goal set collaboratively by the IT manager, the CFO, and the Controller is stable, simple, and maintainable systems that produce happy users. They wanted a high quality ‘austere’ network—not “cheap,” but “no frills.”

 

This company also made the decision to go with a Managed Services Provider (MSP) as part of a strategic move to focus their limited but talented IT resources on core business activities. They determined that as far as third-party relationships, they didn’t want a tactical IT partner—that is, a provider that only manages a device or set of devices. They wanted a partner that would participate in strategic planning, design, and architecture, as well as a partner who could assist them in day-to-day management of sophisticated devices from Tier 1-Tier 3 support.

 

Areas that we recommended they turn over to an MSP encompassed much of the security infrastructure, including the DMZ, firewalls, SPAM filters, SSL VPN, Load balancers, QoS devices, AD, Servers, and Consolidated Event Management. (The caveat, of which they are cognizant, is that an MSP can only be brought in after their infrastructure has been assessed and remediated.) Hiring and managing the in-house talent to effectively support all the equipment listed above would run $80-110k per year; the MSP we recommended performs the same services for $48k per year.

 

One of their primary goals, right after end user happiness, is network stability for the VoIP system. We encouraged them to focus on simplicity in order to make the network able and supportable. Since they had determined that they did not want their core IT staff supporting a non-business value add system then this system also had to be simplified so that the MSP taking over the VoIP management wasn’t saddled with the same issues.

 

We continue to work with senior management on effective IT strategy. As far as next steps, the CFO wants an IT roadmap, that is, a doable plan that is sized right for the company. Immediate action items include:

 

  1. Data Center power distribution and re-cabling.
  2. Replaced the 10-year-old ATT WAN with a new Sprint MPLS WAN.
  3. Virtualization (there is no more server rack space left)
  4. Disaster recovery site implementation
  5. Employing a different back up method from the tape backups currently being used.
  6. A comprehensive Microsoft licensing strategy that includes an audit of current licenses.

 

My reason for providing a high level of detail in this story is to give you clear examples of IT issues that may track with your own. If any of the problems or strategies that this client is dealing with ring any bells for you, it may be time to examine your own IT function and how your financial management relates to it. If your senior financial manager is not getting involved with IT strategy or decision making, you may want to better align the two. If you don’t, there may be trouble brewing behind the scenes.



iSCSI SAN Business Case

June 27, 2008

 I just spent my entire week at the Burton Group Catalyst conference in San Diego, California (www.burtongroup.com). Burton Group has the best industry analysts and researchers in security, identity management, virtualization and the SAN industry.

The number of Virtualization Hypervisor vendors vying for position is exciting. Clearly VMware is in the lead and the immediate safe bet for credit unions. However, with the purchase of Xen Source by Citrix there is no doubt that Citrix is going to be a major player with their XenApp server. Virtualization is here to stay for many reasons, but for credit unions the major areas of impact are:

  • Cost savings stemming from data center server consolidation;
  • Compliance – Disaster Recovery accomplished with server mobility and portability that a virtual infrastructure brings;
  • Ease of desktop management using VDI (virtual desktop infrastructure);

I really don’t think that Fiber Channel SANs should be even considered for credit unions unless they have a legacy investment in Fiber Channel technology. iSCSI SAN technology is necessary for credit unions to really take advantage of the compliance and disaster recovery benefits of virtualization. Credit Unions today need the following to build a business case for an iSCSI SAN:

  • Easy administration;
  • Supportable with current IT staff and current IT skills;
  • Scalable for production virtualization and DR needs;
  • Affordability;

What are the benefits of an iSCSI SAN?

  • Runs over the current TCP/IP Ethernet network;
  • Reduces network infrastructure costs by reusing existing infrastructure;
  • Uses generic 1GB NICs and switches;
  • Reduces SAN complexity and increases manageability;
  • Reduces cabling costs;
  • Reduces outside integration support;
  • Increases Disaster Recovery options by using routable TCP/IP technology and there are no distance topology restrictions;
  • Leverages existing network services in QoS, DNS, VPN and MPLS investment;
  • Leverages more of the current in-house credit union IT staff talent;

The big argument from FC fiber channel SAN vendors is that iSCSI is not as fast as FC. This is true, but is irrelevant unless you are a $3 billion or larger credit union. In my opinion credit unions worth $200 million to $3 billion in assets will not strain iSCSI to the point that this is even an issue.

There are ways to make iSCSI faster if needed: software initiators, TOE NIC adaptors and iSCSI HBA. See the Chris Wolf site for more great information (www.chriswolf.com). www.virtualization.info also includes good virtualization and iSCSI information as well. 

Credit unions that have an existing investment in FC SAN will want to protect this investment. Soon you will be able to bridge your iSCSI SAN to your FC SAN using the protocol FCoE (Fiber Channel over Ethernet) but this protocol and the bridge gateway switches needed to do this have not yet matured.

Let me know what you think.


Backup and Corruption Protection: “I want to go tapeless”

May 15, 2008

I constantly hear:

1) “I want to virtualize my environment.”

2) “I want to eliminate tapes and go tapeless.”

3) “I want buy a SAN.”

4) “I want to replicate between my headquarters (HQ) and DR facility.”

The key here is appropriately mixing technologies. By design, and based on the laws of physics, virtualization and SANs need speed and high performance. iSCSI is surpassing Fiber Channel this year in shipments and is the only way for credit unions to go. The cost of supporting FC and the training uplift needed for staff and personnel is not worth it. My point is that your SAN and virtual hosts need to be screaming demons from a performance perspective. 

Back-up is another story. Back-ups can be slow and can be scheduled. When it comes to back-up ask yourself what you want to do. For example do you wish to:

1) Use traditional back-up software?

2) Leverage VCB and a proxy server to back-up your virtual hosts to tape?

3) Implement an ‘in the cloud’ backup solution?

4) Use a disc-to-disc approach for backup (cool method)?

You should also ask yourself the following questions:

1) What is the method for corruption protection? This is key to the design and architecture of your system.

2) What is the present cost to back-up? Costs include Iron Mountain, tapes, and maintenance fees.

3) What is the cost to lose back-ups now? Costs include communications with members, and fines.

4) Do you want to plan for archiving now? Do you want to use a SAN approach of integrate with cheaper back-up discs when archiving? What about your core system?

5) Can you integrate all aspects of the CU? This includes Microsoft, Imaging, and Unix. Can you integrate your core system into the back-up solution? What about the imaging system? What about your Microsoft environment? In summary, can we embrace a comprehensive solution with the back-up system? 

6) Can you truly “go tapeless”? What about the O/S tape from the core system?

I hope that this gives you a general impression of what questions to ask when discussing backup, DR, SAN and virtualization.


Implementing an IT Disaster Recovery Plan That Works (Part 2)

May 3, 2008

Here’s another story of a larger credit union client of mine.

The credit union said to me one day: “Bill, we have all this stuff and my staff if good. How do we pull all this software and hardware together into a comprehensive Disaster Recovery program?” Over the past two years, the credit union had acquired the following:

  • Fatpipe load balancers;
  • VMware;
  • Backup software (Issue: tapeless versus traditional? They were partial to tapeless);
  • Doubletake;
  • Platespin;
  • HP NAS appliance;
  • iSCSI SAN from Lefthand;
  • DR Site;
  • Connectivity;

The executive summarized his predicament to me like this: “the products all appear to be good.” I agreed, but based on his current network problems, tight back-up windows, huge WAN latency and more, it appeared that several of these products had overlapping functionality causing them to argue and step on each other. “There is no way I can roll this out into production without being sure” he said.

The credit union asked me to come up with solutions in several areas, in particular the executive wanted answers to these questions:

  • How can they repurpose the HP NAS so that the investment is not wasted?
  • What is the best way to use the iSCSI SAN from a block level replication perspective?
  • How is back-up and corruption protection going to be handled?
  • What function will Doubletake play in the new design?
  • How will the WAN network respond to the new design? The credit union had a combined MPLS and point-to-point architecture.
  • Why are backups barely being completed overnight? This could be indicative of bigger issues that need to be solved first.

For larger credit unions, one must approach back-up differently, often through production uses of a SAN.

The following summary of questions will help you understand Disaster Recovery as two necessary categories including 1) backup and correction protections and 2) production SAN and virtualization.

These two categories are what I explore with clients in the process of determining the exact level of customization required for their DR environment.

Backup and Corruption Protections

1. Do you want to keep managing backup tapes for Microsoft systems, core systems, imaging systems? You should know the costs for this process (for example, Iron Mountain)

2. What is the risk for this process? You might consider losing tapes, theft, or other eventualities.

3. If you move to a tapeless system what is the local restore capability?

4. What about “in the cloud” solutions? There are some cool ones that work really well.

5. Does your backup solution provide local corruption protection?

6. If I make a call to Microsoft for support will they take my call?

7. Will the tapeless solution recover the O/S of the core processor? The only answer is no, so are you really tapeless?

Production SAN and Virtualization

1. Why do I need virtualization and an iSCSI SAN?

2. Why is virtualization perfect for my day-to-day operations?

3. Why is virtualization a ‘magic bullet’ for my DR initiatives?

4. What are the downsides of virtualization?

5. How do I know if I can virtualize a system?

6. How do I assess the readiness of my virtualization?

7. What about Microsoft systems that are iSCSI aware? SQL 2005 and Exchange 2007 are both options. You should consider how this might change your iSCSI SAN and virtualization objectives.

That’s enough information to get you started on the path to implementing an IT Disaster Recovery Plan that works. (See my previous post for questions related to a smaller credit union.)