Drive IT Costs Down with Server and Desktop Virtualization

February 13, 2009

The more I visit with my clients I see the effect that server virtualization is having with credit unions. 4 years ago when I started recommending virtualization strategies all I heard was crickets. Now the tide has changed and literally the only question to ask oneself is, “Why am I not virtualizing?” In today’s environment the reality is, that questions are being asked, “can I do more with less?” VMware and Microsoft Hyper V give credit unions this flexibility.  I have a few items that I list below which I believe need to be brought to the table prior to implementing virtualization. I am going to be writing more on this subject over the next few weeks. Virtualization is analogous to a carpenter buying a ¼ inch drill bit. A carpenter doesn’t want a ¼ inch drill bit; what he wants is a ¼ inch hole. No one wants virtualization. What people want the benefits of virtualization. Here are a few of them and some planning questions to ask.

  • What actual hard costs are you looking to save? For example, I have a client who was adamant about not doing virtualization until midway througha data center upgrade project shesaid to me, “how canIsave onsome rack space? I don’t want to buy another rack ifI don’tneed to.” I replied by pointing to 4 servers and saying, “those 4 servers can all be virtualized into one.” These were very old Microsoft 2000 and 2003 servers that she has no intention of upgrading on the short term. Walla – she saves on rack space, cooling costs, server hardware costs, etc.
  • Snapshot VMs – consider if this is one of your goals.
  • If HA (high availability) and snapshot’ing of VMs is important you will need to look at iSCSI SANs as a backend to the Virtual Servers in order to enable this functionality.
  • Consider HA with virtualization on your LAN. I have several credit unions that love this. They have long term goals of growing the environment to support FR needs,etc., but on the short term all they want to be able to do is server consolidation and have a more stable and reliable server infrastructure.
  • They want to be able to do server upgrades and patches and have instant recovery to application failures

I have worked with Joe Fletcher with Johns Hopkins Federal Credit Union for several years and I think one of the ways to review the value of a technology and a strategic path chosen is to see it in action. Joe is a strong IT credit union leader in the mid atlantic area and has experienced the following benefits from VMware virtualization and iSCSI SANs. Here is a listing of his successes:

  • Budget, he has been able to cut costs significantly because of the recent increase in web servers. His core processor is requiring the CU’s to provide more hardware for web based applications than ever before. My client has been able to virtualize all of these applications. He also upgraded many of his older servers to new version of Windows without the expense of buying physical hardware. Finally, he was able to build this environment in one budget cycle by adding the hardware costs for the core required web servers and reaching the same total as purchasing two LeftHand SAN’s, VMware enterprise licensing and HP Procurve Gig Switches.
  • He is thrilled with performance of the machines and the ease of setup, both on the VM and LeftHand side of the fence. VM to VM performance has been great; and he is upgrading core switching to increase the other performance. He has also been working with his core providers to virtualize other applications to improve performance.
  • Consolidation. With old core systems having to remain up for 6 years and a lack of space in general the ability to get rid of 15 servers has opened up the datacenter, making airflow better, making the DC cleaner and giving them room manage the servers without stepping over machines.
  • Disaster Recovery. This meets his overall goal of portability and recoverability. Eventually the branch office will have a replicated SAN and ESX hosts for recovery in the event of a disaster or more likely a small issue like power failure or hardware issues.

Drive IT Costs Down with Desktop Virtualization VDI

February 7, 2009

I almost forgot to mention that I have been excited about this concept of desktop virtualization for a long time. I have only 2 customers out of 40 credit unions who are fully deployed on thin clients using Citrix. The client side requirements for Citrix have usually made the technology a show stopper due to the raw horsepower needed from internal credit union IT staff and consultants to get all the mapped drives, printers, scanners, etc to work properly. Don’t get me wrong, I love Citrix. I have been a huge supporter of Citrix since back in the Winframe days. However, credit unions integration with Citrix has been definately flagged with a warning sign that says warning ‘jump dont dive’. 

The Vision has always been ooo sooo sweeet… have all clients cenrally managed like a Unix host. The concept of running a Credit Union on a Windows mainframe has always been appealing. Since most credit union have limited IT budgets and staffing possibilities; then the possibility of centrally managing workstation session has never ending appeal.

Recently I have personally witnessed the power of VDI – Microsoft Virtual Desktop Infrastructure with a CUSO client. I have had high hopes for VDI since I know that the ability to control client side devices is a world apart from Citrix and Terminal Services. This client of mine has integrated Symitar and VDI over a VPN connection to a remote office working multiple session and printing worked like a charm.

In addition to desktop virtualization mentioned above I have another client who is running Citrix servers in a full virtual server environment! This is exciting as well because most people will snuff at this idea. It is real and they are running 150 users in this environment on virtual Citrix servers with a LeftHand iSCSCI SAN.

iSCSI SAN Business Case

June 27, 2008

 I just spent my entire week at the Burton Group Catalyst conference in San Diego, California ( Burton Group has the best industry analysts and researchers in security, identity management, virtualization and the SAN industry.

The number of Virtualization Hypervisor vendors vying for position is exciting. Clearly VMware is in the lead and the immediate safe bet for credit unions. However, with the purchase of Xen Source by Citrix there is no doubt that Citrix is going to be a major player with their XenApp server. Virtualization is here to stay for many reasons, but for credit unions the major areas of impact are:

  • Cost savings stemming from data center server consolidation;
  • Compliance – Disaster Recovery accomplished with server mobility and portability that a virtual infrastructure brings;
  • Ease of desktop management using VDI (virtual desktop infrastructure);

I really don’t think that Fiber Channel SANs should be even considered for credit unions unless they have a legacy investment in Fiber Channel technology. iSCSI SAN technology is necessary for credit unions to really take advantage of the compliance and disaster recovery benefits of virtualization. Credit Unions today need the following to build a business case for an iSCSI SAN:

  • Easy administration;
  • Supportable with current IT staff and current IT skills;
  • Scalable for production virtualization and DR needs;
  • Affordability;

What are the benefits of an iSCSI SAN?

  • Runs over the current TCP/IP Ethernet network;
  • Reduces network infrastructure costs by reusing existing infrastructure;
  • Uses generic 1GB NICs and switches;
  • Reduces SAN complexity and increases manageability;
  • Reduces cabling costs;
  • Reduces outside integration support;
  • Increases Disaster Recovery options by using routable TCP/IP technology and there are no distance topology restrictions;
  • Leverages existing network services in QoS, DNS, VPN and MPLS investment;
  • Leverages more of the current in-house credit union IT staff talent;

The big argument from FC fiber channel SAN vendors is that iSCSI is not as fast as FC. This is true, but is irrelevant unless you are a $3 billion or larger credit union. In my opinion credit unions worth $200 million to $3 billion in assets will not strain iSCSI to the point that this is even an issue.

There are ways to make iSCSI faster if needed: software initiators, TOE NIC adaptors and iSCSI HBA. See the Chris Wolf site for more great information ( also includes good virtualization and iSCSI information as well. 

Credit unions that have an existing investment in FC SAN will want to protect this investment. Soon you will be able to bridge your iSCSI SAN to your FC SAN using the protocol FCoE (Fiber Channel over Ethernet) but this protocol and the bridge gateway switches needed to do this have not yet matured.

Let me know what you think.

iSCSI SANs – Plan, Plan, Plan – Why?

June 27, 2008

There is quite a lot of attention given to the manufacturers of iSCSI SANs. We’ll often here about how great they are with virtualization, and that they are low cost and easy to manage compared to FC Fiber Channel storage and other approaches. There is precious little information, however, regarding the usage strategy for a new iSCSI SAN within a credit union environment. For credit unions that are in the planning and strategy development process, proper planning can make or break a SAN initiative and kill ROI assumptions that are made at the outset of the effort.

What are the benefits of proper planning of an iSCSI infrastructure as it relates to applications? One of the large benefits is performance. Like any other environment, credit unions are subject to server sprawl. In the long run, virtualization will take care of this problem, but Microsoft SQL has to be independently planned. Why is this? The “best practices” for SQL published by Microsoft are well-known and best for large enterprises that can afford SQL DBAs and “coders”. What often happens in the larger environments is an SQL developer with bad coding habits can negatively impact SQL performance. These poor SQL developers are often employed by vendors selling applications to credit unions. This badly written code sits on servers and then the iSCSI SAN arrives. What happens next?

Credit unions typically can’t afford SQL DBAs (database administrators) and when they can, the DBA is generally application-focused and not particularly skilled with the O/S and hardware side of SQL tuning. The main point is that prior to placing the databases on the LUNS of the SAN, SQL consolidation into a SAN and general data base needs to be planned.Database SQL

As you can see from the diagram, proper SQL management relies on three disciplines: 

  1. Application programming;
  2. Operating Systems;
  3. Hardware;

Let’s go back to SQL performance again. In a typical pre-virtualization environment, sever sprawl has taken over. When an SQL system fails, the box is isolated. Although troublesome, it can be dealt with independently: the system failure impacts only that specific application and the related back-end SQL database.

In an iSCSI SAN environment this ends. One has to pay attention to the network and application architecture. This is critical because how one integrates SQL into an iSCSI SAN will impact performance of that SAN for years to come. Although processes on a SAN are not necessarily shared, they are not independent either. Additionally, most credit unions are not buying two iSCSI SANs for their HQ and DR sits on the first budget cycle, so paying attention to the proper setup and deployment of the core SAN system is even more important than ever since failover and redundancy of multiple units may not come into play right away.

Below are the important questions to ask regarding the application of SQL prior to integrating your new iSCSI SAN:

1. Are you virtualizing the SQL head server? 

2. Are you breaking apart the application service of SQL from the data base itself?

3. Are you going to have a hybrid SQL environment with a virtual O/S instance and the database placed on the SAN itself?

4. Are you planning to upgrade the SQL to 2005 prior to integration?

5. Would you find a benefit in consolidating SQL servers across the enterprise into instances running on one or two servers? Most credit unions run SQL on a lot more servers than they are actually aware of (think: imaging, antivirus, helpdesk, MSD).

6. Is HA (high availability) with SQL a goal? HA is accomplished through clustering. It is best to integrate clustering prior to joining with your new iSCSI SAN.

Remember that pre-planning with Microsoft SQL is important as credit unions enter the virtualization and iSCSI SAN worlds. Much of the planning revolves around a solid best practices approach to SQL use and management, just as much so as it does with iSCSI itself. How much ROI one can get from an iSCSI SAN is predicated on this long-term pre-planning. The main areas that I think will drive strong ROI results for a credit union are:

1. Can I support this type of environment from a management costs perspective?
2. How can I instruct my staff on these critical path items so that my cost of ownership remains low?
3. What IT management issues will cause my costs to go up? These issues might include inefficient use of SAN resources, poor planning or executing requiring more SAN target resources sooner than expected

 The items that I have not covered with this posting but will follow soon are: 

  • iSCSI benefits and developing a business case
  • iSCSI and DR
  • iSCSI and backups
  • iSCSI and infrastructure routing

Backup and Corruption Protection: “I want to go tapeless”

May 15, 2008

I constantly hear:

1) “I want to virtualize my environment.”

2) “I want to eliminate tapes and go tapeless.”

3) “I want buy a SAN.”

4) “I want to replicate between my headquarters (HQ) and DR facility.”

The key here is appropriately mixing technologies. By design, and based on the laws of physics, virtualization and SANs need speed and high performance. iSCSI is surpassing Fiber Channel this year in shipments and is the only way for credit unions to go. The cost of supporting FC and the training uplift needed for staff and personnel is not worth it. My point is that your SAN and virtual hosts need to be screaming demons from a performance perspective. 

Back-up is another story. Back-ups can be slow and can be scheduled. When it comes to back-up ask yourself what you want to do. For example do you wish to:

1) Use traditional back-up software?

2) Leverage VCB and a proxy server to back-up your virtual hosts to tape?

3) Implement an ‘in the cloud’ backup solution?

4) Use a disc-to-disc approach for backup (cool method)?

You should also ask yourself the following questions:

1) What is the method for corruption protection? This is key to the design and architecture of your system.

2) What is the present cost to back-up? Costs include Iron Mountain, tapes, and maintenance fees.

3) What is the cost to lose back-ups now? Costs include communications with members, and fines.

4) Do you want to plan for archiving now? Do you want to use a SAN approach of integrate with cheaper back-up discs when archiving? What about your core system?

5) Can you integrate all aspects of the CU? This includes Microsoft, Imaging, and Unix. Can you integrate your core system into the back-up solution? What about the imaging system? What about your Microsoft environment? In summary, can we embrace a comprehensive solution with the back-up system? 

6) Can you truly “go tapeless”? What about the O/S tape from the core system?

I hope that this gives you a general impression of what questions to ask when discussing backup, DR, SAN and virtualization.

Implementing an IT Disaster Recovery Plan That Works (Part 2)

May 3, 2008

Here’s another story of a larger credit union client of mine.

The credit union said to me one day: “Bill, we have all this stuff and my staff if good. How do we pull all this software and hardware together into a comprehensive Disaster Recovery program?” Over the past two years, the credit union had acquired the following:

  • Fatpipe load balancers;
  • VMware;
  • Backup software (Issue: tapeless versus traditional? They were partial to tapeless);
  • Doubletake;
  • Platespin;
  • HP NAS appliance;
  • iSCSI SAN from Lefthand;
  • DR Site;
  • Connectivity;

The executive summarized his predicament to me like this: “the products all appear to be good.” I agreed, but based on his current network problems, tight back-up windows, huge WAN latency and more, it appeared that several of these products had overlapping functionality causing them to argue and step on each other. “There is no way I can roll this out into production without being sure” he said.

The credit union asked me to come up with solutions in several areas, in particular the executive wanted answers to these questions:

  • How can they repurpose the HP NAS so that the investment is not wasted?
  • What is the best way to use the iSCSI SAN from a block level replication perspective?
  • How is back-up and corruption protection going to be handled?
  • What function will Doubletake play in the new design?
  • How will the WAN network respond to the new design? The credit union had a combined MPLS and point-to-point architecture.
  • Why are backups barely being completed overnight? This could be indicative of bigger issues that need to be solved first.

For larger credit unions, one must approach back-up differently, often through production uses of a SAN.

The following summary of questions will help you understand Disaster Recovery as two necessary categories including 1) backup and correction protections and 2) production SAN and virtualization.

These two categories are what I explore with clients in the process of determining the exact level of customization required for their DR environment.

Backup and Corruption Protections

1. Do you want to keep managing backup tapes for Microsoft systems, core systems, imaging systems? You should know the costs for this process (for example, Iron Mountain)

2. What is the risk for this process? You might consider losing tapes, theft, or other eventualities.

3. If you move to a tapeless system what is the local restore capability?

4. What about “in the cloud” solutions? There are some cool ones that work really well.

5. Does your backup solution provide local corruption protection?

6. If I make a call to Microsoft for support will they take my call?

7. Will the tapeless solution recover the O/S of the core processor? The only answer is no, so are you really tapeless?

Production SAN and Virtualization

1. Why do I need virtualization and an iSCSI SAN?

2. Why is virtualization perfect for my day-to-day operations?

3. Why is virtualization a ‘magic bullet’ for my DR initiatives?

4. What are the downsides of virtualization?

5. How do I know if I can virtualize a system?

6. How do I assess the readiness of my virtualization?

7. What about Microsoft systems that are iSCSI aware? SQL 2005 and Exchange 2007 are both options. You should consider how this might change your iSCSI SAN and virtualization objectives.

That’s enough information to get you started on the path to implementing an IT Disaster Recovery Plan that works. (See my previous post for questions related to a smaller credit union.)