A Buyers Guide for MPLS Wide Area Networks Questions to Ask (Part 2)

August 15, 2008

The following is a continuation of a previous posting where I review common questions and compliance issues that credit unions frequently ask. I hope that this helps in your planning and decision-making process. 

1. Flexibility In Case Of Disaster: Recovery and Avian Flu Remote Working. This topic is big from a business resumption perspective and it is important to have options during uncertain or chaotic times. Has the carrier explained how it will handle the redirection of branches to the DR site in the event of a disaster? What is the process for such a redirection? Can instant changes be made if the CU owns the routers? What about instant changes to a “managed service” carrier?  

2. Flexibility When Faced With Day-To-Day Operational Issues. This issue relates to finding and determining problems on your POP diverse sites, or “Big Branches”, including routing issues on two routers or Class of Service (CoS) issues between two separate routers.

Another situation might be whether or not a new subnet or manned subnet can have encryption, Quality of Service (QoS), and routing all happen seamlessly and immediately after having zero calls to the service provider. This situation might occur if the credit union or a 3rd party local integration vendor needs to perform troubleshooting. If your answer is no, then what is the process? 

3.IMPORTANT: MPLS Project Rollout. You should know how an MPLS project rollout is handled if the CU chooses either a managed or unmanaged product. There is a massive difference here between each carrier’s approach. Some carriers are so cumbersome that it adds a significant burden to the project rollout. To bid on and manage the rollout communication process between last mile carriers (Verizon) and long haul carriers (Qwest, AT&T, Spring), you might consider using a broker.

In a recent conference call I participated in, two credit union clients of mine were discussing MPLS. One had successfully rolled out their MPLS to 14 sites last year, and one was in the final stages of their selection process. One of the important suggestions my client made to the CU was to clean up cabling prior to rollout. This means extending DMARCS and ensuring there is necessary documentation and cabling in place for the last mile carrier technician. I couldn’t agree more: nothing has delayed MPLS rollouts for me more than this issue.

4.Security. Understanding the recommended security architecture is essential. There are enormous differences in carrier options, and the nuances will make your head spin. Carrier marketing department security will not suffice for a credit union. You should know the overall MPLS encryption architecture and understand HQ to DR site encryption. 

If the carrier is recommending a “hosted” call center, then you should know how the network encryption will be addressed, as well as laptop security access into the credit union network. Security issues revolving around working at home, traveling, and avian flu, and DR remote site workers all need to be taken into account.

 5.Disaster Recovery and Wireless Access. It is necessary to know how remote workers can access the CU network in the event of a disaster. Would users be accessing the Internet and hitting the CU network via the credit union’s own security infrastructure, or would the employee be leveraging an “in the cloud” solution for remote access? A client of mine uses an “in the cloud” firewall service, and it makes me very nervous….with a Tier 2/3 provider as well.

6.Credit Union Network Remote Access For Home Worker or Travelers. You will definitely need to describe how home workers and travelers will remotely access the credit union network using the new MPLS network. Will users be utilizing the CU security infrastructure (a private approach), or a carrier network via another method (semi-private approach)? You might also be asked whether “end-point” security is offered for remote access users, and to explain how it works.

I recommend a CU use its own security infrastructure in most cases to enable remote access for the network and to provide end-point security.

7.Quality of Services (QOS) or (COS). This issue relates to disaster recovery QoS protection from HQ to the DR site. You need to know how the carrier will protect replication traffic and prevent ripples that will affect the entire network. In my experience, carrier QoS/CoS will not work for a credit union. Whatever they might call it (four buckets of protection, color coded etc.,), it is all marketing hype. Credit unions need granularity in QoS. I will write about QoS in more detail as requested.

You might also be asked what would happen if you needed to examine Citrix traffic and manage applications within ICA (independent component analysis) or print traffic within ICA. Also, how fine grained and differentiated can you get with traffic types? You should understand the following types of traffic:

I) AD (replication versus authentication)
II) Exchange
III) Citrix ICA
IV) Terminal Services RDP
VI) Citrix
VIII) VDI – other
IX) DR Replication

Other differentiations you will need to make are those between CIFS (Common Internet File System) traffic and IPC (Inter Process Communication) pipe traffic as well as authentication traffic versus AD replication. How are RTM (Round Trip Measurements) for servers and clients met and achieved, and what is the sampling rate? How are they doing this at a layer 7 level? You will need to know how many buckets of CoS/QoS you get (carriers give you between 4 and 6), and whether or not it is layer 7-aware. Finally, issues regarding how bandwidth and queuing is controlled should be understood, and how you achieve ZERO dropped packets.  

8.The SLA (Service Level Agreement) . For this type of question, you are going to need to know the SLA process, including how the service is set up, including how to get a troubleshooting-engineer. If you have a POP diverse routing problem, would the same engineer be likely to fix problems with encryption and VoIP jitter issues?

Other issues might include how to find a noisy broadcasting NIC (Network Interface Card), including which escalation path should be chosen and how long it should be. You should also know how many mission critical calls versus troubleshooting calls you get monthly, and who determines the SLA. Do you determine it or do they? Do they have an SLA example that has been set up for a client?

I hope that this guide helps you during the implementing of your MPLS network. You should now have a good idea of what questions might be asked, and what to look for in establishing solutions for your credit union.


A Buyers Guide for MPLS Wide Area Networks Questions to Ask (Part 1)

August 7, 2008


This guide has been put together to help credit union buyers quickly understand the nuances of MPLS networks. After being personally involved in the implementation of all major carriers, I have developed considerable knowledge and gained valuable insights into the details of network implementation. I think this guide will be useful for any credit union, regardless of size, to help wrap one’s mind around the scope of these projects. Hopefully this list will spur your thoughts to deeper questions about carriers, integrators, and your own staff.

MPLS migrations have become very common for credit unions over the past few years due to the fact that MPLS (WANs) offer credit unions tremendous flexibility, especially router flexibility in the case of a disaster. An additional impetus is that carriers (Sprint, AT&T, Verizon, Qwest) are trying to move clients away from older technology and toward their newer, cheaper to maintain, platforms. But just because carriers say in a slide presentation that they offer a particular functionality, credit union buyers should be aware that this DOES NOT mean that a particular functionality is good for the credit union. I have witnessed disasters first hand stemming from marketing miss-information regarding what the big telecoms sell. What you should remember is that the slides that you are observing in sales presentations are marketing slides. They are not “how to” guides nor are they necessarily accurate representations of current technology or products offered by the carrier.


To develop a good branch strategy, the following items should be kept in mind.

1. Bigger branches need High Availability, meaning that despite there being no disaster at the HQ Data Center, branches cannot communicate with core systems.

2. The CU must be able to seamlessly connect to the core system resources through the proper handling of routing paths. It is important to understand which routing method (EIGRP, BGP, etc.,) will be used for multiple access points from the CUs biggest branches. The devil is in the details with this.

3. MPLS network encryption will be an important item for the NCUA. It’s important to be sure about how this will be handled, where the encryption endpoints are, and who is doing the encrypting.

4. QoS (Quality of Service). See details in part two.

5. Managed and Unmanaged MPLS networks.

6. A credit union might be easily fooled into believing there is a multi-billion dollar organization protecting them when a carrier is managing their network. I have not found this to be the case. A managed network can work quite well when one has simple requirements, but credit unions do not have simple requirements. In fact they are quite complicated, and the decision to choose a managed or unmanaged network needs to be examined carefully based on the flexibility given to the credit union.

An unmanaged network is highly flexible, much more so than a managed network. CUs require maximum flexibility, and by flexibility I mean “owning, acquiring and managing” all of your own equipment and having the ability to self-manage the network. Of course, the usual fear is “do I have enough staff to do this?” In reality, if a network is set up and integrated properly, a CU can operate with higher degrees of user satisfaction and flexibility than through a carrier. If staff competency and training is an issue, there are other viable management options to consider that can bridge this gap.

In summary, MLPS network items that require the most attention include the following:

  • 1. Branch connectivity, including high availability and disaster recoverability.
  • 2. ATM connectivity.
  • 3. Disaster Recovery preparedness including redirection to a DR site of branches, ATMs, shared branching, and 3rd-party connectivity. Avian flue remote workers should also be considered.
  • 4. Encryption.

5. QoS

6. SLA (service level agreements) including knowing how, in the above categories, problems with the carrier network are addressed. It is important to know what role your broker-of-choice has, and what recourse the CU has when carrier mistakes occur.

7. From a pre-project planning perspective you should make sure that all DMARCS extensions are completed prior to the last mile carrier going on-site to turn up the lines.