What tends to happen, eventually, is that one of these outages becomes damaging enough to cause management to demand a better infrastructure. There’s always a catch, however, and in most cases it’s money. Luckily, you can build a highly reliable network without breaking the bank, as long as you focus on eliminating high-impact, single points of failure.
In this article, I’ll explain how to create a highly available SMB network. In order to be as vendor-neutral as possible, I will try to avoid specific technologies, and will instead lay out some goals, along with common methods for meeting those goals. For similar reasons, I will not detail exact costs, but where possible, I’ll give you relative costs. Finally, I won’t include virtualization options in this article, and will instead focus on standard client-server infrastructures. While virtualization can allow for very highly available infrastructures, a fully virtualized infrastructure is beyond the grasp of most small to medium sized businesses.
Client design: rapid recovery
Before discussing the best options for availability on the client side, we should point out one highly important fact: Client failures are not generally high-impact failures. One client machine failing will typically only affect one user. While there are occasions (such as the failure of a major executive’s system) that break this rule, in general, client redundancy is not worth the expense. Instead of redundancy, the focus for client systems should be on rapid recovery.
Enabling rapid recovery of client systems involves three major components: commodity hardware, network storage of user data, and disk imaging. Essentially, the goal is to be able to replace the user’s system as quickly as possible. If you use commodity hardware, it becomes easy to maintain shelf spares that you can quickly swap for a defective system in the case of hardware failure. For major software failures, maintaining an up-to-date base disk image allows you to rapidly redeploy a working image onto the machine. In both cases, however, user data must be housed on the network, or data loss will occur.
The details on how to implement this will be dependent on your chosen OS, but the basic principles are the same. Where possible, when making tradeoffs, try to choose the option that reduces downtime as much as you can. For example, to image a system you could use a basic image with just the OS, and use a software installation service to individually install applications once the machine is online. Alternately, you could use a complete image with the OS and all of the apps preinstalled. In general, the second option is going to be faster, but has the disadvantage of being harder to maintain. If reducing downtime is your primary goal, however, using complete disk images is the obvious choice.
Laptops, however, can complicate the client design. Since laptops are not always connected to the network, using entirely network-based data storage is not acceptable for these users. The solution, generally, is to use a caching mechanism. For example, on Windows systems, you can use offline files to handle the caching of raw data files, and allow roaming profiles to be cached to handle user profile issues.
Next, examine your cable plant. If you are installing new cable, it’s generally a good idea to pull double the number of cables for each drop. There are a few reasons for this. First, if you have some kind of major failure with a cable (rats chewing through insulation are not unheard of), you can quickly switch off to a different jack. Second, if at some point in the future you decide to use NIC teaming to increase the speed or reliability of the system’s network connection, you will have that option. Also, most of the cost of running cable is in labor, so it is generally very cheap to start with two cables for a drop, but very expensive to go back and add a second one later.
Switching design basics
On the switching side, your key considerations are:
- Whether or not to use a collapsed core
- Core redundancy and trunk redundancy
- Switch choices and load out
- Spanning Tree design
- L3 switching model
For the base design, for a network of this size, you are typically looking at a collapsed core design, similar to the one shown below. This means that all of the real switching complexity is concentrated at the core.
The advantage of a collapsed core design over a typical core/distribution/access model is that, in a small- to medium-sized network, you will likely not need the additional back-end speed that a full hierarchical design provides. By using the same switches for core and distribution functions, you lower the cost of entry quite drastically, while maintaining a high level of performance and features. Also, if you expand in the future, you can easily repurpose your old core/distribution switches as plain distribution switches.
Your core switches are perhaps the most critical component in the network, so you should make them as redundant as possible.
The advantage of a collapsed core over a flat (single layer) design are also mostly economic. While you could buy larger modular core switches and use them for access functions as well, you seriously impede your ability to scale.
For instance, if you have a switch that supports six blades in addition to management blades, you probably have the option of filling it with either six 10GbE blades (each with six ports), or six 1GbE blades (each with 24 ports). On the first option, assuming you send one 10GbE link to each access switch, and that each access switch supports 48 1GbE ports, your effective maximum coverage is 48 x 36, or 1728 ports. With the second option, the maximum number of supported ports drops to 24 x 6, or 144 ports.
Now, there are obviously a lot of other factors that actually determine the maximum coverage. Switching fabric limits and actual use cases are good examples. However, in most cases, you will still find that while cheaper initially, a single layer design will generally cost you more in the long run.
For redundancy, you should always ensure that you have redundant cores and redundant trunks to each access switch. Core switching is perhaps the highest-impact area of the entire network. In a non-redundant design, a failure in the core can take down the entire network, so this is not an area where you should skimp. The typical model (with two core switches) is to configure one trunk port on core switch for each access switch. This gives you two trunks per access switch, each trunk connected to a different core switch. As you get larger, the models get a bit more complicated, but the general rule of thumb is to connect each access switch to at least two core switches.
Both core and access switches should also adhere to certain specifications. On the access side, the requirements are pretty basic. A typical 1U fixed configuration switch should be fine in most cases, as long as it contains at least two ports of the correct speed for uplinks. If redundant power supplies are offered, they are almost always a recommended purchase, but they are not really a requirement for access switches.
Additionally, make sure the access switches support a reasonable feature set, including VLAN Management, a management interface that you or your staff are familiar with, and basic RSTP configuration. You shouldn’t be doing anything complex at the access layer, so features like Layer 3 switching and access lists aren’t necessary.
For core switches, the feature set requirements are necessarily more severe. It goes without saying that all of the features needed at the access layer are needed at the core, but additionally you will definitely need fast Layer 3 switching capability. If you plan to use a dynamic routing protocol, ensure that it is supported. It’s also generally a good idea to ensure some basic firewall features are available. Even if you do not plan to block any traffic at the core, the ability to do so on occasion can come in handy.
Physically, cores should almost always be in a modular chassis form factor, because that is typically the only way to get the needed level of redundancy. As mentioned earlier, your core switches are perhaps the most critical component in the network, so you should make them as redundant as possible. Dual management modules are a must, as are redundant power supplies. Additionally, with some vendors, different wattages of power supplies are offered for different switch loadouts. Make sure that each power supply you buy has a high enough rating to support the entire switch.
Implementing spanning tree
Spanning tree will build a loop-free topology, or tree, beginning at the root and spreading outwards.
For spanning tree, there are several areas of concern. The first, and perhaps most important, is root placement. Spanning tree will build a loop-free topology, or tree, beginning at the root and spreading outwards. It does this by effectively blocking traffic on certain redundant links in the topology, with a focus on trying to find the fastest loop-free path back to the root. This makes the selection of the root switch very important. In a simple environment like this one, all traffic that transits more than one switch will go through the root.
You should configure the bridge priorities on the core switches to make one core the preferred root and the other a backup root. You should make sure that the priorities on the access switches are higher than the core. A good technique is to use the default priority for access switches and a lower priority on the cores. This ensures that if a new access switch is brought online, it will have no chance of winning an election.
Another neat trick, if your switches support individual spanning trees for each VLAN, is to configure each core to be the root for different VLANs. For example, you could make Core1 the root for all odd numbered VLANs and the backup root for all even numbered VLANs. Then make Core2 the root for all even numbered VLANs and the backup root for all odd numbered VLANs. The end result is a crude type of load balancing over your trunks.
One final set of tricks for spanning tree is to correctly designate edge and uplink ports, assuming your switches support these features. By designating a port as an uplink port, you allow spanning tree to reconverge faster from indirect link failures. By designating a port as an edge port, you allow the switch to immediately place the port in a forwarding state, which can save you a lot of headaches when clients that boot very rapidly or boot from the network try to get an IP during boot.
As you might have guessed, the trick with each of these enhancements is to place them on the correct ports. Luckily, in a basic collapsed core like the one shown previously, the correct ports are easy to locate. Just enable each uplink port on each access switch as an uplink, and enable all other ports on each access switch as edge ports. Neither feature should be enabled on the core switches.
Layer 3 switching
The last major switching design decision involves Layer 3 switching. As far as redundancy is concerned, there are only two major concerns: routing method and gateway redundancy. Regarding routing method, in this size of network, you generally do not need to utilize a dynamic routing protocol. A dynamic routing protocol is useful when there are redundant Layer 3 paths between any two points. However, in the simple collapsed core design specified here, there should not be any redundant Layer 3 paths. Instead, there are two gateways (Core1 and Core2) each with the same Layer 3 path. With this design, the need for redundancy is on the gateways themselves. You can provide this by implementing gateway redundancy using Virtual Router Redundancy Protocol (VRRP) or Hot Standby Router Protocol (HSRP).
These two protocols cause your core switches to share a virtual IP that your host systems use as their default gateway. When the primary core goes down, the other core very quickly begins responding to the virtual gateway IP, ensuring that your hosts are not aware of the failure.
Server design: power and redundancy
For servers, the situation is a little more complicated. Being more complex (higher layer) devices, there are a lot more variables to take into account. In general, however, there are two major considerations: general configuration, and which load balancing or redundancy mechanism to use.
When loading out the server, it is generally pretty cost effective in the long-run to pay for redundancy wherever possible.
The first consideration is pretty straight forward. While the actual preferred load-out will vary a bit based on the server’s intended use, here are some general guidelines that will apply to most builds.
First and foremost, power is king in system uptime. While not technically part of server configuration, make sure that your servers are protected by adequate UPS systems. Complex N+1 systems backed by diesel generators are nice, but probably not necessary in most environments. Instead, you should generally shoot for at least 15 minutes of protection for every server, and if you have many servers, features to automatically shutdown the server OS should probably be employed. Double-conversion type UPS systems are generally preferred and are not terribly cost-prohibitive.
Another environmental factor is cooling. Make sure that cooling is adequate in the server room, as heat buildup can be worse than power problems if not carefully controlled. If available, raised flooring with AC blowing from the bottom on the front side of the rack and good overhead exhaust vents on the rear of the rack can help cooling efficiency immensely. Also, when racking equipment, be aware of air intakes and exhausts. Make sure all of your air intakes are on the same side of the rack. Otherwise, you will have one server exhausting hot air, only to have another sucking that same hot air into the intake.
Back to server builds; one thing to look out for when choosing the server chassis is the fan arrangement. Most mainstream server manufacturers these days do a pretty good job of fan and heatsink placement, but it is still something to keep an eye out for. The major heat generating components in most servers are the processors, hard disks, and power supplies, so make sure all of these have adequate cooling.
When loading out the server, it is generally pretty cost effective in the long-run to pay for redundancy wherever possible. Server failures are generally high-impact in that they affect many users. Redundant power supplies, NIC teaming, and a good RAID setup are almost mandatory.
Hardware load balancing and clustering
Once you have made each server reasonably redundant, you can take a look at making each service as redundant as possible. The methods for doing this will depend upon the service, but they generally fall into one of three categories: Hardware load balancing, clustering, or service-specific redundancy (replication, for example).
Hardware load balancing is an effective way to provide both load balancing and high availability for a farm of systems offering relatively static data. Essentially, the hardware load balancer accepts incoming packets for the service, and, using internal logic, determines which server in the farm is best able to respond to the request. The logic used is usually based on system availability, number of current connections, average response time, and similar metrics. By monitoring all systems in the farm, the load balancer can make intelligent decisions about which server will provide the best overall experience for the client.
Hardware load balancers are generally placed in-band, meaning that they sit between the client and the server farm. This gives the additional advantage that there is no direct contact between the clients and the servers, but it creates a distinct disadvantage in that the hardware load balancer becomes a single point of failure. For this reason, most hardware load balancers include the ability to cluster, allowing you to create a “virtual load balancer” out of two or more individual units.
Hardware load balancing (HLB) is also usually fairly expensive. Simple, single-service units for HTTP and other simple services can be had for about the price of a small server, while more complex units from the major players can cost upwards of $10,000 each.
Clustering is another common technique that can provide both load balancing and redundancy. Clustering takes two or more systems and combines them into a virtual server, usually with shared storage. With some clustering technologies, the need for identical hardware for all nodes and shared storage arrays can make it prohibitively expensive. Other clustering technologies (such as Network Load Balancing on Windows or Beowulf on Linux) can be implemented relatively cheaply.
Clustering has the advantage that it can be implemented on virtually all services, no matter how complex. Additionally, in some implementations, clustering can be configured into an Active-Active role, where all members of the cluster actively accept and process traffic. Like HLB, the biggest disadvantages of clustering are expense and complexity.
Service redundancy: DNS and e-mail
The final server redundancy method that we’ll cover is service-specific redundancy. This is probably the most common, and almost always the least costly method of achieving service redundancy. While the methods used differ significantly based on the service, the general idea is usually that multiple servers offer the service and keep their combined data synchronized via some form of replication. DNS is a great example of this in its simplest form.
In DNS, one server is designated the primary server for a namespace. When changes occur to the DNS database, they are written to the primary server. Secondary DNS servers contact the primary server at regular intervals, check the serial number of the database to see if changes have occurred, and if so, pull the changes down. In this way, multiple servers maintain the ability to answer requests for the namespace.
In general terms, service-specific redundancy is probably the way to go in a small network unless you have a very specific reason for using another method. Most common services will include some type of service-specific redundancy. Services which do not generally include a service-specific redundancy option include web servers (HLB is typically used), some database servers (clustering is typically used), File and Print servers (though Distributed File System on Windows servers can provide some redundancy, clustering will likely be needed for full coverage), and most DHCP servers (50/50 or 80/20 scope splits are used instead). These cases will be pretty clear-cut, since you will likely only have one redundancy option. In all cases where you have a choice of method, in a small network you are usually best served by choosing the simplest method for providing the redundancy.
Before moving on, one service that deserves a bit of additional discussion is e-mail. E-mail is such a critical component in modern businesses that it requires special attention. While you can make your internal e-mail servers redundant, this still doesn’t help when your Internet access is completely down. Additionally, having your external mail records pointing back into your network makes for an inviting Denial of Service target. Luckily, there is a simple solution to both problems: Mail bagging.
In a nutshell, mail bagging is the act of sending mail destined for your domains to one or more externally hosted entities at one or more major datacenters. These datacenters then give you a very high uptime SLA (usually as a complete package of hardware, software, and environment) for a fairly minor monthly fee and store and forward all incoming e-mail to your internal e-mail servers. If your Internet link fails, the provider simply queues up all inbound e-mail and waits for the link to come back up. Similarly, because your MX records point to the external provider, your actual mail servers are obfuscated. Additionally, you can usually get anti-spam and anti-virus services from the provider if needed, which can significantly reduce the amount of mail that your internal servers must process.
Since mail bagging has become commonplace and relatively inexpensive, it is strongly recommended in almost every environment.
The last area to examine in the design of a highly available network is external network connectivity, which is also probably the least redundant in most networks. External connectivity can be split into two basic types: Internet, and site-to-site (S2S). Each type has very different concerns and requirements, but let’s begin with Site-to-site connectivity.
S2S connectivity is used to interconnect the organization’s remote locations. S2S connections are the primary paths for company data that is internal. While S2S links are usually owned by external organizations (leased lines, etc.), they will transmit largely the same types of data as the internal LAN does, and are usually treated as internal links. This may not seem like a major distinction, but it makes a large difference in how you configure your network, particularly as it relates to router and firewall configuration.
However, the first and most obvious consideration to the S2S design is the type of connectivity you choose to use. You can either use traditional media, consisting of MAN or WAN connectivity (usually leased from a provider), or you can use a site-to-site Virtual Private Network (VPN). Let’s begin with traditional media.
The main advantage traditional media has over a VPN is that it is at least partially separated from your Internet link. This means that if you have a traditional S2S link, you may actually have an option of a backup path in case of S2S link failure: A VPN tunnel over the Internet link. Additionally, this separation means that you do not have to be concerned with Internet traffic interfering with S2S traffic. Finally, your S2S traffic is effectively shielded from external attacks due to its private nature.
The disadvantages of traditional media, however, are numerous. First, it’s usually prohibitively expensive for small businesses. Second, traditional media will generally require dedicated hardware (a router and CSU/DSU, for example), further increasing the cost and providing another possible single point of failure. Finally, in order to provide redundancy with traditional media, you need to provide it physically, by installing new links. With a S2S VPN, redundancy in your Internet link provides redundancy for your VPN.
The bottom line is that in a small to medium business setting, it usually does not pay to use traditional media. However, if you place a premium on the separation of the link from your Internet link and the advantages that brings, then traditional media is the correct choice.
If you choose to use traditional media, one major concern you will be faced with is the overall WAN topology. In this regard, if your WAN layout is centralized, with a HQ or hub location surrounded by remote locations that connect to the hub, a hub and spoke is probably the best layout. If you are decentralized, then try to match up your links to your data traffic as much as possible. For example, if the Chicago location spends most of the time communicating with the St. Louis location, then try to directly link those locations.
For VPN connections, topology isn’t nearly as much of a concern, but it should still follow the guidelines listed above. Try to follow the data path with your topology.
One final consideration for S2S links: If you choose to add redundant physical links for S2S connectivity, you will need to enable dynamic routing on the S2S router to get the full benefit.
On the Internet link side, the decision is actually much simpler, in most cases. Redundant Internet links, with the possible exception of very primitive configurations (dial-up modem backup, for example), are out of reach for most small and medium businesses. This is because to truly have redundant Internet links, you need two physically independent links (i.e. not multiplexed into a larger link at the demarcation point) from two different providers who are preferably connected to two different Tier 1 networks. In any other case, you likely still have a single point of failure somewhere. Because of the difficulty and expense of getting truly redundant Internet connectivity, it is not usually seen in the SMB space.
If you actually require this Internet link redundancy for business services, your best bet is probably to co-locate your critical servers with a managed hosting provider that already has the necessary level of redundancy.
The last thing to mention in the external connectivity discussion is the need for redundancy on outbound networking equipment, such as your external routers and firewalls. Again, these are high impact areas, so redundancy is probably warranted wherever possible. Firewalls are usually low-hanging fruit in this regard, while routers can be a bit of a pain when using certain Internet links. Still, even for routers, if you can afford to provide hardware redundancy or even quick recovery (such as shelf units), you should probably do so.
Testing and Conclusion
Once you have completed building your new network, before you do anything else, you need to make sure to test it thoroughly. This means that you need to first implement some type of automated monitoring system. If you do not already have one, this is a very important tool to use. Once you have enabled all of this redundancy, when something fails, it will likely not be immediately noticeable. This can quickly lead to a situation whereby something fails, the systems fail over without a hiccup, and then they run non-redundantly for months until another failure finally takes down the network.
To avoid this situation, you need a monitoring system that can monitor not only the equipment “farms”, but also the individual nodes. Additionally, in the case of tiered applications (front-end/back-end) the system should be capable of testing the full application. One way of doing this simply is to submit data to the front end that requires a backend query and then evaluating the response to ensure that the query was successful. Other than this, though, the testing tool does not have to be overly complex. While management apps that can automatically take recovery steps, query internal system variables, and the like may be nice, remember: The main goal is to find out about errors when they occur.
Once you have the monitoring system in place, schedule a maintenance window and systematically test every part of the redundant infrastructure. Disconnect redundant power supplies, network links, power down redundant nodes, shut off management modules in the core switches, and so forth. Make sure that you thoroughly test every redundant portion of the infrastructure, make note of anything unexpected, and resolve the issues found.
When finished, you will have a network that runs like a top even when things are going wrong, giving you plenty of time to find and resolve problems in the correct manner… And allowing you to enjoy the occasional quiet afternoon at the office.