Diffusing The IT Blame Game With Network Visibility

How well does your company communicate internally? Specifically, how well do your IT departments communicate with each other? Enterprises typically contain four or more IT sub departments (Security, Network Operations, Virtual DC, Capacity Planning, Service Desk, Compliance, etc.) and it’s quite common for them to be at odds with each other, even in good times. For instance, there’s often contention over capital budgets, sharing resources, and headcount.

But let’s be generous. Let’s say that in normal operations things are usually good between departments. What happens if there’s a breach though, even a minor one? Then things can change quickly. Especially if there are problems with acquiring accurate monitoring data for security and troubleshooting areas. Finger pointing can quickly result.

So, what can you do? We recently discussed this on an LMTV podcast How To Diffuse The IT Blame Game. One of the answers is to create complete network visibility for network security and network monitoring and troubleshooting activities.

For instance, here are three common sources of issues for most IT organizations:

• There is a lack of proper access to network data

• Analytic and security tools can be modified, moved, or just plain disappear without permission

• Capture and analysis of monitoring data can create business risk and problems for other departments

Lack of data access is self-explanatory; you just don’t have access to the data you need, when you need it. One reason is that if you need to make changes to the network, you typically need to get permission from the company Change Board (your network oversight governance organization). This usually takes days, maybe weeks depending upon the business.

Besides Change Board approval being an issue for connecting equipment to the network, this is also a common issue for SPAN port filter configurations as well. Any change to the network routing switch could potentially create a service impact. SPAN ports also constantly need reprogramming to capture new data. This could affect others using that particular filter and cause an unknown loss of data to the security and monitoring tools currently in use. The IT engineer may or may not know that the new filter is clipping important data – until there’s a problem, and someone gets blamed.

A second issue is that you may not have the budget you need for certain types of equipment. Even if other departments have the equipment, they often don’t want to share. Sharing is often a problematic issue for IT departments because the security and monitoring tools often get moved or reconfigured which causes irritation among staff members. Besides individual tool sharing, some enterprises have created “crash carts” that have a set of common diagnostic tools for immediate troubleshooting purposes. However, these crash carts and their tools are often not reset to default settings, which means that the next user has to waste time resetting and reconfiguring the equipment. This stress is heightened if there is an event, such as a security breach, network failure, or application failure. These incidences result in troubleshooting time delays, higher costs, and SLA/QoE problems. This is true even if the sharing problem turns out to be that monitoring data filters were changed without permission, as this itself can cause network and application outages or increase mean time to repair (MTTR).

A third common issue is that the capture of the data leads to other problems. For instance, encrypted data can be captured, decrypted, and then the data passed in the clear to monitoring analysis and storage devices. This is a good and necessary thing – you want and need to be able to analyze the data. Unfortunately, the other side of the coin is that this can, depending on what you do with that data and where it goes, cause regulatory compliance issues. Several standards, like PCI DSS and HIPAA, require that data in motion and data at rest be secured. In addition, should this clear text data be captured as part of a network breach, you have just increased your company’s financial liability.

As mentioned earlier, one of the biggest challenges for IT staff today is to get the proper network information they need, when they need it, so that they can make informed decisions about network security and problem resolution. Proper network visibility is the solution. Without this visibility, how do you know that you haven’t been breached? If you have been breached, what was affected? IT professionals know they cannot prevent all attacks, so they need to focus on quickly detecting signs of infiltration. This helps all IT departments avoid becoming the victim of the blame game. No one wins in the blame game.

But what can you really do about the problems. Here are some examples of how you can increase network visibility and eliminate some of the pitfalls:

• Add taps to replace SPAN ports. Taps are set and forget technology which means that you only need to get Change Board approval one time to insert the tap and you are done.

• Add a network packet broker (NPB) to eliminate most of the other Change Board approvals and eliminate crash carts. The NPB is situated after the tap so you can perform data filtering and distribution whenever you want. By implementing a tap and NPB approach, you may be able to reduce your MTTR times by up to 80%.

• Add an NPB to perform data filtering. The NPB performs data filtering to send the right data to the right tool whenever you need it. This improves data integrity to the tools and improves time to data acquisition.

• Add an NPB to create role-based access to filters. This eliminates the “who changed my settings” issue and allows multiple departments to share the same NPB.

• Add virtual taps to get access to the often hidden East-West data in a virtual data center or cloud network.

No one wins at the blame game as it’s a zero sum game. Even if one department appears to win, the whole group typically loses. One of the best things an IT department can do is increase network visibility because it gets at the core of the issue instead of treating symptoms. This is what will help reduce incidents, reduce long term costs, reduce troubleshooting times, and increase staff happiness.

If you want more information on this topic or network visibility solutions, check out the video podcast How To Diffuse The IT Blame Game and the ebook The Definitive Guide to Network Visibility Use Cases.

Author: Keith Bromley is a senior product marketing manager for Keysight Technologies with more than 20 years of industry experience in marketing and engineering. Keith is responsible for marketing activities for Keysight's network monitoring switch solutions. As a spokesperson for the industry, Keith is a subject matter expert on network monitoring, management systems, unified communications, IP telephony, SIP, wireless and wireline infrastructure. Keith joined Ixia in 2013 and has written many industry whitepapers covering topics on network monitoring, network visibility, IP telephony drivers, SIP, unified communications, as well as discussions around ROI and TCO for IP solutions. Prior to Keysight, Keith worked for several national and international Hi-Tech companies including NEC, ShoreTel, DSC, Metro-Optix, Cisco Systems and Ericsson, for whom he was industry liaison to several technical standards bodies. He holds a Bachelor of Science in Electrical Engineering.