Network Troubleshooting Tip - Using Markers to Cut Trace Analysis Time

When we get to the point in an investigation where we are about to break out Wireshark, the complexity of the packet analysis can seem quite daunting. And yet, by covering a few key points, we can dramatically cut the time needed to analyze any diagnostic data.

In my previous post we looked at the importance of a basic understanding of the topology of the system under investigation. In this blog I'll cover the use of markers; a ridiculously simple, but amazingly powerful, concept. A marker places a distinctive packet in network packet trace data that we can easily find with Wireshark.

The RPR manual contains six pages of information on markers, covering suggested markers and what to use them for. If you haven't used markers before you are in for a real treat. Once you get the hang of them, you'll wonder how you ever did without them.

Let's imagine you've been investigating an intermittent slow response time problem for a bunch of users. Nobody is quite sure what's causing the problem, although the application and platform teams insist it's not them. You know the drill; if the cause isn't obvious it must be the network, right?

Luckily, a user experienced the problem this morning, and you had packet traces running. The bad news is that you have 500 GB of trace data (about 5 billion packets) and the user is vague about the time of the problem.

The first strategy you can adopt a three-step procedure:

• Filter the trace data to pick out just the traffic for the user's PC or VDI server using the IP address

• To filter large amounts of data, take a look at the Filter feature of Workbench for Network Engineers Community Edition

• Use Wireshark with TRANSUM to find the slowest APDU Response Times

• TRANSUM is included in Wireshark 2.4

• Determine if it's a client, network or service by studying the Service Time, Request Spread and Response Spread

• See my Sharkfest 17 session

This approach has it's merits, but there's a drawback. When you find a slow APDU Response Time you don't know if it's related to the problem the user saw or just a transaction that regularly takes a long time to respond. This is where markers win out. If we set up a simple marker mechanism for users, we can get them to send a distinctive packet right after they get the problem.

We have three favorite markers:

1. Get the user to send a ping immediately after the problem occurs

• The best way to do this is to drop .bat file (as an icon) on their desktop, the user double clicks on the icon and the .bat file sends the ping

• Make the ping a distinctive length; we use 101 bytes, 102 bytes, 103 bytes, etc.

• Use ping -n 1 -l 101 app_server_ip - use the option -n 1 to prevent Windows sending 4 pings (which is the default)

1. If you are looking at a web-based system, get the user to enter a distinctive URL in Internet Explorer, Chrome, Firefox, whatever.

• If the main application server is web01, get the users to browse to http://web01/marker101.html

• This will cause a 404 error but that doesn't matter as you just need to be able to find the marker

• Make it even simpler, set the URL as a hot link icon on the user's desktop

1. Use the TribeLab Sonar tool

• It just presents a button from a simple web page

• User's love it - one customer calls it the Big Blue Button

The Sonar marker is very easy to find in a Wireshark trace; just find the string TLMarker and you'll get something like this:

You can clearly see the marker in frame 173, and just above in frame 145 you can see a 29.485-second response time.

For a ping marker use the Wireshark Find using the Display Filter option and the filter expression icmp && data.len==101

What I've shown you here is just one simple use of markers; you can see this is powerful stuff. If you want more examples, take a look at Matthew York's fantastic session on markers at Sharkfest 16. You can view it on:

• YouTube - https://youtu.be/M2SE8hf4h2s

• TribeLab MP4 - https://community.tribelab.com/mod/page/view.php?id=533

One final thing I should share with you. Workbench has a Marker Find feature that allows you to rip through hundreds of files to find Sonar and Ping markers, and it does it in seconds. Better still, you can right click on the markers found and launch Wireshark with the right trace file loaded, positioned at the marker trace entry. Saves you hours of trawling through trace files.

Best regards...Paul

PS: Grab a copy of our latest paper Five Overlooked Ways to Quickly Identify Network Problems with Wireshark from here.