Network Troubleshooting Tip - Using Markers to Cut Trace Analysis Time
When we get to the point in an investigation where we are about to break out Wireshark, the complexity of the packet analysis can seem quite daunting. And yet, by covering a few key points, we can dramatically cut the time needed to analyze any diagnostic data.
In my previous post we looked at the importance of a basic understanding of the topology of the system under investigation. In this blog I'll cover the use of markers; a ridiculously simple, but amazingly powerful, concept. A marker places a distinctive packet in network packet trace data that we can easily find with Wireshark.
The RPR manual contains six pages of information on markers, covering suggested markers and what to use them for. If you haven't used markers before you are in for a real treat. Once you get the hang of them, you'll wonder how you ever did without them.
Let's imagine you've been investigating an intermittent slow response time problem for a bunch of users. Nobody is quite sure what's causing the problem, although the application and platform teams insist it's not them. You know the drill; if the cause isn't obvious it must be the network, right?
Luckily, a user experienced the problem this morning, and you had packet traces running. The bad news is that you have 500 GB of trace data (about 5 billion packets) and the user is vague about the time of the problem.
The first strategy you can adopt a three-step procedure:
Filter the trace data to pick out just the traffic for the user's PC or VDI server using the IP address
To filter large amounts of data, take a look at the Filter feature of Workbench for Network Engineers Community Edition
Use Wireshark with TRANSUM to find the slowest APDU Response Times
TRANSUM is included in Wireshark 2.4
Determine if it's a client, network or service by studying the Service Time, Request Spread and Response Spread
This approach has it's merits, but there's a drawback. When you find a slow APDU Response Time you don't know if it's related to the problem the user saw or just a transaction that regularly takes a long time to respond. This is where markers win out. If we set up a simple marker mechanism for users, we can get them to send a distinctive packet right after they get the problem.
We have three favorite markers:
Get the user to send a ping immediately after the problem occurs
The best way to do this is to drop .bat file (as an icon) on their desktop, the user double clicks on the icon and the .bat file sends the ping
Make the ping a distinctive length; we use 101 bytes, 102 bytes, 103 bytes, etc.
Use ping -n 1 -l 101 app_server_ip - use the option -n 1 to prevent Windows sending 4 pings (which is the default)
If you are looking at a web-based system, get the user to enter a distinctive URL in Internet Explorer, Chrome, Firefox, whatever.
If the main application server is web01, get the users to browse to http://web01/marker101.html
This will cause a 404 error but that doesn't matter as you just need to be able to find the marker
Make it even simpler, set the URL as a hot link icon on the user's desktop
Use the TribeLab Sonar tool
It just presents a button from a simple web page
User's love it - one customer calls it the Big Blue Button
The Sonar marker is very easy to find in a Wireshark trace; just find the string TLMarker and you'll get something like this:
You can clearly see the marker in frame 173, and just above in frame 145 you can see a 29.485-second response time.
For a ping marker use the Wireshark Find using the Display Filter option and the filter expression icmp && data.len==101
What I've shown you here is just one simple use of markers; you can see this is powerful stuff. If you want more examples, take a look at Matthew York's fantastic session on markers at Sharkfest 16. You can view it on:
YouTube - https://youtu.be/M2SE8hf4h2s
TribeLab MP4 - https://community.tribelab.com/mod/page/view.php?id=533
One final thing I should share with you. Workbench has a Marker Find feature that allows you to rip through hundreds of files to find Sonar and Ping markers, and it does it in seconds. Better still, you can right click on the markers found and launch Wireshark with the right trace file loaded, positioned at the marker trace entry. Saves you hours of trawling through trace files.
PS: Grab a copy of our latest paper Five Overlooked Ways to Quickly Identify Network Problems with Wireshark from here.