Excessive DNS Retransmissions on a Mac - aka "I really want an IPv6 address!"

When I teach Wireshark classes, I have everyone capture their own traffic. Nothing breaks the ice like telling people, "OK, it's time to sniff yourself". It also helps them correlate what they see in the packet capture file (aka pcap) and what they did as the user. Two weeks ago, I noticed something new as I was capturing on my Macbook.

Normal DNS

DNS is pretty straight forward at the client side. You send a query to your DNS resolver, and they either answer from cache, or go to other DNS servers for the information (recursion). The response should be quick, and have a positive reply. The same DNS transaction ID would be used for both the query and response. That is what the two hosts (and Wireshark) use to correlate the two packets. Transaction IDs should not be reused immediately. That's how thousands of concurrent queries are kept separate.

Normal DNS Retransmissions

However, if the client (or DNS server acting on the clients behalf) receives silence, they ask again. Different operating systems have different timers. I've seen Windows 10 commonly use approximately 25 milliseconds, and a Mac use up to 1 second. Often retransmissions occur because the Windows clients time out so quickly, the server's original response and the client's retransmission pass each other in the cloud. Below you can see that if the client had just waited another 639 microseconds, it would have had it's answer.

Sometimes the network does actually drop a packet. Hey, if we looked at the physics of all of this, we would know it's amazing it works at all. How do you know if it is packet loss versus an impatient client? Easy, you would see two queries and only one reply at the client side pcap. That means either the client request made it to the server, or that the server reply didn't make it to the client. The drop could be caused by many things; firewall, router, cable noise, magic, etc.

Unexpected DNS Retransmission

Here's where we go off track. Wireshark's Expert system began flagging DNS retransmissions somewhere around 3.0. It's wonderful seeing how many times a DNS query or response had to be retransmitted. It is usually attributed to packet loss, but the whole reason you capture packets is to be sure.

I chose one of the domain names from the Wireshark Expert list, and added a filter to focus on it.

dns matches cloudlock

The matches operator invokes the Perl Compatible Regular Expression (regex) engine in Wireshark display filters. The filter above will give me all of the DNS packets that have the word "cloudlock" in them, no matter the position nor case.

In case you would now like to play along, here is a link to the pcap https://www.cloudshark.org/analysis/b20c121d39ee/ladder?order=&filter=dns.id+in+%7B0x71b7+0x1879%7D&endpoints=&label=info. I'm using Cloudshark, and have already filtered to the first two transaction IDs. Looking at the process in Ladderview makes it easier to follow.

The client makes two queries for www.cloudlock.com, A for IPv4 and AAAA for IPv6. That is normal when both stacks are bound. PS, did you know you were sending double the necessary DNS traffic when you only use a single stack? The first response is the CNAME record and IP address. The client now has everything needed to connect to the server. Instead of waiting to see if there will be a AAAA reply and use that address, the client makes a new AAAA query for the CNAME record and uses the same transaction ID. That's the new part. Apparently the client really wants to use IPv6, I'm cool with that. I have Comcast and I use IPv6 all day long. But by using the same transaction ID, it is being flagged as a retransmission.

How This Affects You

What does this mean to you? Wasted time with false positives. SIEM, IDS, and the like can interpret this as the same transaction ID with a different hostname in a short amount of time, and flag it as attempts to poison DNS cache. Or your log files could flag it as retransmissions the way Wireshark did, and now you chase down a packet loss problem that is not there. On the upside, this does not happen every time the same host name is queried. On the downside, there is no pattern for when it happens except it always happens the first time. There is no correlation to TTL of the CNAME record either.

Summary

• There should only be a single query for each host.

• Asking for a CNAME record using a different query type (A vs AAAA) will not necessarily get new results.

• This behavior is an interesting and inefficient choice by Apple, and goes against the RFC.

• When a dramatic increase of DNS retransmissions begin to appear in your pcaps or log files, check to see if it is due to the second AAAA query.

I hope you found this information valuable. Let me know if you have noticed this behavior by the Macs on your network.

Betty DuBois is the Chief Detective for Packet Detectives, an application and network performance consulting and training firm based in Atlanta, GA. She has been solving mysteries since 1997.

Experienced with a range of hardware and software packet capture solutions, she captures the right data, in the right place, and at the right time to find the real culprit.

Betty presents each year at SharkFest, the Wireshark Developer and User Conference, and is active in the Wireshark community.

Using packets to solve crimes against the network and applications is her passion. Teaching others to do the same is her calling.

Do you have a Packet mystery that you'd like Betty to solve? How about a team who needs training on how to catch the culprit themselves? Go to bettydubois.com for contact information. Your mystery will be solved in no time.