In Part 1, I showed how I used AI agents to answer a question I had about the tool Wireshark. As read through the answer the three AI agents gave me to the same question, I realized I needed to iterate and refine my question. So, I asked each of the AI agents this iteration of the original question:
Prompt:
“Why are ip addresses sometimes displayed in reverse order in Wireshark’s ‘info’ column and sometimes not?”
In order to emphasize the idea of sometimes, I added it twice to the prompt. It’s a signal to the AI agent to pay attention.
Here are each agent’s response:
Co-Pilot’s Response
Copilot
In Wireshark’s “Info” column, IP addresses are sometimes displayed in reverse order due to the way they are resolved and presented. Let me explain:
Address Resolution:
When Wireshark captures network traffic, it records raw IP addresses (both IPv4 and IPv6) from the packets.
These raw addresses are then resolveda to their corresponding hostnames (if available) using DNS (Domain Name System) lookups.
The resolved hostnames are displayed alongside the IP addresses in the “Info” column.
If a hostname is successfully resolved, Wireshark shows it in the format: hostname (IP address).
However, if the hostname cannot be resolved, Wireshark displays the IP address in reverse order (reversed octets) as a fallback.
For example:
Resolved: www.example.com (192.168.1.10)
Unresolved: 10.1.168.192
Configuration and Preferences:
To enable address resolution and display hostnames, follow these steps in Wireshark:
Enable Resolve Network Addresses:
Go to View → Name Resolution and ensure that Resolve Network Addresses is enabled.
Use an External Network Name Resolver:
In Preferences → Name Resolution, select Use an external network name resolver (this option is enabled by default).
This setting allows Wireshark to perform reverse DNS lookups to resolve IP addresses to hostnames.
Note that resolved addresses are not updated automatically after changing settings; you need to reopen the dialog to see newly available names1.
Display Filters and Address Order:
Sometimes, when applying display filters, the order of IP addresses may appear reversed due to the filter conditions.
For example, if you filter for ip.src==10.38.14.55, it will match packets where the source IP address is 10.38.14.55.
However, if you use a broader filter like ip.src==10.38.0.0/20, it will match both 10.38.14.55 and 10.38.1.200.
In such cases, the displayed IP addresses may seem reversed, but it’s based on the filter criteria2.
Troubleshooting DNS:
If you encounter DNS resolution issues, ensure that your DNS server is reachable and correctly configured.
Verify the DNS server settings in your system preferences or network configuration3. Remember that Wireshark’s display behavior depends on various factors, including address resolution, user preferences, and applied filters. The reverse order is often a fallback when hostname resolution fails or when specific filters are used.
Gemini’s Response
Wireshark’s “info” column might show IP addresses in reverse order due to a feature called reverse DNS lookups. Here’s the breakdown:
Standard Display: By default, Wireshark displays the raw IP address (e.g., 192.168.1.1). This is the numerical address assigned to a device on the network.
Reverse DNS Lookups: Wireshark can attempt to translate these IP addresses into hostnames (e.g., [invalid URL removed]). This makes it easier to identify the devices communicating.
Reverse Order in Display: When a hostname is available, Wireshark might show it in the “info” column. However, hostnames follow a hierarchical structure, written in reverse order compared to the IP address. For example, the hostname “[invalid URL removed]” translates to the IP address structure “com.google.192.168.0.1” (this wouldn’t be the actual IP for Google, but illustrates the format).
So, you might see:
192.168.1.1 (standard IP address display)
[invalid URL removed] (hostname after successful reverse DNS lookup)
Here’s why you might see both:
Disabled Lookups: Reverse DNS lookups can be disabled in Wireshark’s settings. This improves performance but keeps the raw IP addresses.
Failed Lookups: Sometimes, the DNS server might not have a record for the IP address, resulting in no hostname available and just the IP displayed.
In summary:
Standard display shows raw IP addresses.
With reverse DNS, hostnames are displayed, which might appear reversed due to their hierarchical structure.
Disabled lookups or failed lookups will result in just the IP address.
White Rabbit Neo Response
In Wireshark, the 'info' column typically displays the source and destination IP addresses of packets being captured. The order in which these addresses are displayed can vary depending on the direction of the packet.
In some cases, Wireshark may display the addresses in reverse order, indicating that the packet is in the opposite direction from what is expected. This can happen when:
The packet is a response (e.g., a reply to a request) and the addresses are swapped.
The packet is a request from a device on the local network to a device on the internet, and the addresses are displayed in the order of the local network first and then the internet.
The packet is a request from a device on the internet to a device on the local network, and the addresses are displayed in the order of the internet first and then the local network.
Wireshark’s ‘info’ column is designed to show the most relevant information for the packet, which can include the source and destination addresses. The order of the addresses is determined by the direction of the packet and the context of the conversation.
For example, if a packet is a request from the local network to the internet, Wireshark will display the local network address first and then the internet address, indicating that the packet is going from the local network to the internet. If the packet is a response, Wireshark will swap the addresses to show the response from the internet to the local network, again indicating the direction of the packet.
This behavior is not always consistent and can depend on the specific circumstances of the packet being captured. It is important to note that Wireshark’s ‘info’ column does not necessarily represent the actual order of the addresses in the packet, which can be determined by examining the packet details or using other tools.
Analysis of Responses
In general, the AI agents are distracted by “reverse DNS lookup,” (the processes of starting with an IP address and then resolving to the domain name…which is the reverse of what normally happens), versus displaying the IP address in reverse order, which was my actual question. “Distracted” may be charitible. “Confused” may be more to the point.
This action of confusing one concept for another indicates that AI is “not quite there yet.” These are some of the best AI agents out there with the most work being done on them being asked a question in a domain that’s heavily documented. They should have enough raw material to work with. But they don’t quite get it right.
Here’s how each answered.
Co-Pilot
Co-Pilot’s response is different than it’s previous response with more detail that sounds plausible. However, in checking Co-Pilot’s references for its answer, it’s as if the answers and the cited references don’t match. Frankly, I don’t see how it got from A (the source’s it cites) to B (the answer it gave me). The more I dig into Co-Pilot’s response, the less I trust it. Here are it’s sources: – Doc pages from www.wireshark.org – From ask.wireshark.org – From 101labs.net
Co-pilot is great on truthiness, but poor on actually showing its work. Frankly, this is odd because Co-Pilot is based on Chat GPT, arguably the most used AI agent on the planet right now. My assumption is that it actually derived its answer from other sources, no the ones it cited.
My trust is low because my time is valuable. I don’t want to waste my time going down rabbit holes that lead to know where because Co-Pilot pointed me there.
Gemini
Oh Gemini, I was rooting for you. But you also let me down when you said, “Wireshark’s “info” column might show IP addresses in reverse order due to a feature called reverse DNS lookups.” That’s not quite right and certainly not the whole story.
Gemini has an interesting feature where you can see the other drafts it generated before presenting you with its final draft. What’s interesting is that it got close to the answer in one of it’s initial drafts, but it did not present the answer in it’s final draft to me. The answer was:
“When you see an IP address in reverse order (e.g., com.example.1), that’s Wireshark displaying the result of a successful reverse DNS lookup. The order is reversed because it follows the domain name structure, where subdomains come before the top-level domain (TLD)”
Gemini also has an interesting feature when you can ask it to validate it’s answers. It will then add highlighted text. Brown highlighted text is text Google says is different from the original answer and cautions you to do more research. Green highlighted text is text Google says is similar the original text. In this instance, neither added any knowledge. I’m not quite sure why Gemini even bothers adding the brown highlighted text as it doesn’t add anything significant to the answer.
Gemini’s previous drafts and text it adds when clicking on “Double-check response.”
White Rabbit Neo
Honestly, I can’t verify the veracity of White Rabbit Neo’s response. It sounds plausible, but it doesn’t give me any citations. It feels more like truthiness more than anything. It’s answers could be true, or they could contain shades of truth. I simply don’t know.
What I find really interesting is that it contains a statement that it directly contradicts Co-Pilot’s statements. To quote:
It is important to note that Wireshark’s ‘info’ column does not necessarily represent the actual order of the addresses in the packet
Whereas Co-Pilot said:
However, if the hostname cannot be resolved, Wireshark displays the IP address in reverse order (reversed octets) as a fallback.
These seemingly contradictions sound like they’re echoes of online arguments that each of the AI agent ingested in it’s quest to find an answer, or possibly explanations from great technologists who aren’t great explainers, making the assumption that their audience has the same background they have. Or something else.
Either way, the more I dig into White Rabbit Neo’s answers, the less confidence I felt in its answers which is the same thing I felt about Co-Pilot and Gemini.
What’s the Real Answer?
It’s important to understand that I asked a question that I actually had. I wasn’t out to formally test the accuracy of these AI agents; I was out to get an answer… the truth, if you will…to a question that I was truly curious about. Let’s go back to what originally prompted the question: it was this result I saw in a common tool, Wireshark:
Specifically, I was curious about “PTR 247.1.168.192.in-addr.arpa OPT”. Knowing private address spaces, I recognized that IP address to actually be “192.168.1.247.” Why was it in reverse order?
The simple answer is this: Wireshark’s displaying a PTR record and that’s the format in which PTR records are stored as this blog post from Cloudflare explains “Whats a DNS PTR Record?”. It’s a quick read, by the long and short of it is this: A Records associate domain names with IP address while PTR Records do the opposite. They associate IP address with domain names. The format is simply in reverse order.
Mere mortals don’t have to worry about A Records and PTR Records. These records are the data that makes the infrastructure of the Internet work. I’m one part mere mortal and one part tech geek, so I was curious.
What are the Lessons Learned?
When I started out asking Co-Pilot this simple question, “Why are ip addresses displayed in reverse order in Wireshark’s ‘info’ column?” I thought I’d get the answer and move on with my day. After all, it’s a well known domain of knowledge with tons of documentation and about a tool that’s popular in the networking world. On Youtube there are roughly 19,500 videos with the word “Wireshark” on the title. That’s a lot of content for the AI agents to ingest.
Still, I did not expect that my question would lead me to compare answers from three AI agents created by some of the largest corporations on earth, Microsoft, Google, and Meta (White Rabbit Neo is a derivation from Meta’s ollama.) I also did not expect that the process of iterating on my question and comparing answers actually lowered my confidence in the answers these systems gave me.
To get all the value out of AI you can get, keep these things in mind:
Be prepared to iterate the question you’re asking. This means asking the question, reviewing the response, and asking the question with more specificity to get more of what you want. Repeat as necessary.
You cannot view AI agents as being fonts of truth. They aren’t. A more effective mental model is that you’re talking to an idiot savant. AI can out read you; it can out process you; but I still don’t think it can out think you. You have to be prepared to think to make sure it’s giving you correct answers. Also, you have to ask it very specific questions to get the answers you want need.
If you think AI is an oracle that gives you correct answers without doing your own thinking, you’re going to get bit when you rely on it for real world consequences and it lets you down because it gives you answers that has truthiness, but not the truth. Rely on truthiness and you’ll lose money, a job, your friends, your health, or worse if you rely on something that’s fundamentally incorrect.
AI models vary on their ability to tell you how they reached their conclusions, something they should all do. You should be able to say something to the effect of, “Tell me why you say the things you do.” This is one aspect of the concept of transparency/explainability in AI ethics. Take advantage of it when it’s available. Sadly, explainability has a long way to go even though Transparency/Explainability is a foundational principal in:
The Organization of Economic Cooperation and Development’s OECD’s AI Principals.
etc.
Gain confidence in what questions the AI agent can answer for you by testing it with questions to which you already know the answers before you rely on it for answers to real questions you may have. It pays to have a healthy skepticism before you give it your trust. It’s not human; witholding your trust is not going to hurt its feelings.
AI’s getting there and Chat GPT got there faster than most of us thought. But the fact is that AI is not there yet, with “there” being defined as human’s ability to trust the answers AI is giving us because they are true. To get value out of AI and not get burned, verify then trust. Even then stay vigilant.
