Wireshark Reviews

It always gives me sense of satisfaction when I have a challenge and can leverage some knowledge to figure out.

Today I was in the lab and was powering on two Cisco switches when I noticed that they weren’t labeled with their IP addresses. I’m not sure why I did not label them, but now I have to pay for it.

For those of you who have not been in this situation before I will explain. My switches have a DB9 serial connection and of course good luck finding a computer with a serial port. So now I have to rummage through the box of wires to find the serial to USB adapter. I have had to buy a second one in 2 years since my original does not have a Windows 7 driver, but I digress. After I find the cable, I have to find the installation disk because last week I migrated to a new laptop…. I’m sure you get the picture.

On to plan B. I know the switches have IP addresses since I hard code IP addresses on all of my switches.

Now here’s where a bit of knowledge comes in. I know that when a device powers up and either obtains an IP addresses via DHCP/BOOTP or statically has an IP assigned it will send out a specific ARP called a gratuitous ARP.

Perfect, now all I have to do is make sure the switch port is connected to my subnet, start any protocol analyzer (I chose Wireshark) and power up the switches.

In this video I show you how to find the Gratuitous ARP quickly, create a display filter and lastly, locate the 2 switches’ IP addresses.

Enjoy
http://www.youtube.com/watch?v=EUmHdVeBBNc

NAT Packet Analysis Using Wireshark

One of the most popular questions I get when people get the hang of protocol analysis is the daunting exercise of multitrace analysis. As with anything else the best advice is to start with the basics before tackling anything complicated.

Multitrace analysis is only effective if you truly understand your vendors products, networking and how it relates to the OSI model or packet analysis. I always suggest that you start at layer 1 and work yourself up. The key is to know what fields in the frame or packet changes, or remains the same. Ideally when you figure this out you can use a better capture or display filter

A multitrace capture of a hub, switched, or bridged network is most straight forward since a hub or switch is transparent at layer 1 or 2 and doesn’t change anything in the packet.

When you move up to layer 3 or routing, several things change in the packet such as MAC address, IP TTL and TOS. Of course your mileage will vary, and any device could be configured to muck with more bits in the packet, but I figure I would give you a point of reference.

At layer 4 we get into application gateways, proxy, firewalls and NAT type devices where the following packet fields gets modified; MAC address, IP address, IP TOS, TCP/UDP port numbers, TCP ACK/SEQ values, etc.

Lastly at layer 7, we are dealing with multi-tiered applications and basically everything changes in the packet.

In this video example I do a multitrace analysis of a simple netgear router/NAT/firewall device where I take a trace from the WAN and LAN side to compare. Not to sound like a broken record, but please remember that your devices might behave totally differently and these notes and techniques should only be used as a reference in your environment.

http://www.youtube.com/watch?v=J9FzaFryQIw

Multitrace analysis can be the most interesting, rewarding and unfortunately, most frustrating exercise an analyst will face.

Before we get to the packet analysis, setting up your tools for simultaneous capturing can be a feat in itself.

The time issue is the most critical when using 2 devices since the time is used to calculate the delay, jitter or latency. Some people are fine with syncing both devices to a common ntp server.

Then there’s the “how the #!!$!@#!!” do I physically capture . This is where you have to be familiar with the problem, the network you are working on and what equipment is available to you. If you are lucky enough to be able to change the speed and duplex to 100 half duplex a good old hub fits the bill. Other than the mirror/span command, a tap is also very helpful. Trust me every one of these suggestions comes with their own caveats. You may have to try different tools for different scenarios.

For example, if I am doing a simple pc bootup/login baseline, I am interested in things like total data transferred, which IP’s I am talking to, protocols used, errors, etc. In this case speed and duplex is not important and I can go with a hub. But if I was troubleshooting why something is taking too long, like a backup or replication, changing the speed and duplex would not be a good idea.

If you are lucky enough and can capture from one device, the time accuracy issue goes away and life does get a bit easier. But now you have 2 different captures in the same trace, Yikes!!!! Not to mention that different network interfaces have different latency or behaviors. I remember trying a usb to 10/100 ethernet adapter to capture packets and quickly realized that this adapter added 30 ms to every packet. Again, if I was troubleshooting latency, this won’t do.

Lastly, if you’re fortunate enough, you might even have an application that takes multiple trace files and calculates all sorts of stuff out for you (hmm.. next article?).

In this example I use Wireshark, my laptops WiFi and Ethernet ports to capture my packet traversing a residential home router. I show some tips and tricks along the way and hope this will help you out.

http://www.youtube.com/watch?v=CAS_Kb4VYjo

Documenting a Problem With Wireshark

I remember talking to a group about the ‘superman syndrome’ where the analyst wants to swoop in and save the day. I explained that like most forensic tasks, protocol analysis can be tedius, confusing and downright boring at times. Alright who wants to capture some packets now!?

If you can’t see it, you can’t fix it. That is why I like to use protocol analysis to minimally document the problem that I’m experiencing. Even if the packets don’t show any anomalies, that worth knowing as well, isn’t it? If you do see an anomaly, you might not have the solution but at least you know what it looks like when its broken.

Ideally protocol analysis is most helpful when you have two traces to compare; the good and bad trace. In most realistic scenarios, the client will not have a good trace and just the current bad trace. I’m our classes I review how to make use of what you have.

In this example the customer had a DSL line with an issue and another DSL line what worked fine. The customer mentioned that whenever the DSL circuit ‘acted up’, they simply rebooted the modem. Both DSL circuits went to the same carrier, ordered at the same time, provisioned the same way and even use the same hardware. Perfect, example of something I can compare. I also noticed that these are not just modems, but they route, dhcp, firewall and NAT.

What I found, is that the problem circuit was having issues passing larger frames, while the other had no issues. After the reboot the problem circuit now behaves like the good one. Upon further investigsation I noticed the problem modem had older firmware and suggested they get that firmware updated.

So, even though I couldn’t ‘fix’ the problem, we know exactly what the problem is and what to look for if the problem returns.

http://www.youtube.com/watch?v=OBT5XGOA3EU

I am surprised that this exercise we do in class still proves to be helpful as well as quite popular.

There are many utilities out there to help find rogue servers, but why bother when you already have Wireshark installed. When you get comfortable with this exercise you can save some steps by creating a capture filter for just DHCP packets, or better yet, just DHCP server packets. As always with protocol analysis, there are many ways to do this exercise and this is just my preference since it forces me and the attendees to review the DHCP process as they go through the packets.

Rogue DHCP servers are becoming more common these days since a DHCP server can simply be a part of an application loaded on your computer. The introduction of tablets and smart phones that can provide hotspot support, are also DHCP servers. I even see more applications out there that turns your laptop into a WiFi hotspot so you can tether it to your tablet or smart phone.

Don’t worry, I haven’t forgotten the classic example of an employee who wants wireless access in a nearby conference room and simply connects the LAN port of his wireless router at his desk and starts dishing out IP addresses.

I like the added twist where I ask people to identify the legitimate DHCP based on paying attention to the story, not the packets. I can’t tell you how many times I figure out a problem by going back to the user and having a conversation rather than going over the trace a million times.

I think people forget that Wireshark and protocol analysis is an exercise in forensics and you need a story for context and to make sense of the packets.

I have said many times that many times the answer comes from the story, not the packets.

http://www.youtube.com/watch?v=uyvEa7Nh80A

While troubleshooting a Wifi performance issue on a large BYOD network, I was explaining to the customer a lot of people on a wireless network sending a lot of small packets can cause a performance issue by robbing precious time from other Wifi clients.

They didn’t quite understand how this could happen since many users’ computers and phones are idle and just simply connected to the WiFi network. I illustrated the impact of having common applications installed on a smartphone/tablet as well as browser extensions or add-ons would have on a network by using Wireshark.

The trickiest part of this exercise is actually capturing the Wireless packets. You can use Riverbed’s Airpcap adapter, or any other vendors WiFi packet capturing product. Just keep in mind that in many cases where you have encryption enabled, its easier if you join that network to see the packets.

To this day I am surprised how many network analysts lack WiFi troubleshooting tools and either rely on their wired lan tools or strictly use the vendors monitoring applications as their sole source of information. I remember a few years ago I did a tools presentation for a vendor and asked the group how much confidence they would have in their auto mechanic if he only had one tool on the bench, or if he lacked specialty tools for your specific car’s make and model.

With Wireshark I was able to give them an ‘under the hood’ view of their network. You don’t need to have an extensive protocol analysis background to quickly realize that this is one busy network. As I have many times in the past, “Packets don’t Lie”.

On a wired network this is less of an issue since a wired network is more bandwidth bound. On a wireless network at home this isn’t an issue either since you aren’t sharing the wireless network with as many people.

In this case, the customer had over 200 people on an access point which cumulatively creates an issue.

In this video I use Wireshark to illustrate the traffic generated by these various applications.

http://www.youtube.com/watch'v=xDuRhQ6swrI

Excellent packet analyzer tool. I have used this a lot and had very good luck with it, it is pretty easy to use and can provide a lot of information and insight when troubleshooting network issues.