darrenqu.net

Observability

tshark and tcpdump: A Working Guide for Network Engineers

1120 words 6 min read

packet-capture

Capture filters versus display filters, the field extraction that turns a capture into data, and which of the two tools to reach for.

on this page

Every network problem eventually reduces to the same question: is the packet actually arriving, and does it look the way I think it looks. Counters, logs and device state are all summaries of that, and summaries are where wrong assumptions hide.

tcpdump and tshark answer it directly. This is the working subset I use, and the distinction between them that took me longest to internalise.

The distinction that matters most#

Capture filters and display filters are different languages, applied at different times.

A capture filter (BPF syntax, -f in tshark, bare arguments in tcpdump) is evaluated in the kernel before the packet is written. Packets it rejects are never stored — they cost almost nothing and they are gone forever.

A display filter (Wireshark syntax, -Y in tshark) is evaluated after full dissection. It can reach any field of any decoded protocol, but the packet must already have been captured and parsed.

tcpdump -i eth0 'tcp port 443 and host 192.0.2.10'
tshark -r capture.pcap -Y 'http.response.code >= 400'

The syntaxes look similar enough to confuse and are not interchangeable. tcp port 443 is a capture filter; tcp.port == 443 is a display filter. Using one where the other is expected produces a syntax error at best and a silently wrong filter at worst.

When to use which:

  • Capture filter on a busy link, or for a long capture. This is how you avoid filling a disk and dropping packets. You cannot recover what you filtered out, so filter broadly.
  • Display filter when analysing. Capture wide, narrow afterwards, iterate as your hypothesis changes.

The failure mode worth avoiding: an over-tight capture filter that excludes the very packets that would have explained the problem. Capture more than you think you need — you can always narrow later, and you cannot widen.

tcpdump: capture#

tcpdump -i eth0 -nn -s 0 -w capture.pcap 'host 192.0.2.10'

The flags that matter:

FlagWhat it does
-i eth0Interface. -i any captures on all, with a different link-layer header
-nnNo DNS or port-name resolution — faster, and no surprise lookups from a box mid-incident
-s 0Full packet. Modern defaults do this, older ones truncate to a snaplen and give you useless payloads
-w file.pcapWrite raw packets. Always do this rather than reading on screen
-c 1000Stop after N packets
-G 300 -W 12Rotate every 300s, keep 12 files — a ring buffer for intermittent faults
-eShow link-layer headers, including MAC addresses
-v / -vvMore detail when printing to screen

-w then analyse is the habit worth building. Reading packets on the terminal as they fly past is how you miss the one that mattered.

For an intermittent problem, the rotating ring buffer is the tool:

tcpdump -i eth0 -nn -s 0 -G 300 -W 12 -w 'trace-%Y%m%d-%H%M%S.pcap'

An hour of history in twelve files, bounded disk usage, running until the fault reproduces.

Useful filter expressions:

host 192.0.2.10                       # to or from
net 192.0.2.0/24
port 443
portrange 8000-8080
src host 192.0.2.10 and dst port 53
tcp[tcpflags] & (tcp-syn|tcp-fin) != 0    # connection setup and teardown only
vlan 100                                   # tagged traffic
icmp6 and ip6[40] == 134                   # IPv6 Router Advertisements
not port 22                                # exclude your own SSH session

That last one is not optional when capturing on a box you are connected to. Without it, your capture of your capture of your capture fills the file.

tshark: analyse#

tshark is Wireshark’s dissection engine without the GUI. It captures too, but its value is reading.

tshark -r capture.pcap -Y 'tcp.analysis.retransmission'

Display filters worth having memorised:

tcp.analysis.retransmission            # retransmits
tcp.analysis.zero_window               # receiver out of buffer
tcp.flags.reset == 1                   # connection resets
http.response.code >= 400
dns.flags.rcode != 0                   # failed lookups
icmp.type == 3                         # destination unreachable
udp.port == 4791                       # RoCEv2
dhcpv6.msgtype == 2                    # DHCPv6 ADVERTISE — a server replying
icmpv6.type == 134                     # Router Advertisements
frame.time_delta > 1                   # gaps in the conversation

Extracting fields is where tshark earns its place#

This is the capability tcpdump does not have, and the reason to learn tshark properly:

tshark -r capture.pcap -T fields \
  -e frame.time_relative -e ip.src -e ip.dst -e tcp.dstport -e tcp.len \
  -E header=y -E separator=,

That is CSV. Any dissected field of any decoded protocol, on stdout, ready for sort, awk, a spreadsheet, or an Elasticsearch index.

Once you can do this, a capture stops being something you scroll through and becomes data you can ask questions of:

tshark -r capture.pcap -T fields -e ip.src | sort | uniq -c | sort -rn | head
tshark -r capture.pcap -Y 'dhcpv6.msgtype == 2' -T fields -e eth.src | sort -u

That second one is a rogue-DHCPv6 hunt in a single line, and it is exactly the kind of question that is tedious in a GUI and trivial here.

Built-in statistics are worth knowing before writing your own:

tshark -r capture.pcap -q -z conv,tcp        # TCP conversations
tshark -r capture.pcap -q -z io,stat,1       # throughput per second
tshark -r capture.pcap -q -z endpoints,ip    # per-address totals
tshark -r capture.pcap -q -z expert          # everything the dissector flagged

-z expert first, always. It surfaces retransmissions, malformed packets, checksum errors and protocol violations that the dissector already noticed, and it takes one second. Starting anywhere else is doing the tool’s work by hand.

Which one#

tcpdumptshark
Installed everywhereUsuallyOften not
FootprintTinyBrings the Wireshark stack
Protocol decodingShallowEverything Wireshark knows
Field extractionNoYes — the main reason to use it
StatisticsNoBuilt in
Capture on a busy linkBetterHeavier

In practice: capture with tcpdump, analyse with tshark. tcpdump is on the device already, it is light enough to run on a loaded box, and it writes a pcap. Move the pcap somewhere with tshark and do the thinking there. On appliances and switches you often have no choice anyway — tcpdump or a vendor wrapper around it is all there is.

What I would tell someone starting#

Learn which filter language you are in. Nearly every wasted hour with these tools starts with a display filter in a capture position or the reverse.

Always -w to a file. Terminal output is for confirming the capture is running, not for analysis.

-nn always. Name resolution is slow, adds its own traffic, and can hang on a box whose problem is that DNS is broken.

Exclude your own session. not port 22.

Start with -z expert. The dissector has already found the obvious problems. Read its findings before forming a theory.

Capture wider than you think you need. Disk is cheap; a reproduction you cannot get back is not.


References

← more in Observability