IT Services

How to get most out of traceroute (with FreeBSD specifics)

On this page:

The traceroute command (Windows traceroute implementation: tracert) tells you more than you think. Note regarding firewalls: FreeBSD's traceroute implementation uses UDP instead of ICMP by default. The default UDP port is 33434 and then default port + 1 to port + (max_ttl - first_ttl + 1) * number of probes at the destination host is used.

 

Recommended readings / sources

 

traceroute in practise

  • Make things less confusing by using only 1 probe (instead of the standard 3): traceroute -q 1
  • ICMPs are often rate limited. Running mtr might hit these limits.
  • ICMP response source = ingress interface by convention (even though RFC 1812 says to use the egress interface which would prevent traceroute from working properly).
  • Does a hop look like a /30? Do a whois for the other side to get more useful information.
  • Know the path the traffic should take.
  • Be aware of multiple possible paths and load balancing as traceroute results are likely affected.
    • Multiple hostnames at one hop, each with one or more timestamps indicate the presence of load balancers.
    • Asymmetric paths often start at network boundaries. Try to control asymmetric paths using your SRC address. Make tests for both ends of the /30 to identify asymmetric path issues. Figure out how SRC and /30 interact with routing protocols. Most routers default to SRC traceroute from the egress interface.
    • Equal-cost multi-path (ECMP) may make multiple potential paths show up for each hop. Compare TTL values. ECMP is most confusing when multiple unequal length paths are involved.
  • Traceroutes from both sides are best; use the real source and destination addresses; snippets of traceroutes with missing information are bad; using IP addresses from a middle hop of another traceroute often results in wrong conclusions.
  • Traceroute shows the forward path only, the reverse path is completely invisible.
  • Save "known good traceroutes" to a file for later comparisons.
 

Problems traceroute might reveal

  • Long delay between two hops? There might be an ocean between the hops. Try traceroute to an organisation on a different continent (e.g. a university).
  • Asymmetric routing is a reason to disregard high round-trip times (RTT) at a single hop: traceroutes to the destination might have taken a different route than the way back. Traceroute does not show the hops on the way back.
  • Check DNS if traceroute is very slow. Deactivate address to name resolution for each hop: traceroute -n
  • traceroute might reveal a DNS problem as the first line of output shows the target's IP and hostname.
  • An asterisk indicates a lost packet: device not reached, device did not respond, or response did not make it back. This could mean asymmetric routing or traffic used for traceroute (UDP, ICMP) is filtered.
  • ICMP inside of MPLS-LSP makes traceroutes look really weird: returned ICMP packets must travel to the end of the LSP before going back to the sender, this makes every hop in the LSP appear to have the same RTT as the final hop.
 

How to interpret traceroute information

Traceroute output may indicate: geographic location, interface types and capacities, router type and its role, network boundaries and relationships, device port numbers.

  • Most implementations send 3 probes --> 3 latency measurements per hop, completey independent from each other; thus three asterisks * * * if there's no response (or a filter in place).
  • Traceroute errors:
    • !H (host unreachable), !N (network unreachable), !A, !X, !Z (traffic administratively prohibited).
    • See bottom of FreeBSD's traceroute man page for more.
 

What might affect round-trip time?

  • Use timestamps to interpret traceroute output correctly. Note the clock time as timing is important in diagnosing network issues.
  • Latency is not only related to the way to the destination an back. It also includes processing at the destination.
    • Serialisation delay - caused by encoding data as packets.
    • Queuing delay.
    • Propagation delay - on the wire.
    • Buffer bloat.
    • In case of ICMP: time taken to generate an ICMP Type 11 Code 0 (Time Exceeded) response. NOTE: a router may be configured not to send the ICMP or generate the ICMP with a very low priority - this is not a network constraint.
    • Backbone ISPs treat traceroutes and pings with low priority, thus there is usually an increased response time at these hops.
  • Utilisation: a 10G interface sends either 10G or nothing at any given point in time. Observing this over time is utilisation. Noticing problems usually starts at 95% utilisation.
  • From a router's perspective (data plane vs control plane): it takes more effort to respond to traffic addressed to the router itself than just forwarding traffic --> low prio for traceroutes and pings.
    • A router is fast for tasks ASICs are made for.
    • There is no ASIC for ICMP, the often underpowered standard CPU has to generate the ICMP message.
    • Processing BGP or OSPF is highly optimised.
 

How to make FreeBSD use ICMP insted of UDP for traceroute?

# FreeBSD ICMPv6 traceroute6
traceroute6 -I example.com
# FreeBSD ICMPv4 traceroute
traceroute -P ICMP example.com

# bonus: all routers on the local IPv6 network vtnet0 is connected to
ping6 ff02::2%vtnet0

Make a payment