Recently, my attention was drawn to something Odd about our traceroutes - namely, that traceroute and ping to an intermediate host on a route could have
wildly different values.
This
really bothered me, once I was forced to think about it.
I had previously assumed (wrongly) that the unexpectedly high second hop RTTs (and similar subsequent) values across our service provider were due to low priority in processing ICMP/tracereoute packets (many routers treat these things as low priority, for various good reasons).
That was a good enough "explanation" that I'd not really thought beyond that (or, it hadn't bothered me enough to get properly intrigued).
And I hadn't done pings to those intermediate hosts, and compared them side-by-side.
Shame on me.
And sure, ping and traceroute by default use different protocols (until you do traceroute -I).
But that's not it either.
Maybe traceroute sends so many more packets at a time than a ping that you hit a rate limit (1 per 500ms is a rate limit on some routers)?
ping is ~1 per second;
traceroute fires out loads in groups of 3 spaced per hop (well, TTL increment) quite closely together.
That's not it either.
Maybe a firewall was breaking things?
But no, that makes no sense; both in this case are ICMP Echo, and it's unlikely they're going to treat ICMP Echo to destination A differently to Destination B on the Internet.
I'm familiar with a bunch of other common pitfalls with interpreting traceroutes, but this wasn't one of those.
As someone who really likes networking, this should have prompted investigation long ago, but it's not bothered me enough to go work it out (aka "I had more pressing concerns").
Until someone said "Explain this" and presented a side-by-side ping and traceroute with Odd Results...
Then, of course, you start THINKING about the problem, and, if you're not familiar with the underlying configuration and particularly some potential configurations of service provider networks outside your own control will probably cause you to pull your hair out.
So why...?