Tuesday 20 February 2018

MPLS causes some weird effects - aka Why is traceroute so much slower than ping for some hops?

Recently, my attention was drawn to something Odd about our traceroutes - namely, that traceroute and ping to an intermediate host on a route could have wildly different values.

This really bothered me, once I was forced to think about it.

I had previously assumed (wrongly) that the unexpectedly high second hop RTTs  (and similar subsequent) values across our service provider were due to low priority in processing ICMP/tracereoute packets (many routers treat these things as low priority, for various good reasons).
That was a good enough "explanation" that I'd not really thought beyond that (or, it hadn't bothered me enough to get properly intrigued).
And I hadn't done pings to those intermediate hosts, and compared them side-by-side.
Shame on me.

And sure, ping and traceroute by default use different protocols (until you do traceroute -I).
But that's not it either.

Maybe traceroute sends so many more packets at a time than a ping that you hit a rate limit (1 per 500ms is a rate limit on some routers)?  ping is ~1 per second; traceroute fires out loads in groups of 3 spaced per hop (well, TTL increment) quite closely together.
That's not it either.

Maybe a firewall was breaking things?
But no, that makes no sense; both in this case are ICMP Echo, and it's unlikely they're going to treat ICMP Echo to destination A differently to Destination B on the Internet.

I'm familiar with a bunch of other common pitfalls with interpreting traceroutes, but this wasn't one of those.

As someone who really likes networking, this should have prompted investigation long ago, but it's not bothered me enough to go work it out (aka "I had more pressing concerns").

Until someone said "Explain this" and presented a side-by-side ping and traceroute with Odd Results...

Then, of course, you start THINKING about the problem, and, if you're not familiar with the underlying configuration and particularly some potential configurations of service provider networks outside your own control will probably cause you to pull your hair out.

So why...?

Friday 16 February 2018

Chromebooks? Yes Please.

We've started seeing more and more Chromebooks.

To those in education overseas, they're not exactly news, but they have recently become (slightly) less unusual in South Africa, and are (intermittently) available from local suppliers. With the advent of Android-compatible models, we can now use them across all of our "core" software.

So far, I've been very pleased with them from a sysadmin point of view.
Acer R11 C738T

Read on for more experiences....

Tuesday 6 February 2018

The joys of bash - some light scripting for n00bs

There's a lot to be said for scripting in a sysadmin's life - indeed, if you don't do any scripting, are you even a sysadmin...?

I've slowly been learning bits of bash and various related Unix utilities that are useful for processing text files - like the copious log files FreeRADIUS spits out with all sorts of useful information. I like "just in time" learning - it's often the only learning I have time for...
A screenshot of a bash shell script.
In particular, I wanted to know whether certain users are abusing the system, by connecting far too many devices, and to have a count of unique devices running on our LAN for a measure of the popularity of our wireless system (and perhaps for some capacity planning).  We limit device numbers for various reasons, and devices must be registered (on paper) prior to connection - there are only n spaces on the form, so n+1 or more == breaking the rules. Sure, electronic NAC is the obvious next step, and re-investigating Packetfence remains on my list of things to do...

Looking through currently connected clients where more than n clients are simultaneously connected is a fool's errand, even if your wireless system has such a view (UniFi has one) - and it won't catch people who break the rules over the course of a day, or when you're not actively looking, or across two different manufacturers of Access Points. A 24 hour long RADIUS logfile record on the other hand... Well, that has a lot of potential.

Of course, processing of this kind is quite easily accomplished in a simple shell script...

Thursday 1 February 2018

Poorly Documented Feature: Canned Responses in Delegated Accounts (Gmail)

It's no secret. We love Gmail. However, sometimes, there are features that don't get used much that are absolute "killer" features - but it turns out they're not always well documented.

A common scenario is creating generic "role based" accounts that receive large volumes of mail, and to then delegate access to this mailbox to several individuals to deal with the responses.

Of course, a lot of incoming emails means a lot of outgoing responses (often the exact same thing hundreds of times), and there's a really handy feature, Canned Responses, in Labs that makes this a pleasure.
Labs Canned Responses.
Thank you Googler Chad P,
bulk email responders LOVE you.
However, if you switch across to a delegated account, shock, horror!
No Labs!
Does that mean no canned responses in delegated accounts? No it does not (phew)...