Tuesday, 2 July 2019

One year on, in a different kind of school...

A year ago (to the day), I started what I would seriously consider a "dream job title" - that of Network Architect at a University (at the first place in sub-Saharan Africa to have an Internet connection - so long ago that its first IP address allocation was done within an RFC!). It ticked all my boxes in terms of favourite things to do, and promised to throw me in at the deep end in a much more challenging environment. I didn't have any particular desire to leave my last position (which I enjoyed immensely), but when I got two phone calls some time apart strongly suggesting I ought to apply for the position, and with an interesting list of "things we need done", after much hand-wringing, I jumped at the chance, and made it through the selection process. #GreatSuccess.


Securing Internet Routing: RPKI OV and ROAs

For some time now, I've had a ticket in my queue to "Investigate RPKI". A few weeks ago, we experienced some strange internet outages that turned out to be because not all is well with RPKI Origin Validation at one of our upstream ISPs...

Friday, 10 May 2019

On the State of Firewalls: are NGFWs (becoming) obsolete?

Between the last blog post and this one, I’ve moved from K-12 into Higher Education, at the first place in Sub-Saharan Africa to have Internet connectivity. This is a vastly different environment in some ways – in particular, firewalling is quite different. You’re dealing with a user population that is entirely adults. Some of those adults engage in legitimate research on things that some would consider a bad idea (malware) or “morally dubious” (porn, pop-up ads, etc.), or needs unfiltered traffic (network telescopes, honeypots, big data “science DMZs” ). The particular University I work at has generally had a liberal outlook with regards to personal freedoms (and concomitant responsibility) – I think that’s generally a good thing and exactly where higher education should be.

We’re currently looking at doing a hardware refresh of our ~7 year old enterprise firewalls – mainly because the support on the current solutions is eye-watering. The present solution works fine (although it has quite limited capacity for logging – about 8 hours of our traffic), and it’s approaching vendor EoL status. Interestingly, even moving to a newer (and, because Moore’s law, more performant) hardware platform from the same vendor saves us money over a number of years. So we’re thinking about what we need, and that’s prompted some musings about the state of firewalls…

Monday, 4 June 2018

On PABXs...

We eventually decided that our existing PABX solution no longer met requirements. In particular, it was very opaque, and extremely expensive, with a licensing model (and vendor locked-in handset requirements) that were punishingly expensive.

We considered "DIY" with something from Yealink (most likely an S300), with an interim set of BRI and POTs links to move away from Avaya onto another platform, before investigating SIP trunks for the "uplink", but we looked at our work schedules (and the offers that came in) and we ultimately decided to get a VOIP telecommunications company to help us install it, implementing SIP from launch.

Here are a few things we learned along the way...

Thursday, 26 April 2018

GSuite mail gateway using Ubuntu and Postfix

Whilst a lot of vendors will tell you they "support Gmail", it turns out the level of support can be... iffy.

In the end, it is often easiest if you create an email "gateway" that leverages the likelihood of just about everything on your campus being able to throw SMTP at port 25 (even though that's deprecated for mail submission), alongside GSuite for Education's SMTP email relay functionality.

This is particularly important once you start getting serious about email security, and that email from your domain MUST flow through particular email servers (because of SPF, DKIM & DMARC).

Read on for more!

Tuesday, 20 February 2018

MPLS causes some weird effects - aka Why is traceroute so much slower than ping for some hops?

Recently, my attention was drawn to something Odd about our traceroutes - namely, that traceroute and ping to an intermediate host on a route could have wildly different values.

This really bothered me, once I was forced to think about it.

I had previously assumed (wrongly) that the unexpectedly high second hop RTTs  (and similar subsequent) values across our service provider were due to low priority in processing ICMP/tracereoute packets (many routers treat these things as low priority, for various good reasons).
That was a good enough "explanation" that I'd not really thought beyond that (or, it hadn't bothered me enough to get properly intrigued).
And I hadn't done pings to those intermediate hosts, and compared them side-by-side.
Shame on me.

And sure, ping and traceroute by default use different protocols (until you do traceroute -I).
But that's not it either.

Maybe traceroute sends so many more packets at a time than a ping that you hit a rate limit (1 per 500ms is a rate limit on some routers)?  ping is ~1 per second; traceroute fires out loads in groups of 3 spaced per hop (well, TTL increment) quite closely together.
That's not it either.

Maybe a firewall was breaking things?
But no, that makes no sense; both in this case are ICMP Echo, and it's unlikely they're going to treat ICMP Echo to destination A differently to Destination B on the Internet.

I'm familiar with a bunch of other common pitfalls with interpreting traceroutes, but this wasn't one of those.

As someone who really likes networking, this should have prompted investigation long ago, but it's not bothered me enough to go work it out (aka "I had more pressing concerns").

Until someone said "Explain this" and presented a side-by-side ping and traceroute with Odd Results...

Then, of course, you start THINKING about the problem, and, if you're not familiar with the underlying configuration and particularly some potential configurations of service provider networks outside your own control will probably cause you to pull your hair out.

So why...?

Friday, 16 February 2018

Chromebooks? Yes Please.

We've started seeing more and more Chromebooks.

To those in education overseas, they're not exactly news, but they have recently become (slightly) less unusual in South Africa, and are (intermittently) available from local suppliers. With the advent of Android-compatible models, we can now use them across all of our "core" software.

So far, I've been very pleased with them from a sysadmin point of view.
Acer R11 C738T

Read on for more experiences....