When your firewall dies, and you need something, FAST.

For a number of years, I've been mentioning to colleagues, managers, interested members of staff and random strangers in the supermarket (well maybe not the last one) that I don't like single points of failure in enterprise ICT infrastructure.

I occasionally picture my network layout in my head, and think about the single points of failure with dread.

"One of these days, that single fancy firewall is going to die. And we're not going to be happy about it". 

Said firewall "died" last week...

Caching servers are a great idea with new IOS releases...

Would you like to save hundreds of gigabytes of Internet traffic?

That's some serious caching...

With the recent release of IOS 10, it's clear our users have been chomping through a fair few megabytes of data... Hopefully, we don't see too many people visiting us with "early adopter flu" in the next few days...

http://www.techradar.com/how-to/phone-and-communications/mobile-phones/ios-10-problems-here-s-how-to-fix-the-most-common-issues-1328553

Netflix, have you heard about RFC 952?

Having had a firewall melt-down last week (the subject of another post, when I get around to it), I rebuilt our entire firewall ruleset from scratch, because the config backups simply brought back the undesired effects (clearly something in the 45,000 lines of config disagreed with it...).

As a result, I've ended up "experimenting" with some options, with sometimes unintended, or non-obvious effects.

Netflix. It's broken. Oh noes!

One early casualty was Netflix - one of the few video streaming things we allow (because: bandwidth [available on plentiful and cheap national traffic] and legality [has a legitimate presence in South Africa]). Children love copyright infringing content; I try to make it easy to access legal stuff instead.

Synology RackStation RS3614xs+

Some time back, we received two Synology RS3614xs+ units so that we could address our rather painful lack of on campus storage (my Thruk monitoring system has a lot of machines that complain about lack of disk space cluttering up the place; most typically are over 90% full, which is not ideal). Off campus storage, thanks to Google Apps for Education, is not exactly a problem any more.

Synology RS3614xs+

We chose these after carefully surveying the low-mid range NAS/SAN market. They had a number of features which appealed, including dynamic flash/SSD caching, AD integration, integration with Google Drive, and 10 gigabit network interfaces; they also feature redundant power supplies and the possibility of adding more storage through expansion chassis. I'm particularly excited about having them set up as a redundant pair. The price/performance seemed about the best we were going to get. And it never hurts when a fellow sysadmin has had similar units in production for a while, and rather likes them.

One thing I'm particularly excited to see is faster student logins, one shared profiles end up cached on SSDs and spewed into the LAN at multi-gigabit speeds.

So, it's probably about time we configured these and put them into production...

OMG. I can't Teach, the Internet is down...!

Earlier in the week, we had our first major/prolonged outage since I've been here. A contractor (somewhat ironically installing a route for a second fibre to prevent this sort of thing from being an issue) drilled through the only fibre cable serving one of our biggest teaching blocks; 12 classrooms and a (deprecated) computer lab - and taking out Afrikaans, Geography, History and about 1/6th of the English department. It also reveals that the school makes "production" use of some very popular unofficial communications channels, particularly WhatsApp.

In a sort of "is this a hidden camera show?" scenario, it unfolded a bit like this:

Despite starting drilling a good 10cm away from the existing fibre, I pointed out that the drill bit was not straight and was pointing upward (i.e toward the existing fibre). This was not considered to be a problem, perhaps because geometry/trig is irrelevant once you leave school.

Next, I pointed out (from the other side of the wall, in between drilling "sessions", through a window) that the cable had started to move/wobble/vibrate a bit. Then a bit later, alarmingly so - so if they weren't pushing on the exposed cable by accident (I can't see through brick - yet), suggesting that they were almost certainly hitting the existing cable with the drill bit.

Drilling continued.

Then I announced that the link light had gone out on the switch.

*facepalm*.

One day, Apple might actually work in Education.

On Wednesday last week, I had some fruitful (pun not originally intended, but greatly amusing with hindsight) discussion with some people who resell Apple goods and services and various MDM solutions in South Africa, and later on, attended a presentation session on "Apple Classroom" and other goodies Apple is dangling like so much ripe fruit - often just out of reach. I also spend a while talking to a teacher and some techies from a school in another town - we share many of the same challenges, and it's often helpful to hear what others are doing. They have particular challenges in scaling management of their 300 iPads.

Also, there was a little time for networking with fellow techies at other schools around here about some of the challenges we face. The IT division at local university, Rhodes, used to organise "techie chats" - they were always rather valuable as getting together most of the Clue in town teaches you a lot, very quickly - and particularly the Rhodes people, by virtue of their membership of various management boards knew what was coming from the Internet (through the NREN, TENET, who is basically our ISP via the Albany Schools Network) some time in advance - and would share that info, when it wasn't embargoed. Perhaps someone ought to carry on that tradition...

Apple Classroom app icon

I was particularly pleased to see that the presenter took pains to note specifically which features were (kind of) available in South Africa, and which were definitely not - quite a change from Apple's own marketing, and it inspires a fair degree of confidence in their ethics and company. I've previously had the same person show me an MDM solution (which we spent time going through and quickly realised wasn't quite going to help us in our complicated free-for-all BYOD scenario) - and they were quite happy to discuss and show me the limitations of that software - all too rare in tech sales in many companies.

Teachers may find Apple's "Getting Started with Classroom" useful - but bear in mind not all features are necessarily available to you.

Here are most of the features of relevance to education in IOS 9.3 that were discussed in the presentation....

Juniper EX4600 switches - first impressions

We recently* took delivery of two Juniper EX4600 switches to ultimately replace our rather decrepit old Nortel Passport 1624G collapsed core/distribution switches. When they don't even bother to consistently relay DHCP messages, it's time to retire them (I had a hack involving a Mikrotik routerboard - the "swiss army knife" of the under-resourced network architect - in place to patch the DHCP issue, but I didn't like it being so). That should give you an idea of what refresh cycles around here are - the Nortels were even second hand when the school acquired them!

Juniper EX4600 with two optional modules installed

This means that we ought to expect such high end L3 switches or core routers to remain in service for 8-10 years, so we need to ensure that when we buy something like the core of the network, it's going to keep pace as much as possible with likely developments over then next decade. "Predicting the future" in tech is hard (and often futile), but networking doesn't often go through sudden massive changes (networking tech inertia is high - see how we're mostly still using IPv4?). That means you're going to want to install units that support your long term vision of the network for a long time, and will likely ensure you'll be able to deploy the technologies you're likely to need (particularly faster backbones; IPv6; adequate resilience as the school network moves from "sometimes, email is useful" to "OMG, I can't teach because the internet is down"). It also means we're probably ultimately going to lose them to old age "bathtub curve" failure, rather than a formal refresh cycle...