When your firewall dies, and you need something, FAST.

For a number of years, I've been mentioning to colleagues, managers, interested members of staff and random strangers in the supermarket (well maybe not the last one) that I don't like single points of failure in enterprise ICT infrastructure.

I occasionally picture my network layout in my head, and think about the single points of failure with dread.

"One of these days, that single fancy firewall is going to die. And we're not going to be happy about it". 

Said firewall "died" last week...

Caching servers are a great idea with new IOS releases...

Would you like to save hundreds of gigabytes of Internet traffic?

That's some serious caching...

With the recent release of IOS 10, it's clear our users have been chomping through a fair few megabytes of data... Hopefully, we don't see too many people visiting us with "early adopter flu" in the next few days...

http://www.techradar.com/how-to/phone-and-communications/mobile-phones/ios-10-problems-here-s-how-to-fix-the-most-common-issues-1328553

Netflix, have you heard about RFC 952?

Having had a firewall melt-down last week (the subject of another post, when I get around to it), I rebuilt our entire firewall ruleset from scratch, because the config backups simply brought back the undesired effects (clearly something in the 45,000 lines of config disagreed with it...).

As a result, I've ended up "experimenting" with some options, with sometimes unintended, or non-obvious effects.

Netflix. It's broken. Oh noes!

One early casualty was Netflix - one of the few video streaming things we allow (because: bandwidth [available on plentiful and cheap national traffic] and legality [has a legitimate presence in South Africa]). Children love copyright infringing content; I try to make it easy to access legal stuff instead.

Synology RackStation RS3614xs+

Some time back, we received two Synology RS3614xs+ units so that we could address our rather painful lack of on campus storage (my Thruk monitoring system has a lot of machines that complain about lack of disk space cluttering up the place; most typically are over 90% full, which is not ideal). Off campus storage, thanks to Google Apps for Education, is not exactly a problem any more.

Synology RS3614xs+

We chose these after carefully surveying the low-mid range NAS/SAN market. They had a number of features which appealed, including dynamic flash/SSD caching, AD integration, integration with Google Drive, and 10 gigabit network interfaces; they also feature redundant power supplies and the possibility of adding more storage through expansion chassis. I'm particularly excited about having them set up as a redundant pair. The price/performance seemed about the best we were going to get. And it never hurts when a fellow sysadmin has had similar units in production for a while, and rather likes them.

One thing I'm particularly excited to see is faster student logins, one shared profiles end up cached on SSDs and spewed into the LAN at multi-gigabit speeds.

So, it's probably about time we configured these and put them into production...

OMG. I can't Teach, the Internet is down...!

Earlier in the week, we had our first major/prolonged outage since I've been here. A contractor (somewhat ironically installing a route for a second fibre to prevent this sort of thing from being an issue) drilled through the only fibre cable serving one of our biggest teaching blocks; 12 classrooms and a (deprecated) computer lab - and taking out Afrikaans, Geography, History and about 1/6th of the English department. It also reveals that the school makes "production" use of some very popular unofficial communications channels, particularly WhatsApp.

In a sort of "is this a hidden camera show?" scenario, it unfolded a bit like this:

Despite starting drilling a good 10cm away from the existing fibre, I pointed out that the drill bit was not straight and was pointing upward (i.e toward the existing fibre). This was not considered to be a problem, perhaps because geometry/trig is irrelevant once you leave school.

Next, I pointed out (from the other side of the wall, in between drilling "sessions", through a window) that the cable had started to move/wobble/vibrate a bit. Then a bit later, alarmingly so - so if they weren't pushing on the exposed cable by accident (I can't see through brick - yet), suggesting that they were almost certainly hitting the existing cable with the drill bit.

Drilling continued.

Then I announced that the link light had gone out on the switch.

*facepalm*.

One day, Apple might actually work in Education.

On Wednesday last week, I had some fruitful (pun not originally intended, but greatly amusing with hindsight) discussion with some people who resell Apple goods and services and various MDM solutions in South Africa, and later on, attended a presentation session on "Apple Classroom" and other goodies Apple is dangling like so much ripe fruit - often just out of reach. I also spend a while talking to a teacher and some techies from a school in another town - we share many of the same challenges, and it's often helpful to hear what others are doing. They have particular challenges in scaling management of their 300 iPads.

Also, there was a little time for networking with fellow techies at other schools around here about some of the challenges we face. The IT division at local university, Rhodes, used to organise "techie chats" - they were always rather valuable as getting together most of the Clue in town teaches you a lot, very quickly - and particularly the Rhodes people, by virtue of their membership of various management boards knew what was coming from the Internet (through the NREN, TENET, who is basically our ISP via the Albany Schools Network) some time in advance - and would share that info, when it wasn't embargoed. Perhaps someone ought to carry on that tradition...

Apple Classroom app icon

I was particularly pleased to see that the presenter took pains to note specifically which features were (kind of) available in South Africa, and which were definitely not - quite a change from Apple's own marketing, and it inspires a fair degree of confidence in their ethics and company. I've previously had the same person show me an MDM solution (which we spent time going through and quickly realised wasn't quite going to help us in our complicated free-for-all BYOD scenario) - and they were quite happy to discuss and show me the limitations of that software - all too rare in tech sales in many companies.

Teachers may find Apple's "Getting Started with Classroom" useful - but bear in mind not all features are necessarily available to you.

Here are most of the features of relevance to education in IOS 9.3 that were discussed in the presentation....

Juniper EX4600 switches - first impressions

We recently* took delivery of two Juniper EX4600 switches to ultimately replace our rather decrepit old Nortel Passport 1624G collapsed core/distribution switches. When they don't even bother to consistently relay DHCP messages, it's time to retire them (I had a hack involving a Mikrotik routerboard - the "swiss army knife" of the under-resourced network architect - in place to patch the DHCP issue, but I didn't like it being so). That should give you an idea of what refresh cycles around here are - the Nortels were even second hand when the school acquired them!

Juniper EX4600 with two optional modules installed

This means that we ought to expect such high end L3 switches or core routers to remain in service for 8-10 years, so we need to ensure that when we buy something like the core of the network, it's going to keep pace as much as possible with likely developments over then next decade. "Predicting the future" in tech is hard (and often futile), but networking doesn't often go through sudden massive changes (networking tech inertia is high - see how we're mostly still using IPv4?). That means you're going to want to install units that support your long term vision of the network for a long time, and will likely ensure you'll be able to deploy the technologies you're likely to need (particularly faster backbones; IPv6; adequate resilience as the school network moves from "sometimes, email is useful" to "OMG, I can't teach because the internet is down"). It also means we're probably ultimately going to lose them to old age "bathtub curve" failure, rather than a formal refresh cycle...

Enabling VLAN assignment on Ubiquiti UniFi-IW APs

Whilst I've had a fair number of fairly serious headaches when it comes to the deployment of Ubiquiti's UniFi wireless system since term began, sometimes, progress is made, and features they've long promised start to materialise.

They recently released the official new V5.x line of controller software, and an update to the firmware of the Cloud Key controller, to v0.5.0. After waiting a few days and hearing not much wailing and gnashing of teeth on the UNBT forums, I took the plunge to upgrade this morning, which didn't go smoothly (a tale for another time and place). Generally, the AP firmware is full of holes (800+ posts on a thread is not a good sign), so you sort of have to live at or uncomfortably close to the bleeding edge to keep your "customers" happy with these products. Or, you know, pick another platform (at far higher cost).

Anyway...

One of the things this new v5.x line of controller software does is properly enable VLAN control/assignment of/to the front ethernet ports on the UniFi UAP-IW, which is a neat little gadget that combines a basic enterprise 2.4GHz wireless AP with two wired Ethernet sockets, one of which features passthrough PoE. This makes it ideal for either very high density deployments (like in hotel rooms, and in our context, boarding houses) or in "edge" areas you're trying to serve at relatively low cost, but in a fairly feature-rich way.

Ubiquiti UniFi AP IW. Picture from https://www.ubnt.com/unifi/unifi-ap-wall/

These gadgets will enable you to provide, for example, a boarding school house person with a 2.4GHz wireless connection, a PoE powered VOIP phone and a network connection for a wired ethernet device. About the only downsides are a) they're limited to 100Mb/s, and b) they're deep enough - once you've got an ethernet flylead plugged in the back - not to really fit into standard wall boxes (at least around these parts) and c) no 5GHz radios. If it's mounted into an actual wall, you could probably hollow out a little more masonry at the back and have a great time installing loads of them, or you might add another "sticking out of the wall" box over a sunk into the wall box. And you can probably live without 5GHz in the odd spot, and for a few users, 100Mb/s is enough.

The other downside is that 50% of the devices of this type I've seen totally kill UniFi wireless AP networks (when your sample size is n=2, and one of them is really buggered, that's not a particularly surprising percentage!) - that sort of misbehaviour seems to be quite unusual.

Now, with the latest software, they actually allow you to control the VLAN assignment per port, and not just use whatever untagged VLAN you throw at the input port at the back. In other words, they become actually useful in an enterprise network, and will finally fulfill what I ordered them for (the house person exercise above).

Of course, this isn't well documented (or perhaps, they've done silly things like bury it in a PDF manual, but I can't find it). So, let's WABM it...!

The day Google broke email signatures...

We are having an interesting time (since yesterday morning) with email signature images in Gmail at the moment.

It's a minor thing, but it reflects poorly on an organisation's digital corporate image (and gets marketing people really hot under the collar).

Google seems to have changed something, somewhere about how signature images in Gmail work, at least for some users/organisations.

Apparently, this affects "a small number" of their clients, according to their tech support (presumably, they change a thing, roll it out to some of their clients and servers and see what happens... and we're in the guinea pig bunch this time).

What you see, if this affects you, is "broken image" icons in emails received by clients and in your own sent items (and existing email conversations) instead of your signature image.

Neither of the work-arounds their tech support suggested... worked.

But I seem to have figured out what *does* work...

Dear Apple. Please stop pretending you're the answer to all of our problems.

This post was prompted by a brief discussion with a teacher acquaintance of mine who has 300-odd iPads at their school, which they have to manage because their IT staff are too busy, and by my observations of the past year or so of iPads and IOS around here, and a lot of time "thinking" about BYOD/1:1. 

You know what irritates sysadmins more than almost anything else?

Vapourware.

What's almost even more annoying?

UsersWhen people with the power to force purchasing decisions (but who lack "tech clue") buy into it. This is a common problem throughout the IT world.

But this post is going to focus on the first of those inter-related problems, because it's one that's quite easy to fix in this example. If you're Apple.

Simplified 802.1x roll-out for windows clients: The JANET SU1X utility

I'm clearly not the first sysadmin to find setting up windows clients a pain; the UK university network consortium, JANET, has supported the development of a freely available setup utility, SU1X.

You can obtain the files through the JANET site, https://community.jisc.ac.uk/library/janet-services-documentation/su1x-8021x-configuration-deployment-tool or from GitHub on https://github.com/GarethAyres/SU1X (Edit: They seem to have moved it to SourceForge; see updated info)

Apple Cache Server update

In a previous post, we went through the steps I took to install a working Apple cache server.

It looks like our Apple cache server is earning its keep:

Look at all that nice, healthy, bandwidth-saving green!

FreeRADIUS - production SSL certificates

In the previous post, we covered the basic setup of FreeRADIUS.

In this post, we're going to focus on getting the SSL certificates right, and meet some of the common client snafus and their work-arounds (aka "hello Microsoft, please stop sucking at enterprise WiFi").

FreeRADIUS installation and configuration

I eventually abandoned shelved getting a working PacketFence installation (the learning curve and my time availability were not friends); I'll probably go back to setting that up (in the very least so there is a working config example), but I needed a production ready system, fast.

So, now that I've "simplified" to a working "just RADIUS" environment, I should be able to "complicate" it with PacketFence later on (and probably will do - electronic device registration deeply appeals to me on a "proper process" level, and helps with nonsense like RICA, although properly configured RADIUS logging might just obviate that need).

The stuff I learned pounding my head against PacketFence helped, but it wasn't the whole story...

Apple caching server

With the ever growing number of Apple devices [particularly iPads and iPhones] on our campus and (as far as I can see) no local Apple servers/CDN facility in the country, it makes a lot of sense for us to put in a cache.