A year ago (to the day), I started what I would seriously consider a "dream job title" - that of Network Architect at a University (at the first place in sub-Saharan Africa to have an Internet connection - so long ago that its first IP address allocation was done within an RFC!). It ticked all my boxes in terms of favourite things to do, and promised to throw me in at the deep end in a much more challenging environment. I didn't have any particular desire to leave my last position (which I enjoyed immensely), but when I got two phone calls some time apart strongly suggesting I ought to apply for the position, and with an interesting list of "things we need done", after much hand-wringing, I jumped at the chance, and made it through the selection process. #GreatSuccess.
I've had to learn a lot quite quickly, but I'm blessed to have people that know more than I do as colleagues (my line manager was the previous incumbent of my position, and knows a ridiculous amount about all IT stuff, and the team I supervise has combined decades of experience on this campus at lower "layers", so they literally know where the skeletons are buried)! There are useful technical books lying around the place you can just sit down and read. There are several sysadmins lurking around the corridors, and a functioning helpdesk that helpfully means I typically only have to deal with "hard" problems, and my phone rarely rings (although I get quite a lot of SMS alerts from our monitoring system!).
The position has very much lived up to my expectations - it's given me a much bigger, more complex environment to understand. It's more corporate (with the good and bad sides that brings). I have more toys to play with (and they're shiny). I get to work on pretty serious systems, and with some pretty complex topologies (simple enough to sketch on a napkin, but the details can bite you).
It's not all sunshine and roses, of course. Like most organisations, resources are not limitless, and the environment in Higher Education is particularly very constrained financially, and because of the nature of ICT purchasing and the impacts of foreign exchange fluctuations hit us hard (right in the shiny). That just means we have to be a little bit more creative in our solutions to problems, and compromise on some things (which I've done my whole working life). A key side effect of this is there is a lot more open source software around - and the problems are unique enough (or have been annoying sysadmins here for long enough) that there are custom tools and toolchains to do things - one tool we use continuously to manage IP addresses has been in use since at least 1995. Most things are very "Unix-y" (mostly FreeBSD), and so I've had to up my scripting game considerably - but there are enough people with clue around that interesting problems get fixed quite fast if you do something like put it in a ticket that explains the problem and possible solutions - with the intention of me sorting them out, but I do not look gift horses in the mouth - examining their dentition is a good way to learn, too, and I have people to ask questions of! It's been surprising how much more "silo-d" the organisation is, even just within the IT services division under a single Director - to some degree, it's a pretty classic Ops / Dev / Support split, complicated by how Universities work, and the degree of freedom research inevitably calls for in terms of ICT process and policy.
The team I manage is growing; it ranges from a facilities management person (who deals with things like attending building planning meetings and managing HVAC and the like) - layer 0, if you like; 3 network technicians of various grades (Layer 1, with some Layer 2 config), and ultimately two network engineering posts (Layer 2-3+), under the Network Architect. The Networks team, aside from the obvious wireless, edge, core distribution and access devices are also nominally responsible for "networking services" like DNS, DHCP, NTP, IPAM, perimeter firewalls, RADIUS and the like. Much beyond L3, it's onto the Ops sysadmins or AppDev/DMU people. And the Ops sysadmins mostly deal with all the servers that run the networked services within their normal maintenance, so that's one other task off my chest.
We had a fun six months (!) on-and off fighting a horrible bug in a large enterprise vendor's wireless controller software - we kept trying to update beyond a fairly ancient version, with catastrophic results every time (tossing almost all the APs and clients off the network every 5-15 minutes). I'm still amazed it took as long as it did for them to finally admit that it was a problem in their controller software, and then do something about it. Fortunately, you could just roll back to an older version and regain stability, but there were a lot of late nights on Tuesdays and long phone calls to the vendor's tech support during maintenance windows! Why did we want an update? Well, the "stable" version was pre-KRACK patches; everything that crashed was post KRACK patches! We are in an environment where people like to "play"...
There's quite a bit of training that needs to be done, as we're actively building the team (and people are being promoted into new areas they've got relatively little experience in). Specialised recruitment here is hard (it's a small town, and salaries at Universities don't compete that well with national, let alone international, trends in IT - but you get to do interesting things in an interesting place, often with a lot more freedom than in a similarly large "corporate" environment). I enjoy mentoring people, so we can certainly "grow our own", to some degree, but it's quite hard to find the time when you yourself need to fix things (or learn it first!).
Also, I think a lot of people that would do quite well in network engineering keep going across and becoming developers, thinking that's where all the action's at. Sorry, your app does nothing until the network, well, works! With the rise of SDN and infrastructure-as-a-service / network automation / DevNetOps, there's a lot of dev work to be done within networking teams these days.
I've finally, I think, gotten to grips with enough of the immediate macro- and micro- things on the campus that it's time to tackle some bigger projects, rather than fighting immediate fires. Certainly, I have a nice list of things to do growing in the Network Engineering queue...
So, whilst I'm no longer a K-12 "school sysadmin", in North America, they still refer to higher education as "school", so I guess in some ways, I still therefore do "school sysadmin", and there's no need to change this blog's name!
It's been fun, and it continues to be fun!
No comments:
Post a Comment