OMG. I can't Teach, the Internet is down...!

Earlier in the week, we had our first major/prolonged outage since I've been here. A contractor (somewhat ironically installing a route for a second fibre to prevent this sort of thing from being an issue) drilled through the only fibre cable serving one of our biggest teaching blocks; 12 classrooms and a (deprecated) computer lab - and taking out Afrikaans, Geography, History and about 1/6th of the English department. It also reveals that the school makes "production" use of some very popular unofficial communications channels, particularly WhatsApp.

In a sort of "is this a hidden camera show?" scenario, it unfolded a bit like this:

Despite starting drilling a good 10cm away from the existing fibre, I pointed out that the drill bit was not straight and was pointing upward (i.e toward the existing fibre). This was not considered to be a problem, perhaps because geometry/trig is irrelevant once you leave school.

Next, I pointed out (from the other side of the wall, in between drilling "sessions", through a window) that the cable had started to move/wobble/vibrate a bit. Then a bit later, alarmingly so - so if they weren't pushing on the exposed cable by accident (I can't see through brick - yet), suggesting that they were almost certainly hitting the existing cable with the drill bit.

Drilling continued.

Then I announced that the link light had gone out on the switch.

*facepalm*.

Fibre cable, meet drill. 

A 24mm drill bit can really chew things up.

Fortunately, they have access to a basic fusion splicer, and are planning to get a more pro-level Fujikura in the near future. Hopefully, a decent OTDR to rate the cable plant too. (You shouldn't accept a splicing job as complete until you have an OTDR trace of it [or an optical power meter reading, failing that], end-to-end; if you don't own one, make it part of the installation process, and keep the results on file - similarly, make sure copper plant is certified and passes your installed standard).

Interestingly, they didn't seem to know that most multimode fibre (50/125/250) is now the same diameter as most singlemode fibre (9/125/250) (so they wondered why they had to use the "singlemode size" fibre holders on the device, instead of the 900 micron "multimode" ones). Sadly, it started raining and put off completing the repair until the next day, so we had a pretty prolonged outage until late morning the next day. Most teachers took it fairly stoically, thank goodness. This also beats the old problem, which was having to fly in contractors from a city 800km away every time you needed to splice something. The availability of fusion splicers has gone up drastically in the past decade, to the point that there are at least two in this little town, and more within about a 2 hour or so drive.

Tip: Always watch contractors when they're near your vital infrastructure.
Counter: Contractors won't necessarily listen to you, because you don't know what you're doing and they do, which is obviously why you hired them... (Or you're too British about the whole thing and aren't jumping up and down on their head like a troop of enraged howler monkeys).  They should probably realise if you keep going off and grabbing tools they need for the job out of you own offices, and know where stashes of things are around campus in other departments, you might have done this sort of thing before yourself.
In a crisis: If you have to, literally pull the plug, have a discussion, and point out your concern(s) if you don't think you're being properly listened to. You may even want to discuss a "stoppage" procedure ahead of time.
Best practice: If you want something done right, do it yourself whenever possible. "Saving precious time" sometimes backfires. Part of using contractors inevitably means a certain amount of your time will be needed to supervise, make sure they comply with your campus health & safety procedures (if the law in your country makes it your problem, it does in ZA) and, of course, that the work meets or exceeds standards as agreed in the contract.

Eventually, they will come back with some more multimode and splice in a new section; at the moment, the cable is not really maintainable, as you have to splice whilst hanging off a ladder, which isn't exactly easy or comfortable - or particularly conducive to good splicing.

Other than this incident, which left everyone involved fairly mortified, they have been absolutely great, and I have no hesitation in recommending their cable installation services; this is the second time they've done a major job for us. In particular, they do a fantastic job of trenching, sleeving and manholes.

This sort of incident might just be one of the reasons people suggest that multiple fibre cables should enter buildings on opposite sides, and only meet up right at the relevant switch/router. Reusing/sharing routes, fibre patch panels and splicing cassettes between cables, unless port density is really a problem, is probably also a false economy for similar reasons. With the razor sharp crystal clarity of hindsight, this is why I had the original route for this cable going through the other side of the building, until it was pointed out that it would be "much easier" to go this way. This is why documenting decisions is a good thing - you have a reason why you think one way one day, and that may very well be the culmination of a lot of thinking of factors other than contractor convenience!

And yes, that last shared 20-30cm of physical path can actually make a difference to reliability...