Monday, 29 February 2016

Apple caching server

With the ever growing number of Apple devices [particularly iPads and iPhones] on our campus and (as far as I can see) no local Apple servers/CDN facility in the country, it makes a lot of sense for us to put in a cache.

Whilst we have a fairly fast internet connection, we have relatively little international bandwidth (which we share with many other schools). It should also mean much speedier installations of the "core" applications our users make use of, as it will be available as fast as the hard drive can throw cached content onto a gigabit LAN connection; depending on how the cache works, it may even insulate us from the odd Internet outage (at least in terms of carrying on installing apps [unlikely, I think - too much interactivity with Apple's servers for that to work]).

As Apple make a reasonably inexpensive software package to manage this (OSX server), I decided to venture forth into making this work, and acquired the most basic Mac Mini available - the cheapest device I could find to make this work.

Apple's instructions weren't, in my opinion, all that helpful for an enterprise network. They're probably pretty "plug and play" in a simple, flat network, but once you get to a fairly complex, multiple subnet routed enterprise LAN, things get a little more complex (or do they). I was reasonably certain they expected the server to be on the same L2 subnet as the client traffic but I'm not certain of that; certainly, other Apple stuff is very reliant on being on the same L2 subnet (like AirPlay), so I had my suspicions about that. [They're not, in this instance]

Of course, if I had a Macbook Pro or something like that I'd probably test out some theories with a packet sniffer, but I thought I'd rather hack together a working solution first. One day, I'll "borrow" an iPad and run some tests to see if the L2 setup with VLANs and interfaces in each subnet is actually needed; I'm not 100% sure it is, as it seems to report its main IP Address (from our RFC1918 IP on that interface) to the Apple servers when it starts up the caching service - of course, I had all the [unnecessary] subnets set up before I'd even purchased or downloaded the caching software.

It's complex enough that one of our neighbouring schools phoned us for advice a few weeks ago - of course, at that stage, we'd not even ordered an Apple to implement this, although we were strongly considering it, so couldn't help them out. I should probably check if they've worked it out!

[UPDATE] NB this addition of VLANs is utterly unnecessary - the Apple cache redirection actually happens through DNS records and the server registering with Apple's servers. Direct L2 connectivity between clients and cache server is not required for the caching service to work.
This may not be exactly how Apple envisions the kind of setup I wanted being done (one Apple OSX server for all my subnets, but I can't see that they wouldn't allow this, and their network diagrams suggest it's possible), but what I did was (NB the first two steps below may be optional, I've not had a chance to test to see how vital they might be):

  • Created a tagged trunk port (with a native vlan in my normal server subnet) including (tagged) membership of all the wireless VLANs.
  • Created virtual interfaces, one for each VLAN in which you want to work with Apple devices, in the Apple software (System Preferences>Network; click on the little gear icon and then Manage Virtual Interfaces... then add one virtual interface per VLAN; either statically (manually) assign an unused address, or set up your DHCP server to hand out leases for that device in each of your Apple-y subnets). Even the basic Mac Mini supports VLANs; I guess we can thank the BSD "under the hood" for that. This step may not be needed, but I didn't get around to testing any alternatives, as I have no Apple devices to hand.

Overview of the Apple Networking when I was done. Each uses DHCP.
Remember, the MAC address will be the same in each VLAN.

Overview of the various VLANs after you click on the gear icon, then Manage Virtual Interfaces...

Adding a VLAN after clicking on the + button in the window shown abve.
The Name is up to you; the tag is obviously the VLAN ID,
and you want to add it to the Ethernet interface.
  • Bought and installed the OSX server through the App Store (the little A icon in the "dock" at the bottom of the screen) using a pre-existing apple ID and some credit I bought at a supermarket on a voucher (it cost us nearly ZAR400 for the software, and just under ZAR10,000 for the basic Mac mini) - which is quite a lot in local terms, but considerably less than we pay per quarter for internet connectivity.  I'm pretty sure you can work this bit out without screenshots!
  • Configured the OSX Server Caching (I'm not using any of the other included features of OSX server, not even the macbook update server); set it to use 350GB of the drive for cache files (about half the available space). 
OSX Server Caching Overview.
The little graph at the bottom shows disk use approximately
equally split between iOS software, iCloud and "Other",
The globe icon in the "dock" area is the OSX server icon.
You can see the Caching server settings under the Services menu item on the left hand side.
  • Under the Edit Permissions, I set:
    • Cache Content from: all networks
    • Serve Clients with public addresses on other networks 
    • entered the public IP address range for our network
  • Set up our Microsoft DNS servers to serve the _aaplcache._tcp.<subdomain>.<domain> records we needed with our public DNS range (this helps to tell Apple and your clients what cache server they ought to be using). This was repeated for each subnet (we have a DNS name for each subnet, like <Staff|Student|Guest>.wifi.<domain> - each of these had the relevant record added in the prs format, with a range from our lowest public IP to our highest public IP. We're still using NAT down here, and our servers and clients NAT out different addresses; if you have only a single public IP address, your job is probably easier, as it will "just work" [i.e. if you put an apple cache server in each subnet, using only that subnet and the same NAT rules, it should "just work"]!
    The caching server will give you a mostly right command to enter the relevant TXT record into BIND and MS DNS servers (Click on the Client Configuration.... button under Edit Permissions...); you just need to edit the site-specific parts of the suggested command, or use the record settings displayed to edit them manually, if you prefer. 
  • If you use a range of publicly routable (non-RFC1918) IPs directly for clients, you also need to use this DNS route.
    One of the DNS TXT records in our staff wifi sub-domain under our school's top level domain
    (which might be school.local or school.cctld, or whatever you use)
    in other words, you're seeing the response the dns server would give (for school.local - the bit I've obfuscated)
    for a query to
    • I'll highlight this point, because something went squiffy in our DNS records - MAKE SURE THE IP ADDRESSES IN THE prs RECORD ARE CORRECT.
      Somehow, at some stage, we dropped an entire octet of the end of the range statement (yeah....). You can imagine that causes problems. Like the cache totally failing to work... Quite how it worked at one point and then... didn't is a bit mysterious, but it happens, often due to PEBKAC. 
    • Run a lookup on the TXT record from a client in each DNS search domain and MAKE SURE they're right. 
      • dig or nslookup will work quite nicely for this with the correct command syntax, depending on what client you want to query from. 
    • Being a unix-based OS, all the lovely things you get used to in a server shell are there - one of my favourites being the tail command thrown at the relevant logs - in this case, tail -f /Library/Server/Caching/Logs/Debug.log which lets you watch your server doing things in real time from the terminal. 
    • The Stats menu item of OSX server also lets you see graphs of activity

    Stats in OSX server showing cache activity. Not many cache hits just yet!
    I now need to test a little more, but I've seen some traffic working through it, so I think the above instructions should get you to a working configuration. Of course, there are no iPads lying around our office, so I can't test things as I usually would (and I personally don't much like, and cannot generally afford, Apple devices!). [It works :)]

    There are additional settings that aren't obviously accessible through the GUI; at some stage, the system ended up adding localhost IPs to the networks in Edit Permissions... which it did not like at all - and greyed out the option to delete from the GUI, so I had to edit the configuration file (/Library/Server/Caching/Config/Config.plist) with vim (as root, using sudo) to delete the offending entries under <PublicRanges> (the XML, although long, is reasonably obvious).

    As I mentioned at the start, this may be overly complex [it is] (at least the "putting an interface in every VLAN part [don't bother, unless you have another reason for L2 connectivity])) or not quite how Apple expects things, but it's what I deduced with a quick skim of the documentation, and I'm seeing client traffic using the cache, so something must be right! If you wanted more performance, you could consider putting an SSD in, but with only a single gigabit NIC, you'll probably struggle to get the full value from an SSD.

    If this is very successful, we will probably add a second cache at a later date, and peer them (they'll grab content from each other if the other cache has it), as we're not huge fans of single points of failure in our network, and distributed and fault tolerant architectures are just a good idea! On the other hand, if the cache server fails, the clients will just fetch directly, so we've probably got more important SPOFs to fix first...!

    If you're big into Apple devices, and iCloud usage, it may help you to have many caches, peered with each other, but each serving a smaller percentage of your user population (consult the documentation for that scenario) - it really depends what costs you more money - Apple machines and/or disk space, or bandwidth. For most of us, disk space is cheap (and once off), bandwidth is expensive (and ongoing). Always consider TCO!

    Update: It works well


    1. This comment has been removed by the author.

    2. Hi James!

      I read this interesting post.

      One Question: Do i have to set the _aaplcache._tcp.. entries into our internal DNS (xxx.local), or to the external public availabel zone


    3. Hi Martin,

      You should add it to the DNS servers that respond to internal client requests (which are presumably the same as your xxx.local domain).

      Note that if you use different subdomains, you will have to add the _aaplcache.tcp. record to *each of them* (i.e. and or whatever). Sorry if that was not clear. If your whole site is setup as yourschool.local, then you'll only need one record; obviously, you only need to add it to subdomains where you expect devices to use apple cache (which may only be on WiFi, depending on your organisation).

      If you have split horizon DNS, only have the internally facing DNS resolvers doing this; the external ones would never be used.

      1. If you look carefully at the screenshot in

        you will see I'm in the tcp entry under staff, under wifi, under my (blanked out) school domain.

        Other DNS servers will be similar, but I have screenshots from MSDNS, as that is what we use here. :)

      2. In MSDHCP, it's Scope > Scope Options Option 015 DNS Domain Name that governs what clients in that subnet will try and find.

      3. Ultimately, it probably depends on what domain your clients are being put into by DHCP options - whatever the subdomain is that your clients are, then that is where you want to put the _aaplcache._tcp record(s).
        So if your clients are school.local, then there; if they're, then there. If they're one or the other, well, then both - but only *internal* clients need access to those records (split horizon DNS is your friend here). Apple doesn't need to see them, nor do hackers wanting to learn the intricacies of your public IP space from a simple DNS lookup ;)