School Sysadmin: November 2015

Friday, 27 November 2015

PacketFence - Configuration

Previously, I ran through basic installation of the PacketFence ZEN system on an ESXI host system - and the various unexpected hurdles presented. Today, we'll continue with some in depth configuration of PacketFence to be a useful production NAC for a wireless controller based network.

For something that bills itself as "zero effort", there is a surprising amount of effort involved. (i.e. I didn't find a single short guide that worked "out of the box" with what I wanted to achieve - let alone the ZEN working "out of the box", which is somewhat irksome - but understandable, because a NAC requires careful rule settings which will always vary between environments. This has taken a long time to approach a working config - I've been editing and republishing this for the past week or so amidst all the other interruptions to my typical day - which is never exactly helpful when you're doing some hardcore "hacking" - but typical in the life of a Sysadmin!). Of course, there are a lot of "moving parts", but it's not exactly a trivial configuration exercise. Hopefully, this guide will save others a lot of grief and head-pounding frustration! Make sure you follow all the steps. It helps to wrap your head around all the bits and how they mesh together to make a working NAC.

Once we've configured PacketFence, we'll be focusing on pairing this with a managed wireless system (part 3 - specific to HP wireless controllers, but broadly applicable in concept; I'm likely to write a similar guide for Ubiquiti as soon as I obtain some UniFI APs), as this is how all "scalable" BYOD deployments proceed (unmanaged wireless networks [and switches...!] are categorically to be avoided in the enterprise). Wireless typically also works very much better with NAC than with ethernet switches, which are often a bit spotty in their support - in part because you generally have only a single vendor and piece of kit to play with. If your organisation is anything like ours, deployment of wired NAC is also complicated by a lot of old, legacy bits of networking kit floating around - not to mention unmanaged switches in all sorts of dingy corners! An Inline configuration will of course cater to everything (even unmanaged stuff) but it is not ideal, and isn't really an "enterprise" configuration.

At some stage, we will look at NAC for Ethernet too, but that will be "down the line".

You may of course wish to refer to the official manuals (ZEN 5.4.0; full PacketFence 5.4.0). If you get horribly stuck, sit down with a pot of coffee and wade through the entire full PacketFence manual. Its a shame there isn't a sort of "flow chart" of all the processes that mesh together to allow (or prevent...) successful authentication/authorisation. I found the equivalent "packet flow" diagrams for Mikrotik routers very helpful when playing with advanced configurations on those.

This guide will only cover the "generic" things that need to be set up prior to configuring devices like wireless controllers or switches; anything that you would need to do to get such devices configured will be covered in subsequent posts - for example in this one for HP Wireless.

Requirements:

Basic PacketFence ZEN 5.4.0 installation
Access to Domain Controllers and Active Directory (you should possess Domain Admin credentials and/or logins to AD DCs)
Shell access to the PacketFence server (use PuTTY). The default root credentials are noted in the manuals. One of the first things you should do is change them - preferably for certificate-based authentication.

Configuring PacketFence ZEN (5.4.0)

Logging in

Assuming you're where we left off in the previous post in this series, you should be at a login screen. If not, go to https://<IP_of_Your_Server_Management_VLAN_IP>:1443/admin/

Enter the Admin login credentials you created in Step 21 yesterday. You'll be greeted with the Admin Dashboard, which displays a number of metrics about your NAC system:

PacketFence Admin Dashboard

You should see a number of options in the navigation bar at the top of the screen; we'll spend most of today in "Configuration", so click there now.

Packetfence Main Navigation Options

The rest of this guide will often assume you are already in the Configuration section, so click there now if you haven't already; you will be presented with a long list of various things you can configure. I generally like to click on EVERYTHING when I get a new package to see what is there; you may like to spend a few moments doing that (don't change anything though, unless you're sure about it...).

General setting screen in Configuration.
NB in DHCP Servers, include ALL relay IP addresses on networks to which
Packetfence is directly connected unless you want a lot of spurious rogue DHCP detections.

If the settings under the General screen are not correct for your environment, change them now!

Roles

Configuration>Users>Roles

Roles is where you set up user roles (it does exactly what it says on the tin..). This is where you can begin to ensure that different "types" of users have different privileges (such as number of devices they can enroll). You will make use of these with Rules under Users>Sources later on.

To add the various Roles you want, click on the Add Role button, and then enter the information you want. Create a role for each and every "distinct" functional user role you can envision (assume you'll be working with Security Groups in Active Directory - further assume you'll be making a 1:1 mapping to functional roles you define here). Note that you cannot rename a Role in PacketFence once it is created (Delete and start again if you need to). This "functional role" concept is discussed further below under the Active Directory Integration heading.

Adding Roles. Setting Max Nodes as 0 means "unlimited".

In our environment, I've created Roles for:

Junior School pupil users (who can register 1 device regardless of whether or not they are officially doing BYOD)
Staff (who get 5)
Senior School Students on BYOD (who get 2)
Senior School Students not in "official BYOD" year groups (who get 1)

Consider your own "functional roles" and create appropriate Roles for each of them - and don't forget to make matching Groups in AD with which users in those groups/roles will be associated.

Active Directory Integration

You probably already have a directory of valid users; for most schools, this is likely to be in Microsoft's Active Directory (you could also use generic LDAP or a number of other providers - including Google through OAUTH, but we're going to cover AD here, in part because of limitations with LDAP and EAP, and because generally when you have an onsite directory of users, this is better than an offsite one!).
If you're not using AD, you'll need to work out how to get the same sort of things configured for your environment.

PacketFence User
If you haven't already done so, create a new user for PacketFence in Active Directory - it will apparently (according to the PacketFence documentation) need Domain Admin permissions to be able to read the data it needs; I like to create dedicated "software users" with strong passwords for this sort of thing rather than reuse a generic domain admin account (or my own). I would suggest you try to see if indeed it actually needs Domain Admin rights or not (some light Googling suggests you can change access permissions to LDAP data if you want to, which might be a solution).

Functional roles
I would also strongly recommend creating a dedicated Security Group for each of the "Roles" and adding the relevant users to it. This will allow you to remove network access from wayward users if you need to simply by removing them from the relevant group (i.e. "StaffWiFi" "GenericStudentWiFi" StudentBYOD" WiFi, "Grade1WiFi" in your 802.1x authentication) and so on. Think of it in terms of functional roles rather than year grades (i.e. "people who are allowed on the network with 1 device" "people who are allowed on the network with infinite devices" etc). In some cases, grade may well be the same (i.e. Grade 5 allows specific rights) - but don't reuse a grade security group, as you may need to kick someone off the network, but that doesn't mean they are no longer in grade X! In other words, have both Grade5 and Grade5WiFi security groups and so on.
You may also like to consider a "banned" functional role that supersedes the allowed group(s), but that's an exercise for the reader! These functional roles should almost certainly each have their own VLAN and subnet so you can further control access to things through switch/router based ACLs or firewall rulesets.

Configuration>Users>Sources>
Add Source>Internal>AD

Add a new User Source from Active Directory

Here are some of my WiFi specific AD groups (selected)

It is probably easier if all of your users are under one single OU. If they're not, you may have to create a couple of sources (or try creating your Source higher up the "tree" - i.e. point at DC=school,DC=edu rather than OU=Users,DC=school,DC=edu - particularly if for instance students and staff are under different OUs). You could also create sources for your secondary (and perhaps subsequent) domain controllers if you don't want a single point of failure - or use round robin DNS (see below).
HOWEVER, if you make multiple sources, you will find maintaining Rules more challenging, as you will have to keep each source you define up to date. Ideally, a single fault tolerant (DNS round robin is a good bet) source that can query your entire set of potential users - regardless of their OU (staff, students, computers, software users, etc.) is your best bet.

Fill in the various options in the New Source dialog screen.

Name: Give it a memorable name. I generally use things like DCname-OUname
Description: Self evident
Host: The IP address or FQDN of your domain controller (or a round robin DNS name). Leave the port on 389 unless you've changed it, and leave it as None unless you've installed SSL/TLS functionality for AD/LDAP lookups.
Connection Timeout: How long to wait before timing out. Leave as is.
Base DN: This is the OU you wish to search in AD
Scope: Defines what to search relative to the OU. Subtree is probably correct for most cases. See this MSDN article for an in depth look at most of the options.
Username Attribute: Which bit of AD you want to be the "username". Most often "sAMAccountName", which is the normal user login name for your domain. You may also like to use something like "mail" to use their email address instead.
Bind DN is your packetfence AD user in full distinguished name (DN) format
Cache match lets you cache searches; leave unticked for now; there may be reason to turn it on in production systems (particularly if your PacketFence server gets busy)
Password is the password for your PacketFence AD user. Enter it and test it.
Use Stripped username : leave blank

Once you've filled everything in, click on Test; you should get a green "success" bar.If not, check what might be wrong and correct it (make sure DNs are correct; make sure password for the user is correct; IP of your Domain Controller is correct, etc).

Success!

If it works, click on Save. Repeat it for any other OUs you might need (and possibly for other Domain Controllers).

Tip:
If you find writing out the LDAP-style DNs tricky/painful, there's a way of getting them out of AD in a copy/paste-able format. In Active Directory Users and Computers, find the object you want the DN for, right click on it, left click on Properties, left click on the Attribute Editor tab, scroll down to distinguishedName, select it, click on View, select, copy and paste the DN into wherever you need it. NB:I found this trick only seemed to work on my primary domain controller.

Hate typing out DNs? Copy and paste them from AD.

Round Robin DNS

Another option might be to create a "round-robin" style DNS entry for a "generic" FQDN like adserver.yourdomain.com - so long as port numbers are the same - which returns some or all of the IP addresses of your AD infrastructure. It's generally quite a good idea to move towards the idea of a generic "service" FQDN that returns all valid, identically usable servers of that type.
An example I often set up is NTP - I create two (or more) ntp.domain.com A records to each of my NTP servers. I also create lettered specific A record pointers to individual servers (a.ntp, b.ntp....) in case I want to specifically reference one or more of them. A lot of these ideas I obtained from my learned colleagues at Rhodes. RFC2219 is also related (note that MSDNS doesn't really support the CNAME approach to this, which is a shame).

First, create your direct pointer to a specific server.
This will also create the adserver "subdomain".
Repeat for other active directory servers.
DO NOT FILL IN THE WHOLE FQDN in the first box.

Next, navigate to the adserver subdomain in your AD DNS tree, add a new host (A record), leaving the hostname name blank; this means you're creating a record for the "parent domain name" - in this case, adserver.domain.tld:
.

Leave the hostname blank.
Enter the IP address
Click Add host.
Repeat for each round robin IP.

Once set up, you'll end up with something like this:

Round Robin DNS for Active Directory Servers on adserver.domain.tld

Rules

After much head<>desk action and after writing pretty much all the rest of this blog entry, (not to mention a lot of time looking at logs) I suddenly came to the "OMG, I'm a massive idiot" realisation that not much works without setting Rules in Users > Sources (one of those retrospectively "bleedin' obvious" things you feel quietly embarrassed about for a while). This is the "magic glue" that ties together the AD/LDAP information and what you want to have happen.

So, you need to set some rules...

Open up your existing source(s) and add one or more rule(s) to it/them by clicking on the Add Rule button in the bottom right.

The missing magic: Adding rules to your source(s)

Rules take the form of If condition X...., then Y..... Where X conditions and Y actions can be multiple things. Actions *must* at minimum (for authentication rules) set either an access duration (in hours) or an expiry date (a date some time in the future); and a role. Setting an access duration makes sense; you don't really want infinitely long access - you may for example like to set Student registrations to expire at the end of the school year, and staff ones in 4 or 5 years (or, as policy, have everyone register once a year, or a semester, or whatever makes sense to you and your school). Helpfully, the system will print out error messages when you get this wrong and refuse to let you save. Here are two examples of errors you might see - easily fixed by following the instructions they hint at:

You have to set a Role or other access type

Access must be time-bound.

To create the rule, you'll generally want to set some Conditions first; if you don't, it's a "catch all" (so you might want to create a catch all rule that goes at the end of the list of rules that does something like grant guest access or specifically sets a "banned" role. REMEMBER if you put a catch all first, you're going to stop your Rule processing dead at the first stage!

You can obviously create many rules - they are evaluated in order, top to bottom, so you may have to carefully consider the order (much like with firewalls!). New rules are added to the bottom; the list only seems to "fix" its order once saved, otherwise it will merrily add rules and order them alphabetically. If you get annoyed that your list is getting into the wrong order as you add Rules, save, reorder, save and try adding further rules, saving after each addition.

Typically, you're going to want to evaluate "Member Of" type rules. Remember, you will need to use the full LDAP distinguishedName, not just the name of the group!

Here is my test LDAP Source:

Note the four Rules and their order.

Here are each of the Rules, in turn:

Staff WiFi - note the full LDAP DN,
not just the group name in the memberOf condition.
Based on that, it then applies the user Role of StaffWiFi and sets expiry of that Authentication to 1 Jan 2020.

Generic Senior Student
Again, sets the Role and Expiry date based on the group membership.
Student logins expire at the end of the calendar year.
As I'm setting up for 2016, this is the end of 2016.

Very similar rules for BYOD senior students.

Here is a Guest "Catch All" - this is not a very secure configuration as there are NO conditions and it will match anything. It's here to illustrate a) the importance of rule order and b) that by leaving conditions blank, you create a "catch all" rule.
Incidentally, if you don't have a catch all rule and there are no matching rules, the authentication will fail.

The quick way of evaluating your Rules is to run the pftest authentication script in a shell as you make changes with test user cases.

Success - correctly defined Source and Rules in the LDAP source -
gives a user who is a Member Of the Staff_WiFi group the StaffWiFi Role,
and set this to expire in 2020.
The other sources give errors, as they are not yet correctly set up, or not relevant.

LDAP - completely optional

You may want to use LDAP to set permissions (authorise) (vs. AD to authenticate, because you apparently cannot user LDAP for EAP authentication). There is nothing in PacketFence that precludes using both- i.e. you can Authenticate against AD and then Authorise against LDAP; you can of course both authenticate and authorise against both, once you've set up Rules in each. The Source controls Authentication; the Rules control Authorisation.

Create an LDAP source, using similar settings to the AD one above; set the port to whatever you need in your environment. You may find the FreeRadius LDAP wiki entry of use if you get stuck. http://wiki.freeradius.org/modules/Rlm_ldap

Again, round robin DNS is probably going to save you time in the long run.

LDAP configuration.
Note the roundrobin hostname rather than an IP address.
Make sure you use the right port for LDAP against AD.
You'll see I'm using the GC LDAP port; this may not be required.

You may also like to edit /usr/local/pf/raddb/modules/ldap to reflect your environment (using similar settings to those above).

FreeRadius

You're almost certainly going to want to authenticate users using an existing directory system; Active Directory (AD) is the most common. We're going to follow the mechanism that allows clustering, because at some stage, we're going to want to eliminate the SPOF that is a single PacketFence server. RADIUS is a AAA protocol when fully configured - this is Authentication (you are who you say you are), Authorisation (you can do what you can do) and Accounting (we know what you did last summer). This is option 1b (p35 onwards) in the manual. Option 1 (p30 onwards) is less involved if you're never going to cluster. Around here, we like to strive for resilience, so let's do the heavy lifting:

In a shell, issue

yum install samba krb5-workstation

- in ZEN, this seems to be installed already:

Yippee, it's already there.

You need to ensure the FQDN is properly entered in /etc/hosts. Edit it with vim, and ensure that there is a line that gives the management IP, FQDN and hostname:

settings for a host called falkor that is in kc.ecape.school.za

You may wish to add further entries for any other IP addresses that may be on the system, particularly those that are used by clients.
Save it. issue

hostname -f

in the CLI to ensure the correct FQDN is returned. (If it isn't, assuming you've edited the file correctly, rebooting the server is the easiest way to get it to reflect).
If that still doesn't work, you may need to change the hostname in /etc/sysconfing/network (try restarting networking or rebooting to get it to reflect).

Next, you're going to need to configure FreeRadius to use this to interface with AD.

Open /etc/krb5.conf in your preferred editor.

Changes to the file highlighted in yellow

You will need to edit it to match your actual domain details (obviously not likely to be the same as mine). See page 36 in the manual if this is not clear. Save your changes. You can repeat kdc = lines to add redundant Kerberos servers (aka your domain controllers). In theory, you can also allow DNS lookup of most of these things (under [libdefaults]) if you domain's SRV records are correctly set up. However, as the manual doesn't cover this use, I've not gone there.

Next, you need to edit /etc/samba/smb.conf

The default file contains a lot of options you're not going to need. Move it out of the way (mv /etc/samba/smb.conf /etc/samba/smb.conf.old), and then create a new one that looks (more or less) like this, using vim, changing your workgroup and realm to match your systems:

New smb.conf

Save and exit your editor.

Next, we need to get a Kerberos token (make sure your server's time is reasonably accurate!).

In the CLI, issue a kinit <packetfence AD usename>;

enter the password when prompted

Then issue a klist:

kinit and klist

Good job, you just authenticated against Kerberos.

Now you need to spin up samba.

service smb start

chkconfig --level 345 smb on

net ads join -U administrator

enter your password when prompted.

Joined successfully.

If you get errors about being unable to do a DNS update, chances are you or someone else already registered that DNS name in AD's DNS. Get rid of it, refresh your DC's DNS records, and try again.

Oops.

Next. issue

usermod -a -G wbpriv pf

Start winbind

service winbind start

chkconfig --level 345 winbind on

winbind

You may like to test things are more or less working; try issuing:

ntlm_auth --username:<user> --domain:<domain> --request-nt-key

enter the user's password when prompted.
If all is well, you should receive NT-STATUS_OK: Success (0x0)

Note that when you've done this, you will not be able to use the PacketFence Web Admin (Configuration>RADIUS>Domains) to manage RADIUS Domains:

You're going to have to keep managing it in the CLI

You may want to change some settings in ~~/usr/local/pf/raddb/eap.conf~~ /usr/local/pf/conf/radiusd/eap.conf:

Change from MD5 (deprecated) to mschapv2, for example
(DON'T EDIT /usr/local/pf/raddb/eap.conf
as it gets overwritten each time PacketFence restarts)

This is the right file to edit!
/usr/local/pf/conf/radiusd/eap.conf

Restart radiusd.

ntlm_auth

You will need to edit ntlm_auth in /usr/local/pf/raddb/modules to match up to your system. Edit it so that the path to the ntlm_auth file is correct (/usr/bin/ntlm_auth) and the bit that says MYDOMAIN matches your domain.

If you're going to use ntlm_auth, you've got to edit the defining module...

Restart radiusd.

Done!

packetfence and packetfence-tunnel virtual servers

If you look in /usr/local/pf/raddb/sites-enabled, you will find packetfence and packetfence-tunnel configuration files. You can edit them here BUT you will lose these configurations when you restart packetfence (service packetfence restart, or from the web interface, or by rebooting) (but not when you restart radiusd) - which might allow you to test some changes before "committing" them by then editing the "master" files at /usr/local/pf/conf/radiusd. If you want to edit the eap.conf file, this is also the location of the "master" for that file (as noted earlier). Generally, if you see a file in there, assume the corresponding version somewhere under /usr/local/pf/raddb/ will be replaced every time packetfence restarts (so if things break on restarting packetfence or rebooting, consider this as a likely cause).

I edited these files so much that I can't even recall how far off the originals they have "drifted", but my current packetfence virtual server configuration is:

server packetfence {

authorize {
preprocess
mschap
ntdomain
suffix
eap {
ok = return
}
files
expiration
logintime
update request {
FreeRADIUS-Client-IP-Address := "%{Packet-Src-IP-Address}"
}
update control {
PacketFence-RPC-Server = ${rpc_host}
PacketFence-RPC-Port = ${rpc_port}
PacketFence-RPC-User = ${rpc_user}
PacketFence-RPC-Pass = ${rpc_pass}
PacketFence-RPC-Proto = ${rpc_proto}
}
packetfence
}
authenticate {
Auth-Type MS-CHAP {
mschap
}
eap
}
preacct {
preprocess
acct_unique
suffix
files
}
accounting {
sql
attr_filter.accounting_response
update request {
FreeRADIUS-Client-IP-Address := "%{Packet-Src-IP-Address}"
}
update control {
PacketFence-RPC-Server = ${rpc_host}
PacketFence-RPC-Port = ${rpc_port}
PacketFence-RPC-User = ${rpc_user}
PacketFence-RPC-Pass = ${rpc_pass}
PacketFence-RPC-Proto = ${rpc_proto}
}
packetfence
}
session {
}
post-auth {
exec
# skip packetfence if we have already treated it in the inner-tunnel
if (!EAP-Type || (EAP-Type != EAP-TTLS && EAP-Type != PEAP)) {
update control {
PacketFence-RPC-Server = ${rpc_host}
PacketFence-RPC-Port = ${rpc_port}
PacketFence-RPC-User = ${rpc_user}
PacketFence-RPC-Pass = ${rpc_pass}
PacketFence-RPC-Proto = ${rpc_proto}
}
packetfence
}
Post-Auth-Type REJECT {
attr_filter.access_reject
}
}
pre-proxy {
}
post-proxy {
eap
}
}

and my packetfence-tunnel vitual server configuration is:

The main things to watch for are the order of the preprocess, mschap, ntdomain and suffix statements.

Realm

Realms are where you define how PacketFence should direct authentication depending on the username. Open up /usr/local/pf/raddb/proxy.conf in a text editor (vim for example) in a shell connection.

Scroll down to

realm example.com {

and start editing to suit your environment - the obvious thing is to change example.com to your packetfence authentication domain - so if you are telling users to authenticate using domain UPN, then it will be sAMAccountName@yourdomain.tld. - change the example.com to yourdomain.tld.
If you have more complex requirements (like different subdomains for different user classes) hit the books and figure out a regex match, or try creating additional realms. Again, they are processed in the order you place them in the config file, so if things don't work as you expect, carefully consider the rule order.
By convention, the UPN should be the same as their email address, but in some sites it may not be - it isn't in ours (neither the user part nor the domain) - make sure the difference is clear to your users!
Make sure you match the CaSE of the one defined in /etc/krb5.conf. I'm not going to cover setting up auth against DOMAIN\user credentials in this guide, but it is possible.

While you're there, consider changing type = auth to type = auth+acct under home_server localhost { It's at about line 126.

You should probably change the home_server localhost { to listen on virtual_server = packetfence rather than on 127.0.0.1 around line 133.

Around line 615, you should change auth_pool = my_auth_failover to pool = my_auth_failover

Restart packetfence

service packetfence restart

Vital non-PacketFence Configuration

There are a few things you need to do to make PacketFence actually work that don't involve PacketFence itself.

Mikrotik DHCP Relay Agent
with an entry for PacketFence IP
The other two IPs are the existing
DHCP servers (domain controllers)

DHCP

Device registration is inherently a Layer 2 (MAC Address) problem (because that's your device specific unique identifier). That means PacketFence needs to know - and somehow see - all of the relevant MAC addresses on the network. There are several ways to achieve this, but arguably the simplest way is to leverage your existing DHCP Relay (IP Helper) infrastructure to send copies of DHCP traffic in VLANs of interest (if not all of your VLANs) to the management IP address of the PacketFence server. On every single switch/router that is doing this, insert another statement that sends the traffic to the relevant IP address. The PacketFence manual covers additional methods you might use if you don't have DHCP Relay/IP Helpers on your network.

Such IP helpers are also how you might be doing PXE booting across subnets for things like lab imaging.

Once you do that, you should find your Nodes list expands rapidly as clients ask for DHCP leases.

Custom Captive Portal

Configuration -> Portal Profiles
You may wish to customise your Captive Portal - you can change the default one, or add specific ones based on things like which SSID or VLAN they are connecting from. It's reasonably self-explanatory for the most part.

You may find editing things like the AUP text tricky - there seems to be a permissions error on the files, so Apache doesn't have write permission to the file. This can be remedied by changing file permissions (not recommended) or editing them in the CLI using VIM; the files are at /usr/local/pf/html/captive-portal/profile-templates/register/

Optional Steps

Fingerbank

Fingerbank is essentially a cloud-based device fingerprinting service, which allows your NAC to (reasonably accurately) determine what a device trying to access your network is from a combination of various attributes visible over a network. This may allow you to set up specific rules later on, and is probably worth setting up.

Configuration>FingerBank>Settings
Within PacketFence, you will need to use a Github account to obtain an API Key for Fingerbank. A free GitHub account will work fine. Click on the link under Configuration>FingerBank>Settings to set it all up. Once you're logged in with GitHub, click on your GitHub username on the inverse.ca website and it will show you your personal API key. Copy and paste it into your PacketFence instance in the appropriate box.

OMAPI

You can apparently speed the IP/MAC address mapping up by using OMAPI. You have to establish a shell connection to your PacketFence server and issue some commands.
Configuration>MAIN>OMAPI>
In a shell on the PacketFence server, issue:
dd if=/dev/urandom bs=16 count=1 2>/dev/null | openssl enc -e -base64
copy and paste the resulting text (which will look a bit like m4NMkOKc9IxfWk8cL2fP4g==) into the OMAPI base64 key field of the web based configuration
Click save

Back in the shell, restart the dhcpd daemon

service dhcpd restart

Expiration

You might like to save yourself some disk space. Obviously, set this according to your retention criteria. Under Configuration>Main>Expiration, you can set data retention times.

Choose some sane retention times, either according to your internal needs or legislative/regulatory needs.
Eduroam typically want you to keep logs for six months.
Wouldn't it be nice if there were a school equivalent to the HE only eduroam...?

Email

There's a reasonable chance you're going to want to get email to flow out from the PacketFence server. I'm going to assume you're using Google Apps for Education to provision your email; so long as you use an email provider with similar settings, you'll be able to configure your mail following these steps. Postfix is already installed (you should also add cyrus-sasl-plain and mailx using yum), so it's reasonably straightforward - you just have to persuade it to send via Gmail rather than attempt delivery itself.

Give the packetfence user an email address in AD (enter something sane in the email box) packetfence_do_not_reply@yourdomain.com is probably a good bet).
If you have a sufficiently complex password, it may not work properly, and you may have to choose a more restricted character set (echoes of MySQL...!). If you get authentication failures in /var/log/maillog once you're done setting email up, it's probably that - ideally, change the password in AD and GAFE - and everywhere you've set it in packetfence (like all your Sources). You might get away with just changing it in GAFE, but the next password/GADS sync may break it at any time.
If you get bored of seeing connection errors over ipv6 to google's mail servers, edit the inet_protocols line in main.cf to read ipv4 instead of all. Obviously, if/when you put IPv6, into production change it back.
Use GADS to sync your user information (to provision the packetfence user) - ensure it's in an OU that is synchronised through GADS.
Configure postfix (which is already installed) to do authenticated SMTP relay through Gmail.
https://charlesauer.net/tutorials/centos/postfix-as-gmail-relay-centos.php gave me the initial pointers for CentOS (I last did this years ago on Ubuntu). There is a lot more reading at http://www.postfix.org/SASL_README.html
From a shell on the packetfence server, encrypt your login credentials:

echo "[smtp.gmail.com]:587 smtp_user:smtp_passwd" > /etc/postfix/sasl_passwd

replace smtp_user with your gmail address and the smtp_password with your Gmail password

This will likely be something like: packetfence_do_not_reply@yourdomain.com:MyVerySecurePassword

I of course got errors about unexpected characters (like parentheses brackets). So I edited sasl_passwd in vim, inserting the same text (just the bits between the quotes - excluding the quotes).

Then you should run:

postmap hash:/etc/postfix/sasl_passwd

to format the data so Postfix can read it. Delete the sasl_passwd file (but not sasl_passwd.db) so your credentials aren't lying around in plaintext.

Add the following to the end of the /etc/postfix/main.cf file

smtp_sasl_auth_enable = yes

smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

smtp_sasl_security_options = noanonymous

smtp_use_tls=yes# Secure channel TLS with exact nexthop name match.

smtp_tls_security_level = secure

smtp_tls_mandatory_protocols = TLSv1

smtp_tls_mandatory_ciphers = high

smtp_tls_secure_cert_match = nexthop

smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt

relayhost = [smtp.gmail.com]:587

Restart postfix (do this any time you make a change in the main.cf file)

service postfix restart

You should then get emails if you have configured alerts etc.

You can run some tests if you install mailx

yum install mailx

Then

mail <your email address>

enter a subject

enter the body

press enter

.

press enter

it will sent the mail.

tail -f the mail log if it seems not to work. You can force delivery again with postqueue -f

Protip: check the size of your mailqueue as soon as you think your mail might be working - you may like to empty it before you get hundreds of emails about all the things you have not done right just yet... mailq will return the mailqueue, with the number of requests at the end. postsuper -d ALL will get rid of them. I had hundreds, almost all of which were spurious rogue DHCP alerts...

Passthrough

You may want users in Isolation or Registration VLANs to be able to access certain sites or resources. This is called Passthrough and well documented in the FAQ: http://www.packetfence.org/support/faqs/article/how-do-i-let-users-trapped-in-registration-or-isolation-reach-certain-websites-passthrough.html

Troubleshooting

RADIUS

If you need to "debug" RADIUS, open two shell windows, one for interacting with the system using radtest and the other to view the output of radiusd in debug mode (radiusd -X -d /usr/local/pf/raddb In 5.5.1 they made some changes to how RADIUS works; you now have to specify whether you want it to run as the authenitcation or accounting daemon, so the command you probably want is probably radiusd -X -d /usr/local/pf/raddb -n auth If you get errors about ports not being specified, this is probably the issue; to start the accounting server change auth to acct at the end.) - this specifies both running in debug mode, and what configuration files should be used. You will likely have to kill the existing RADIUS process to get it to work (to find the process, ps aux | grep radiusd); use kill -9 <PID> to stop the process; you can then run radiusd in debug mode without getting errors about addresses already being in use:

Finding the PID of a running process
ps aux lists pretty much all running processes
| grep <whatever>searches that output for what you're looking for
The PID is the number after the user running the process.
In this case, you can see the radiusd process, and my grep process.
The PID for radiusd in this case is 6798.
You then use that to kill off the process.

Once that's done, start radiusd in debug mode in one shell instance, and then run some radtests in another.

radtest -x <user> <password> <IP>:<port> <NASID> <radius secret>

If you look at the radiusd debug output, you'll see something like this:

radiusd -X debug output. Note in the example above it successfully authenticated DESPITE being a nonsense user.
That is probably not what you want, and likely indicates an error in your config!

You may find being able to send RADIUS messages from your desktop quite useful. One option is NTRadPing. It's easiest if you set up a client for your desktop's IP address in client.conf that does not expect message authentication. It will only do authentication, not authorisation, so you will probably get RADIUS access deny messages; that said, if you watch your RADIUS screen in debug mode, you'll quickly see if it's properly passing the username and password as you would expect.

If you get horribly, horribly unstuck, simplify.
If you run radiusd -X, it will load the generic freeradius factory defaults - this is what most freeradius guides assume you'll be using, and there's fewer complications in them. If you can't get RADIUS to work, follow a guide like this one [I've since written my own] and you will very quickly get a working RADIUS configuration - then figure out why the PacketFence one differs, and what you can do to make it work!

If you have a simple AP that does RADIUS, you may like to test with that before you get into the complexities of multiple VLAN controller-based RADIUS authentication and authorisation. DD-WRT based ones work quite well and are things techies often have lying around. It took about 20 seconds to configure mine to work with RADIUS on the packetfence server - it was simply a case of pointing the AP at the RADIUS IP and port, giving it a shared secret, and adding the AP to the clients.conf file.

General PacketFence

To watch the general behind-the-scenes PacketFence action:

tail -F /usr/local/pf/logs/packetfence.log

There are other more specific logs in that directory, which you may want to have a look at too, depending on where your issue appears to lie.

There are also some handy troubleshooting and maintenance scripts in /usr/local/pf/bin/

./pftest authentication <user> <password> <optional source>

is particularly useful for testing authentication.

take a look at each of them in turn; running them by executing ./<name> will generally print out the basic "help" page for that command.

System logs

Finally, you may need to peruse the generic system logs; these are in /var/log/. messages is the generic syslog destination that things that don't have a more specific log throw things at. Again, tail (or cat or vim) are all ways of looking at these files; tail -F is particularly useful as it updates as things happen.

The End! (or is it...?)

This concludes this basic PacketFence configuration guide. You may wish to refer to the previous PacketFence ZEN installation post, or subsequent device configuration posts (HP Wireless).

I would recommend you issue a service packetfence restart at this stage. There are several configuration files that are overwritten each time you restart packetfence, so this will help you catch such snafus before they bite you in production!

Thursday, 26 November 2015

Upgrading PacketFence

Sometimes, people immediately upgrade the shiny new toy you're working on - right in the middle of you documenting it. This happened with PacketFence, which went from version 5.4.0 to 5.5.0 the day after I downloaded the installer!
http://www.packetfence.org/download/releases.html
http://www.packetfence.org/download/zen.html

Upgrading is fairly straightforward, and is covered on PacketFence's website:
http://www.packetfence.org/support/faqs/article/how-can-i-upgrade-packetfence.html

As always with an update, be sure to read the notes that come with a release BEFORE you do an upgrade. These should generally be available at https://github.com/inverse-inc/packetfence/blob/stable/UPGRADE.asciidoc

You should only really upgrade things during a planned maintenance window - but if you're still in the experimental stage, you can do it any time. I'm probably going to stick to 5.4.0 until I'm done with the guide, and then run the update and see what happens!

I've actually now run an upgrade (to 5.5.1) see this post to follow the process.

Tuesday, 24 November 2015

Packetfence - Network Access Control

All of us need to control who can get onto our networks (at least, we OUGHT to control who can get onto our networks...). Whilst there are many potential solutions, we want one that is reasonably easy to use, scalable, flexible, resilient and low cost (free is good). Such systems are generally discussed under the tech term "Network Access Control". In this blog post, I'll be covering the basic installation of one such technology - PacketFence.

In particular, the flood of devices triggered by "BYOD" means you're probably going to want a slightly more scalable, rigorous and user-friendly mechanism for sign-ups that doesn't involve filling in bits of paper and manually registering devices. It is quite something when several hundred users turn up and immediately want to get onto the network (at the beginning of 2015, it took us around 2 weeks to deal with the first influx of BYOD for just two grades of official BYOD, and those bringing personal devices onto campus "for fun" - this year we're adding three more grades; we were using Active Directory integrated MAC Authentication; now we've mostly moved to 802.1x, but the solution isn't quite where I want it to be)...

What we want to do is:

Ensure that users can only "register" a certain number of devices, depending on their user class. Whilst you may be able to use something like Active Directory to authenticate user credentials, it does not keep track of (and limit) MAC addresses used, and allows people to use as many devices as they feel like (trust me that students will work this out quickly). Policy and rules only take you so far; sometimes, you have to enforce rules with tech!

So, for instance, you may want:

staff to have as many devices as they want (because if they use more, they can support more BYOD OS ecosystems in class!);
students in the formal BYOD deployment get two (an official work device, like a tablet or laptop, and a smartphone or other WiFi equipped "play" device);
other students get one "play" device.

With many network access control systems, you can "cap" the number of devices registered per user - which is very useful.

Allow self-registration of devices, based on user class rules.
Integrate with our existing user directory (AD/LDAP).
Have a fairly flexible "guest" network.
Potentially, hook into the federated eduroam network (which many of our student assistants, who are registered at the local university would find useful) - sadly only as a service provider given the rules.

You may wonder why we want to limit per user BYOD device numbers - it's primarily to (attempt to) manage bandwidth demands. I've noticed that kids will stream as much rich media on as many devices as they have simultaneously. Why, I don't know, but they do it. There are also other scarce resources like IP addresses (although we're mostly forced to use NAT given our limited pool of non-RFC1918 addresses).

You may find certain jurisdictions require you to keep records of devices you give network access to (in South Africa, that law would be RICA). You should probably also keep a record that matches IP address (DHCP leases) and times back to MAC addresses - particularly if you use real world IP addresses (most of us will sadly have to use NAT). NAC may or may not help you do that, depending on how it works and what logging you do. Make sure anything that logs has reasonably accurate time, preferably tied to your country's "official" source of time.

Packetfence

Introduction

Packetfence is a neat open source solution to enabling Network Access Control. In this blog post, I'm going to cover setting up PacketFence from the PacketFence ZEN (Zero Effort NAC!) .ovf template, which you should install on a compatible virtual machine hypervisor. This is designed to be very easy to configure in a "try before you buy" kind of way. You can also of course deploy it from the full installer and configure everything yourself, but this "preinstalled" version is much more in line with the amount of time the typical school sysadmin might have to evaluate a potential solution.

I'll be using (free) VMWare ESXI as the hypervisor for this example (if you've not got it set up yet, see here). You can also use it in a "liveCD/USB" mode, but we are not going to configure that here. You could also theoretically use Hyper-V if you're more "Microsoft-y"; you will, however have to convert the image using a tool like this to do so.

Installation Steps

Download the required software (a compatible hypervisor and the Packetfence ZEN .ovf)
You will generally need a minimum of four VLANs; setting these up is beyond the scope of this article (and often differs greatly depending on your setup), but you generally need Registration, Isolation, Management and LAN VLANs. In this example, I'll be using something closer to our wireless production network - Registration, Isolation, Management, Guest, Junior Student, Senior Student and Staff. If you're not using VLANs, you really ought to do so; PacketFence supports an "inline" mode of operation, but it is limited and we will not be covering it here.
Make sure those VLANs are trunked through to your hypervisor on the NIC(s) you want to use.
Once the hypervisor is installed and configured (with the requisite VLANs available), you need to deploy the .ovf template. I'm using 5.4.0 in this example. Open the .zip file and extract the .ova inside to a known location
In vSphere client, connect to your ESXI host and load the template (File>Deploy OVF Template...). In the window that pops up, navigate to where you just unzipped the .ova file; select it. Follow the wizard through, choosing sensible options for your environment. Don't choose to turn the machine on at the end; you will probably need to tweak some settings (like VLAN assignments/NICs) before turning it on.
It uses around 8 gigabytes of RAM and 50 Gigabytes of disk space, and configures itself with 2 vCPUs; make sure you have enough resources to support that!
The wizard will then take some time to expand and provision the virtual disk image. Wait until this is complete.

Completed deployment
(We use dragon themed host names for servers around here due to there being a wyvern on the school's crest)
Networking: I recommend using a dedicated NIC port for this on your hypervisor. On the switchport it connects to, create a trunk port with all the VLANs you need to enforce, ensuring the Management VLAN is untagged (native/pvid). Create a new vswitch; put it in all VLAN mode (4095). Ensure it is put in promiscuous mode so that you can see all the traffic, otherwise things might not work too well.

Ensure you have Promiscuous Mode enabled.

Power on the VM.
Watch the console; you will see the CentOS 6.7 guest OS start up.

CentOS 6.7 starting up.
Eventually, you will see a login prompt. On your workstation, connect to the full URL given in the CLI Prompt to do the setup.

Login prompt with web URL
Don't forget to use the HTTPS:// in front; you will get an error if you use plain http.
Accept/bypass any privacy warnings due to the untrusted rootCA.
Select the VLAN configuration mode:

Select VLAN enforcement mode, click Continue
Configure all the required network interfaces; you will need VLAN IDs, IP addresses, subnet masks and default gateways for them all (it's outside the scope of this guide to set these up; you should know how to do this as a sysadmin on your system!). When you create the subnets, make sure they are adequately sized for your expected usage patterns (i.e. is a /24 big enough?).

Management VLAN should usually be that on which your network equipment management interfaces are (what happens if you have more than one of these?)
Registration VLAN is that on which devices seeking access live; packetfence will install a DHCP server here, so use the IP Address as whatever you usually number your default gateway (typically high (often .254) or low (often .1) throughout a network; I inherited a "high" numbered network, but I prefer low).
Isolation VLAN is that into which devices that are "naughty" are placed; again PacketFence will run a DHCP server here. Make sure you document these ranges in your IP Numbering Plan, so you don't end up hitting a nasty surprise several years later...
Add all your NAC-requiring client VLANs. We have quite a few...

Step 2 with many VLANs

Click Continue.
Set up MySQL credentials. Keep them somewhere safe in case you need them. KeePass may be a good idea if you haven't already implemented some other reasonably safe credential storage system. Take a look at point 47 below for some problems you may like to consider if you pick properly random passwords.
Setup the packetfence database
Setup your Packetfence MySQL username and password (store them somewhere safe) - make sure they are strong as if anyone gets in, they control your network access and can wreak merry havok on your SQL data...
(Save your keepass database in case of crash now!) - I've not taken a screenshot of Step 3 for perhaps obvious reasons; the screen has plenty of helpful prompts on it; you won't go wrong if you follow them.
Click Continue
Set up your server's name, domain name and other parameters.

Complete Step 4. Bcrypt is good!
Continue
Change the password for the Admin user. Again, make it something secure! (and again, if you're using Keepass to store non-memorable passwords, save it, back it up...).

Change the Admin Password.
Continue.
Click Start PacketFence

After you click Start PacketFence, the various core services will start up...
You're probably going to want to ensure the server has accurate time. Make sure it can reach NTP servers on the Internet (configure your firewall appropriately). Our upstream ISP blocks NTP (except to their 3rd tier NTP server), so you may need to reconfigure the /etc/ntp.conf file to work in your environment from the packetfence server shell.
Wait until all the services are started. Of course, mine didn't... At this stage, you may need to RTFM. (5.5.0 ZEN deployment guide). RTFMing didn't help much. Eventually, I discovered many of my woes were underlain by a complex MySQL user password probably not being parsed from the configuration files properly.

Mine got stuck. Zero effort they say...?
It turns out (not surprisingly) that PacketFence doesn't like multiple management interfaces. Don't think you're being clever by setting up your "legacy" management VLAN and your "future" management VLAN at this stage... Only set a single VLAN to be (each of) Registration, Management, and Isolation.

More Green Is Good...
See point 47 for some reasons why these last few might not start.
Hint: complex passwords for the packetfence database user might not be a good idea.
Ensure the FQDN you set up in step 19 can be resolved...
After quite a bit of patience, the various services should start. Once they've all started, there ought to be a prompt inviting you to go to the management interface. If they don't, you're going to have to log into the shell and figure out what is going on using logs. The login credentials are giving in the manual.
While you're there, change the root password from the default...!
It's a poor show, but it turns out that Heartbleed still rears its ugly head...

Really? Zero Effort and you haven't patched heartbleed to let a core part of the NAC start...?
Freeradius (radiusd) will refuse to start.
You've got two options: Patch OpenSSL or tell freeradius to just ignore the vulnerability.
Ideally one would patch, and the normal thing to do would be to use the builtin update tools (package manager) - because heartbleed was discovered in April 2014, you'd expect it to be patched by now.
yum update should upgrade all your software to the latest available versions.
Packages are managed with yum in CentOS. For whatever reason, the latest packaged version of OpenSSL for CentOS 6.7 is 1.0.1e, and that was already installed (and might be vulnerable). However, after doing more reading, it seems if the version is 1.0.1e-16 or later it is not vulnerable to heartbleed. So as this is 1.0.1e-42, we're going to override the Radius security setting. If you want to get a later version, you're going to have to compile it yourself or find an RPM somewhere - exercise for the reader!.
To find your version, in a shell, type rpm -q -a | grep "openssl"
You may like to play around with the fastestmirror configuration file to make sure it's getting the fastest possible server (.ac.za are likely to be orders of magnitude faster for me because of SANReN).
Therefore, we're going to tell freeradius to put up with what looks like an unpatched version (but isn't).
Edit the allow_vulnerable_openssl line in the security section of /etc/raddb/radiusd.conf to allow CVE-2014-0160 by changing the no to a yes.
In /etc/init.d/ issue a ./radiusd start command
You will be greeted by a friendly green [ OK ]
Go back to the web configuration screen. It will still tell you radiusd has not started (even though a ps -aux | grep radiusd tells you otherwise).
At this stage, I decide it's time for a reboot. Issue shutdown -r now at a command prompt. Because.
That doesn't help
There are error messages complaining about FQDN not resolving.

Edit /etc/sysconfig/networking so that the HOSTNAME= your actual resolvable FQDN server name.
issue hostname <your FQDN>

Restart networking (service network restart)
If the hostname in the shell prompt changes to the FQDN, carry on. Otherwise redo step 38.
Issue a service packetsense restart - and watch for things that fail. Another configuration error crops up in collectd, which causes radsniff3 to also fail.
At this point, you're probably starting to bang your head against the desk. I know I was. So, skip the wizard...!
Connect instead to http://<IP>:<port>/admin/ and login with the admin credentials you specified earlier.
If things are still wonky, check all the logs in /usr/local/pf/logs/ particularly those relating to services that fail to start; I eventually found RADIUS was chocking on MySQL connection which lead me to the conclusion (and resolution) of the point immediately below.
It turns out that MySQL kind of sucks (at least the implementation where credentials are passed from a config file). If you use complex passwords, it's likely to croak. Just use plain ASCII (letters and number) and you should be OK. If you reset this in MySQL and the two config files (pf.conf and pfconfig.conf) you should find everything launches...

Finally all green!!!

I've been bitten by this before. Quite annoyed it took me this long to remember about complex MySQL user passwords underlying much misery. I strongly suspect it's when you include brackets that chaos breaks loose.
If it doesn't launch, make sure you go over all the various things I discussed already, and, if necessary, start diagnosing the various log files;

/var/log/messages
/var/log/radius/radius.log
/usr/local/pf/logs/
If you've not used linux CLIs before, you will find the commands cat, tail and vim very useful for viewing and editing log and config files. Be very careful what you do, as by default, you login as a super administrator (root); linux assumes you know what you are doing and does not generally (unlike windows) attempt to help you not to shoot yourself in the foot!

Joy! The configuration wizard completes...!

AT LAST!
Click on the Visit Administration Interface Now! button.
Login using the username and password you set up in step 21.

Login prompt
That's the basic installation completed.

The next post will look at actually configuring things to be useful.