Sunday, 16 March 2014

How to do VPN on Demand for iOS at zero cost despite Apple's best efforts to prevent this

Apple support a feature called 'VPN on Demand' despite the best efforts of both Apple and VirnetX to sabotage this. There are three main requirements in order to implement this.
  1. You need to use a supported VPN client which can either be the built-in Cisco IPSec client or a supported SSL VPN client
  2. You need to use certificates for authentication instead of a pre-shared-key
  3. You need to use a MDM (Mobile Device Management) solution to configure and push the settings to the client device (iOS or Mac) including the VPN on Demand settings
With regards to the VPN client it would seem you can therefore use the built-in Cisco IPSec VPN client on your iOS or Mac client, however as we shall see Apple have made this as difficult as possible to achieve. While some techies may prefer an SSL solution and it is certainly the case that Apple's built-in IPSec client is falling behind with regards to newer security standards e.g. IKEv2, I prefer using the built-in client as there is no extra cost for it, nothing extra needs to be installed and in theory being from Apple themselves it will always be compatible with the current OS version.

Before I go on to how to get this working despite Apple's best efforts, why do we want VPN on Demand? This can be for one of two reasons.
  1. To give access to an internal system that is not directly visible on the Internet and to do so in a way that is as seamless and automatic as possible in order to make things as easy as possible for users
  2. or to have a mobile device always route all traffic via a VPN connection so that its traffic is always protected even when using public WiFi hot-spots. By automating and enforcing this you avoid users forgetting to manually establish a VPN connection
The later reason is becoming more and more important these days with not only the threat of cyber-criminals but also Governments snooping on your traffic. So the goal is to have a compatible VPN server, a compatible VPN client and to configure things so that the VPN connection routes all traffic via it, and for the connection to be established completely automatically.

I could have bought a commercial VPN server from Cisco, Juniper, SonicWall, F5, Aruba or CheckPoint but this would have been very expensive. It is not possible to use Apple's own VPN server solution as it only supports PPTP or L2TP, this is despite the fact that Apple use the open-source Racoon software which does support using Cisco IPSec. I initially looked at compiling and installing the standard unmodified Racoon software in OS X but this would risk causing incompatibilities with Apple's own software so my first effort was to install and configure Racoon in an Ubuntu Linux virtual machine using Virtualbox. Total cost £0

I initially and successfully did so with a pre-shared-key (henceforth referred to as PSK) and then added authentication via LDAP to OpenDirectory running on a Mac server. Total cost so far £0

I then setup a self-signed rootCA and server certificate and client certificate using XCA. Total cost so far £0

Note: Apple's built-in VPN client requires that server certificates have the server name also in a subject alternative name field (SAN). While it is possible to do this using the command line openssl tool or Certificate Assistant in Keychain Access, life is made much easier with XCA.

I reconfigured Racoon to use certificates instead of a PSK and likewise the test iPhone I was using initially running iOS 7.0.4. I then hit the first Apple bug. If you modify a VPN configuration on the iOS device changing it from using IPSec with a PSK to IPSec with a certificate, even though all the details are correctly filled in it will fail to connect. This is because even though you have switched from PSK mode to certificate mode which hides the Cisco 'Group' field, it will still try and use the Cisco 'Group' field. As this is irrelevant for a certificate authenticated connection the VPN server will reject the connection due to being very confused. Nothing on the iPhone will give you a clue about this so I had to dig through debug logs on the Linux server to track this down. To work around this you must delete the old VPN configuration and create a fresh one. Once I had cracked this I was able to successfully get the iPhone to connect and use a certificate and username/password to connect. Total cost so far £0 plus a lot of my hair being pulled out.

At this stage we now have a Cisco IPSec compatible VPN server using a certificate, and the iOS device also using a certificate and the username/password being authenticated via OpenDirectory. In theory we are now ready to setup the MDM system. At this point as a diversion I also got StrongSwan working in another Ubuntu Linux virtual machine using the same certificates as another alternative Cisco IPSec compatible server. I did not get it working with LDAP to OpenDirectory at this point because the standard version 4.5.2 of StrongSwan for Ubuntu 12.04 is too old to enable this, but I did get it working with manually defined username/password details. StrongSwan 5 or later would be able to use LDAP authentication.

I already had a copy of Server.app and hence Apple's ProfileManager so I set it up and was able to successfully enrol my test iPhone and push profiles over the air to it. I was using the same self-signed rootCA, plus an additional server certificate but the same client certificate. So far however I am unable to get ProfileManager to push this certificate to the iPhone. As far as I am concerned there is nothing wrong with the certificates as the same client certificate works when emailed to the iPhone directly, and works for the VPN connection from the iPhone, and what's more works when pushed over-the-air from the free Cisco Meraki Systems Manager MDM solution. I therefore gave up on ProfileManager and switched to the free Cisco Meraki Systems Manager. I again enrolled the iPhone and created a profile in Systems Manager to push the client certificate to the iPhone and to also push a VPN configuration set to use that client certificate with a Cisco IPSec connection. This worked with a another major issue still to be resolved as we will see. Total cost so far £0 and the loss of some more hair.

Note: This certificate problem with ProfileManager may be a second Apple bug.

Having now successfully got a Cisco IPSec compatible VPN server working with certificates and successfully setup an MDM solution to push the settings to the iPhone I tested manual connections to the VPN server. This worked and I was able to enter and save and use the same OpenDirectory user credentials. However when I turned on VPN on Demand as part of the profile being pushed to the iPhone I discovered that the iPhone would not save the password, it would not let me edit the VPN configuration, there was also no way in the profile to define the password. As a result every time the VPN connection was told to connect I was asked again to re-enter the password. It turns out this is how currently it is 'supposed' to work. This makes it in not only my but many other peoples opinions an unusable solution for VPN on Demand. I personally consider this to be yet another Apple bug. (In Microsoft speak this would be described as a feature.)

Note: My Ubuntu Racoon VPN server had been configured to allow clients to save passwords and this had already been proven to work with manually configured VPN settings on the same iPhone.

I tried various options like an account with an empty password, not entering a username or %short name% in the VPN profile, even creating and manually editing a mobileconfig file with Xauth disabled. None of these options worked. It was beginning to look like I would have no choice but to pay for a commercial SSL VPN server.

I had when initially investigating this from the angle of 'how to do you save the password' found some discussions from other equally annoyed and frustrated Apple customers regarding this but none of these had listed a solution. By now I had gone as far as asking for a quote for an SSL VPN solution, however I did then find mention of something called Xauth-noauth for StrongSwan.

See http://serverfault.com/questions/476033/strongswan-without-password-on-ios

I found this when I switched to searching for how to connect without a username and password. As you can see other people have hit the same problem as I had. Fortunately someone (ecdsa) had come up with a solution which was to write a special Xauth module for use with StrongSwan which effectively ignores the Xauth details leaving just the certificate authentication active. With this the iPhone no longer asks for a password as it has not been asked by StrongSwan for one.

Note: There are normally two stages to authenticating a VPN connection, the first stage is either via a PSK or by exchanging and verifying SSL certificates, the second stage is eXtended AUTHentication or Xauth and is where the username and password are normally exchanged and verified.

I therefore dusted off my StrongSwan configured virtual machine. Unfortunately this Xauth-noauth module is only for StrongSwan 5.0.2 or later and as previously mentioned the standard version of StrongSwan for Ubuntu 12.04 is the much older and substantially different 4.5.2. As a pre-built StrongSwan 5 or later was not available for Ubuntu 12.04 I had to download the source code for StrongSwan 5.1.2 and build it myself along with all its dependencies. I was however able to do this and install it and thankfully it did exactly what was wanted and no longer asked the iPhone for a username and password. I was therefore able to use the VPN profile from the Systems Manager MDM with VPN on Demand enabled and the iPhone was now able to repeatedly connect without nagging me each time for the password! Yippee! Total cost so far - drum roll please - £0

Building StrongSwan 5.1.2 from scratch

First install dependencies

sudo apt-get install libpam0g-dev
sudo apt-get install libcurl4-openssl-dev
sudo apt-get install libcurl4-nss-dev
sudo apt-get install libldap2-dev
sudo apt-get install libgmp3-dev

Then download and uncompress source code

wget http://download.strongswan.org/strongswan-5.1.2.tar.bz2
tar -xjvf strongswan-5.1.2.tar.bz2

Then configure what modules to enable and compile and install (the configure command is all on a single line)

cd strongswan-5.1.2

./configure --prefix=/usr --sysconfdir=/etc --enable-curl --enable-ldap --enable-pkcs11 --enable-md4 --enable-openssl --enable-ccm --enable-gcm --enable-farp --enable-eap-identity --enable-eap-aka --enable-eap-aka-3gpp2 --enable-eap-md5 --enable-eap-gtc --enable-eap-mschapv2 --enable-eap-dynamic --enable-eap-radius --enable-eap-tls --enable-eap-ttls --enable-eap-peap --enable-eap-tnc --enable-xauth-eap --enable-dhcp --enable-charon --enable-xauth-pam --enable-xauth-noauth

sudo make
sudo make install

Here is the /etc/ipsec.conf I used for StrongSwan 5

# basic configuration

config setup
uniqueids=never

# Add connections here.

conn %default
authby=rsasig
leftrsasigkey=%cert
rightrsasigkey=%cert
keyingtries=1
keylife=60m
ikelifetime=240m

conn ios
keyexchange=ikev1
left=%defaultroute
leftsubnet=0.0.0.0/0
leftfirewall=yes
leftcert=serverCert.pem
right=%any
rightsubnet=10.0.1.0/24
rightsourceip=10.0.1.0/24
leftauth=rsa
rightauth=rsa
rightauth2=xauth-noauth
ike=aes128-sha1-modp2048,3des-sha1-modp1536
esp=aes128-sha1-modp2048,3des-sha1-modp1536
rekey=no
reauth=no
dpddelay=10
dpdtimeout=30
dpdaction=clear
auto=add

Update - 2014-09-13
After successfully running this StrongSwan VPN solution for several months with iPhone and iPad devices I discovered that users were unable to successfully connect over O2 cellular data connections, they were still able to connect over EE data, or WiFi connections. Narrowing down the circumstances was made more difficult by eliminating things like poor data connections, running out of data allowance, international roaming, confirming whether they  they were using WiFi or not. Clearly there was something different about the O2 network upsetting things as the EE and WiFi connections used the same settings and even the same devices. After spending sometime investigating this it would seem that the O2 data network has problems with the size of packets used. For example if you connect using a pre-shared-key (which works) this will be a fairly small packet as the pre-shared-key is a straight forward password, however when you connect using certificates a copy of the certificate has to be sent and this is much bigger and did not work. Fortunately this has been easy to fix, StrongSwan has a command called fragmentation that enables handling splitting larger packets in to smaller ones and this has solved this problem. To use this you need to add the following line to the /etc/ipsec.conf file as shown above.

fragmentation=yes

The last step was tweaking the rules for VPN on Demand. Both Apple's ProfileManager and Cisco Meraki System Manager have very limited options for VPN on Demand that can be configured in their GUI interfaces. In fact they are so limited that effectively they can only accomplish option 1 I listed at the beginning which is to automate providing access to an internal server, and not to automate connecting via VPN all the time for all traffic. Fortunately Apple do document how to manually configure this. Yes I know it is a big shock but Apple does still provide some useful documentation sometimes.

See page 40 https://developer.apple.com/library/ios/featuredarticles/iPhoneConfigurationProfileRef/iPhoneConfigurationProfileRef.pdf

Update 2017-01-11 it has been brought to my attention that the above link no longer works, that is Apple have removed that page. The nearest equivalent currently working link I can find is https://developer.apple.com/library/content/featuredarticles/iPhoneConfigurationProfileRef/Introduction/Introduction.html#//apple_ref/doc/uid/TP40010206-CH1-SW27

I therefore used iPhone Configuration Utility to build a mobileconfig profile containing the client certificate and the VPN settings, I then exported it and edited this as per Apple's documentation above to use the URLStringProbe option to check for the ability to access a URL on my VPN server and if found to be true to trigger a VPN connection. This might be opposite of what you would expect but it is the correct choice to force a connection always to happen and by doing it this way if my server is down the URL test will fail and VPN on Demand will not be activated and the device will still be usable although not protected by a VPN connection. You therefore need to use a URL that is accessible without a VPN connection. I then pushed this out to the iPhone using Systems Manager. With this installed my iPhone now as soon as it gets an Internet connection will automatically trigger a VPN connection which will route all traffic over it and all without any user interaction being needed. Grand total cost a massive £0!

As a final aside, all the above should apply equally to using Mac clients, testing this shall be my next project.

Links to Resources

VirtualBox or ESXi 5.1
Ubuntu 12.04 Server
StrongSwan 5.1.2
Cisco Meraki Systems Manager
XCA
iPhone Configuration Utility 3.5 for Mac

24 comments:

  1. That is some awesome investigative IT work there!

    ReplyDelete
  2. Hi John!

    This is an amazing job.

    But I want to ask you one question.

    You wrote: "I therefore used iPhone Configuration Utility to build a mobileconfig profile containing the client certificate and the VPN settings, I then exported it and edited this as per Apple's documentation above to use the URLStringProbe option to check for the ability to access a URL on my VPN server and if found to be true to trigger a VPN connection".

    Is possible next scenario: I build mobileconfig profile and publish it on some URL; then when app starts, it takes this profile and applied? Main goal - all works should be in automated mode for final user. He has no need to go to Settings and make some manipulations for activation of VPN. He simply runs the app and enjoys of life with VPN :)

    If it possible, can you publish some link for learning this question, because I don't know how to apply mobileconfig profile (or maybe how run it)?

    Thank you for any help you can provide in this question

    ReplyDelete
  3. Action Connect it is working fine in iOS 7 but when VPN server is down the device has to use public internet so I turned it like this Action ConnectURLStringProbe https://www.google.com But it doesn't even establishing VPN connection.Any help is appreciated !

    ReplyDelete
    Replies
    1. The "Action, Connect, URLStringProbe" means that if the probe is successful it will initiate an attempted VPN connection and will keep trying to do so as long as the probe reports a success.

      So if you set it to probe to www.google.com that is likely to always going be online and it will therefore always keep trying to initiate the VPN connection even if the VPN server is down.

      You want instead to have it look for a URL on the VPN server, then if the VPN server is down the probe will fail and it will revert to using the public Internet.

      A simple hello webpage is sufficient for this and can and should be run in the same Linux server. You can use Apache in Linux to do this. It does mean you need to open a port for the webserver through any firewall.

      Delete
  4. Great post. A couple of questions:

    1] You mentioned using xauth-noauth. I've been using that for the same reason. However, today I just read that there is an XAuthEnabled=0 setting

    https://developer.apple.com/library/ios/featuredarticles/iPhoneConfigurationProfileRef/Introduction/Introduction.html

    Have you tried this? If so could you share your experiences?

    2] since I started using xauth-noauth I've noticed that IOS clients can sometimes get in a wedged state when switching networks. This didn't seem to happen at all previously. Have you noticed anything like this?

    ReplyDelete
    Replies
    1. It's been a long while now since I originally did all this work but I do seem to recall seeing that option back then. I vaguely recall trying it by itself and because the VPN server was then still asking for XAuth it did not help. So it might be worth trying along with xauth-noauth but probably not without. One needs to ensure the VPN server is not asking for Xauth as well as ensuring the client does not send it.

      Have you added the fragmention=yes entry to your ipsec.conf? I found after a while that the O2 mobile data network had a problem with the packet size needed to send SSL certificates but EE and normal broadband did not, when changing network connections you maybe changing to a network that hits this.

      If you use the Contact form it maybe easier to discuss this more directly.

      Delete
  5. Great work. However this sounds very complicated even for a knowledgeable software engineer like myself. Is there any way this could be simplified? Perhaps you could even monetize this.

    ReplyDelete
    Replies
    1. Once the VPN server is setup which is perhaps the more difficult part, adding additional devices is merely a matter of generating a new client certificate, creating a mobileconfig file (I use iPhone Configuration Utility) and using an MDM to push it out to the iOS device.

      I agree it is always nice to have a simpler solution and I have found all the gotchas and listed solutions. Unfortunately if one wants a free solution one often has to put more work in. There are plenty of existing commercial VPN solutions which will do more of the work for you and would be proven working solutions, they do however cost a considerable amount, e.g. Cisco, SonicWALL (aka. Dell), Juniper and others.

      Delete
  6. I would certainly pay for an iOS on demand VPN solution.

    ReplyDelete
  7. You mentioned pushing this out via Meraki System Manager. What setting did you select? How did you push this URL out?

    ReplyDelete
  8. As mentioned above, I used iPhone Configuration Utility to create a mobile config file. Like Apple Configurator, Profile Manager and I believe most/all MDM solutions this only has built-in very limited VPN on Demand rules that you can define. I then hand-edited the mobile config file using Text Wrangler. The mobile config file is just an XML file which can be edited. I was then able to replace the VPN on Demand rules section with a rule that better matched my requirements.

    As per my article above again, the Apple documentation lists the different VPN on Demand rules you can define - including ones which as mentioned you cannot do via iPhone Configuration Utility itself.

    Once the mobile config file was complete I then told Meraki Systems Manager to use it to create an MDM Profile by using the mobile config file which I uploaded to Meraki Systems Manager. I then told Meraki Systems Manager to only push that profile to a single specific device - the one it is intended for by setting which tags to match against. I add an extra tag for each device and use the devices serial number for the tag so as to ensure it is unique and simple to follow.

    The mobile config file therefore contains the client device SSL certificate as a p12 which includes the client devices private key, the in my case self-signed rootCA certificate as a pem, the VPN settings i.e. address and type, and the hand edited rule for VPN on Demand.

    ReplyDelete
    Replies
    1. Hi John, Thanks for all the insights. However, when I embedded the p12 into the mobileconfig, my iphone prompts me for a password. This is on IOS 7. Did you face such an issue. I added a dummy password into the XML, but it does not work.

      Delete
    2. There are two different password uses you could be referring to. Firstly the .p12 file itself should normally be password protected, this is ok as when you add the .p12 to the mobileconfig file you can specify the password in the mobileconfig file to unlock the .p12 file.

      The other possible password use is for allowing the user of the iPhone or iPad to 'login' to the VPN server. Now here is where it gets tricky, normally you cannot add a user password like this to the mobileconfig file. Certainly iPhone Configuration Utility, Apple Configurator and Profile Manager do not allow this. This is why I chose to use Xauth-noauth on the StrongSwan server, this is a special dummy Xauth module which does not ask the device for a user name and password. However if you hand edit the XML in the mobileconfig file you can add a user password to it as follows.

      (I had to inert spaces in to the XML below to get it to be accepted in this reply.)

      < key>XAuthPassword< /key>
      < string>myVPNuserPassword< /string>

      This would follow the standard

      < key>XAuthEnabled< /key>
      < integer>1< /integer>
      < key>XAuthName< /key>
      < string>VPN< /string>

      Obviously you would put your own values in.

      Either using Xauth-noath or hand editing the mobileconfig file will solve the problem of the iOS device otherwise having to repeatedly ask you for the password. However each has their own disadvantages. Using Xauth-noauth means it is slightly less secure since you are turning off one type of security, hand editing the mobileconfig file means the password has to be stored as plain text in that file and if the password is changed by the user you have to generate a new mobileconfig file with the new password.

      Delete
  9. This comment has been removed by a blog administrator.

    ReplyDelete
    Replies
    1. Your comment appeared to be an advert for a commercial VPN service and I have previously been pestered by a large number of such adverts masquerading as comments. The whole point of this article was not to create a standard VPN server for iOS users but to specifically create a VPN on Demand solution more appropriate for corporate environments and this is something non of those typical paid-for VPN services offer.

      I apologise if I have unfairly considered your post as such a 'advert'.

      Delete
  10. This comment has been removed by a blog administrator.

    ReplyDelete
  11. I also need VPN-on-demand, I have been going through a similar exercise to you (great job, by the way). I have tried StrongSwan using IKEv2, I have been able to connect from my iPad successfully using either PSK or certificate authentication. I am now having trouble where Apple push notifications do not reach my iPad whilst the VPN is active. Does this work for you with your setup? Did you have to configure anything particular for that? TIA

    ReplyDelete
    Replies
    1. Interesting question. I have not seen any issue myself or had this reported by my users. What makes you think push notifications are not working?

      As far as I can see the following should happen.

      If not connected to the VPN then push notifications should go from Apple directly to the phone, this would then I suspect trigger Internet activity by the phone why in turn would trigger the 'VPN on Demand' to connect to the VPN.

      If already connected to the VPN the way I have mine configured is to route all traffic via the VPN connection, this means the 'public' IP address of the phone becomes the public IP of the office network. As long as the office network allows push notification traffic this should then still work but like all the other Internet traffic would flow via the office and the VPN connection.

      It might be worth doing a test using a Mac on the office network to confirm push notifications work there - this would not involve the VPN. See this free tool for a way to test push notifications - http://twocanoes.com/products/mac/push-diagnostics

      Delete
    2. Thanks. I'm testing push notifications to my iPad by sending email to my gmail account which is set up on the iPad - normally, an alert is shown for new emails, however when connected via VPN consistently I get no email alerts.

      We want to use our VPN as an internet gateway, it does not allow access to an office network. I tried that tool, push notifications to our Mac were successful outside the VPN, and were also successful when connected via the VPN. So it does seem to be an iOS-specific problem.

      If you are able to verify that push notifications are received by iOS devices connected to your VPN server, that would be really helpful to me. I'm still hopeful that there is some "magic" StrongSwan configuration option that I can enable that will make this work for me.

      Delete
    3. If your using the built-in mail client on the iPad all you see is the unread message count increase and either a beep or a vibrate. We use Exchange Active Sync for mail on iPhones/iPads and I know that works fine via the VPN and EAS is a push system although it uses its own push system rather than Apple's.

      In fact now that I think about Gmail would not use Apple Push Notifications either unless that is a feature of a Gmail specific client rather than the built-in client.

      LinkedIn is a good app to test Apple push notifications with. It will show messages on screen, in the notifications centre, and as a number (badge) on the icon.

      Delete
  12. This comment has been removed by a blog administrator.

    ReplyDelete
  13. This comment has been removed by a blog administrator.

    ReplyDelete