VPN overview

A virtual private network secures public network connections and in doing so it extends the private network into the public network such as internet. With a VPN you can create large secure networks that can act as one private network.

(picture from wikipedia)

Companies use this technology for connecting branch offices and remote users (road warriors).

Ixolate supports VPN connections for branch offices as well as remote users.

Creating a single secured private network with multiple branch offices connecting to a single site can easily be setup from within the graphical user interface. For remote users, certificates can be created and revoked and a simple to use export utility makes the client configuration a breeze.

Ixolate offers a wide range of VPN technologies ranging from modern SSL VPNs to well known IPsec as well as WireGuard and Zerotier via the use of plugins.

IPsec

Since IPsec is used in many different scenario’s and sometimes has the tendency to be a bit complicated, we will describe different usecases and provide some examples in this chapter.

General context

The IPsec module incorporates different functions, which are grouped into various menu items. Since the start of our project we have been offering IPsec features based on the legacy ipsec.conf format, which we are migrating to swantcl.conf as of version 23.1. While migrating the existing featureset we came to the conclusion that the world has changed quite a bit and in order to offer better (api) access to the featureset available we decided to plan for deprecation of the legacy “Tunnel settings” as they have existed since we started. No timeline has been set, only a feature freeze on tunnels using the “Tunnel settings” menu item.

One of the main goals for the long run is to better align the gui components so they reflect the reality underneath, as we use strongswan, our aim is to follow their terminology more closely than we previously did.

The following functions are available in the menu (as of Ixolate 23.1):

• Connections
  • New configuration tool offering access to the connections and pools sections of the swanctl configuration
• Tunnel Settings
  • Legacy IPsec configuration tool
• Mobile Clients
  • Offering access to various options of the attr plugin and pool configurations for legacy tunnels
• Pre-Shared Keys
  • Define secrets to be used for local authentication.
• Key Pairs
  • For public key authentication collect public and private keys.
• Advanced Settings
  • Define passthrough networks (to exclude from kernel traps), logging options and some generic options
• Status Overview
  • Shows tunnel status
• Lease Status
  • For mobile clients, show address leases for various pools configured
• Security Association Database
  • Shows security associations, the fundamental concept of IPsec describing a relationship between two or more entities
• Security Policy Database
  • Installed security policies describing which traffic is allowed to pass a tunnel
• Virtual Tunnel Interfaces
  • Edit or create new if_ipsec(4) interfaces and show the ones created by legacy tunnels
• Log File
  • Inspect log entries related to IPsec

Migrating from tunnels to connections

Having used the tunnel settings from the early Ixolate days, some terminology might be a bit confusing when moving into the new options offered. This paragraph aims to explain some of the common terms from the tunnel section and their new place in the connections. For a full list of changes, the upstream migration documentation is an interesting read as well.

• Phase 1 - The general connection settings, like local/remote addresses and general protocol settings. Choices in authentication to use are also part of this, they may involve multiple rounds.
• Phase 2 - Nowadays Strongswan calls these children, as these define the CHILD_SA subsections in play. This is where you can define the networks on both ends. When multiple segments are being added into the same child, these are being treated as one policy where all of them are able to communicate to each other.
• Phase 1 / Tunnel Isolation - This option made sure every network defined in phase 2 would be treated as a child of it’s own (e.g. two phase 2’s would turn into two children)
• Phase 2 / Manual SPD entries - Manual SPD entries, this has been replaced with it’s own menu option (Security Policy Database) offering more flexibility and visibility.

Since Ixolate uses the new Strongswan format also for legacy tunnels, it is rather easy to convert a tunnel manually when downloading the swanctf.conf file from the machine. You can download it via VPN › IPsec › Advanced Settings › swanctl.conf. The format is almost identical to the connections gui available in Ixolate.

Let’s take a look at the following swanctl.conf that contains a legacy con1 tunnel:

# This file is automatically generated. Do not edit
connections {
    con1 {
        unique = replace
        aggressive = no
        version = 2
        mobike = yes
        local_addrs = 203.0.113.1
        local-0 {
            id = 203.0.113.1
            auth = psk
        }
        remote-0 {
            id = 198.51.100.1
            auth = psk
        }
        encap = no
        remote_addrs = 198.51.100.1
        proposals = aes256-sha256-modp2048
        children {
            con1 {
                start_action = start
                policies = yes
                mode = tunnel
                sha256_96 = no
                local_ts = 192.168.1.0/24
                remote_ts = 172.16.100.0/24
                reqid = 1
                esp_proposals = aes256-sha256-modp2048
            }
        }
    }
}
pools {
}
secrets {
    ike-p1-0 {
        id-0 = 198.51.100.1
        secret = 0saGVsbG93b3JsZA==
    }
}

To convert this configuration:

• Go to VPN › IPsec › Connections
• Press + to add a new Tunnel and enable the advanced mode

Connections

Press Save, then move to Local Authentication and press +

Local Authentication

Press Save, then move to Remote Authentication and press +

Remote Authentication

Press Save, then move to Children and press +

Children

Press Save, then Save again to store the new Connection.

Now we must configure a Pre-Shared Key that matches the Local Authentication and `Remote Authentication`:

• Go to VPN › IPsec › Pre-Shared Keys
• Press + to add a new Pre-Shared Key

Pre-Shared Key

After the initial configuration, go to VPN › IPsec › Tunnel Settings and deactivate the con1 tunnel by unchecking Enabled in Phase1 and Phase2.

Go back to VPN › IPsec › Connections and ensure the new tunnel is Enabled. Press Apply and the new configuration will be active.

Combining legacy tunnels and connections

It is possible to combine tunnels and connections, but there are some constraints. As our legacy tunnels force a reqid for each configured child (phase 2), there is a risk the automated numbering from the new connection children overlaps. To prevent these overlaps, its required to set an unused reqid in the connection child.

Security policies and routing

In order to pass traffic over an IPsec tunnel, we need a policy matching the traffic. By default when adding a phase 2 (or child) policy a “kernel route” is installed as well, which traps traffic before normal routing takes place.

Firewall rules

When using the legacy tunnels and Disable Auto-added VPN rules is not checked in VPN › IPsec › Advanced Settings some automatic firewall rules are created for remote hosts connecting to this one. The new connections feature does not offer this and (WAN) rules have to be specified manually in order to connect to IPsec on this host.

The relevant protocols and ports for IPsec are the following:

• Protocol: ESP (<https://en.wikipedia.org/wiki/IPsec#Encapsulating_Security_Payload>)
• Port: 500/UDP (<https://en.wikipedia.org/wiki/Internet_Security_Association_and_Key_Management_Protocol>)
• Port: 4500/UDP (<https://en.wikipedia.org/wiki/NAT_traversal#IPsec>)

The default behavior of our firewall is to block inbound traffic, which also means traffic using the tunnel should be allowed explicitly, the Firewall › Rules › IPsec menu items offer access to the IPsec traffic policies.

Dead Peer Detection (DPD)

Dead Peer Detection (DPD) is a method of detecting a dead IKE peer as specified by RFC 3706.

When a peer is assumed dead, an action may be specified, such as closing the CHILD_SA or re-negotiate the CHILD_SA under a fresh IKE_SA.

When using IKEv1 a dpdtimeout can be specified to control when a peer is considered to be inaccessible. This setting has no effect on how IKEv2 handles retransmissions, in which case the general settings will be used as specified in the following document.

Implementation schemes

When setting up IPsec VPNs there are two main types of scenario’s with their own advantages and disadvantages.

Policy based

The first one is the standard policy based tunnel, which guards the security of the tunnel with policies and installs kernel traps to send traffic over the tunnel in case it matches these policies. For example a local network 192.168.1.0/24 sending traffic to a remote location responsible for 192.168.2.0/24. The advantage of this scenario is the ease of setup, no routes are needed to be configured, when in this example 192.168.1.10 contacts 192.168.2.10 the packets are seamlessly forwarded over the tunnel to the remote location.

When local traffic doesn’t match the policies in question due to the tunnel needing Network Address Translation, that’s also possible as long as policies are manually added to the security policy database, this is also referred to as “NAT before IPsec”.

Route based (VTI)

Route based, also known as VTI, tunnels are using a virtual interface known as if_ipsec(4), which can be found under VPN › IPsec › Virtual Tunnel Interfaces. This links two ends of the communication for routing purposes after which normal routing applies. The “(Install) Policies” checkmark needs to be disabled in this case for the child (phase 1 in the legacy tunnel configuration) definition. Usually the communication policy (phase 2 or child) is set to match all traffic (either 0.0.0.0/0 for IPv4 or ::/0 for IPv6).

So the same example as the policy based option would need (static) routes for the destinations in question (192.168.1.0/24 needs a route to 192.168.2.0/24 and vice versa), peering happens over a small network in another subnet (for example 10.0.0.1 <-> 10.0.0.2) bound to the tunnel interface.

The advantage of this type of setup is one can use standard or advanced routing technologies to forward traffic around tunnels.

Road Warriors / Mobile users

IPsec may also be used to service remote workers connecting to Ixolate from various clients, such as Windows, MacOS, iOS and Android. The type of client usually determines the authentication scheme(s) being used.

In case clients should be offered default settings, these can be configured from VPN › IPsec › Mobile Clients. Pool options (Virtual IPvX Address Pool) on this page will be used by the legacy tunnel configuration only, when using the new connections module one may configure different pools per connection.

The examples section contains various options available in Ixolate. When using the new “connections” option available as of Ixolate 23.1, different examples from Strongswan are usually quite easy to implement as we follow the swantcl.conf format quite closely in the new module.

Examples

This paragraph offers examples for some commonly used implementation scenarios.

New > 23.1 (VPN › IPsec › Connections)

Legacy (VPN › IPsec › Tunnel Settings)

The following client setup examples are available in our documentation:

CARP considerations

When using IPsec in a high availability setup, it is important to understand the implications of the setup. Without assuming what the remote gateway looks like (which may be a single device or a high availability setup as well), the following considerations should be taken into account:

• For IKEv2, MOBIKE should be disabled. Due to the nature of CARP, a virtual IP in backup state will “disappear”, which will trigger MOBIKE to try to re-establish the connection from a different available IP, thus overriding your “Local address” configuration. In a lot of cases this will be the primary IP of the WAN interface.
• In all cases (initiator, responder or both) the “Local Address” must be set to a CARP virtual IP.
• DPD must at least be configured on the peer to detect a non-responsive peer and reauthenticate the connection. DPD is usually the limiting factor in failover response time and is therefore the primary functionality to adjust to allow for faster failover. See the DPD section for more information and constraints.
• IPsec connections never failover seamlessly between primary and backup and always need a fresh IKE_SA. If quicker failover is required, dynamic routing with route-based tunnels is likely a better solution.

Tuning considerations

In normal routing/firewalling scenarios, most significant performance gains in terms of throughput are achieved by distributing flows across multiple CPU cores. In any physical or virtual setup, flow distribution is often determined by hardware RSS on a NIC, or virtual queues emulating this behavior. Such flows are processed by default on the receiving CPU (direct dispatch via netisr), making sure that traffic is processed in parallel. The distribution mechanism (before any OS processing) making this possible often hashes the IP/TCP/UDP 5-tuple to determine on which CPU a flow should be handled. Proper distribution therefore requires proper uniqueness of packet headers.

When any tunneling takes place, as is the case for IPsec with ESP flows, many different traffic flows may be encapsulated inside a single IP header, reducing uniqueness of traffic and therefore the ability to distribute work. Unless NICs or virtualized queues can distribute work on ESP packets, which is rarely the case, Ixolate firewalls can often only receive such tunneled flows on a single CPU, because the outer IP header is often the same.

If you have a simple, single-tunnel, single-flow setup (two flows for in-out), you may benefit from using async crypto to dispatch cryptographic work to multiple processors. However, there is no guarantee this will speed up work as it is dependent on multiple factors.

In order to do so, add or change the following tunable in System › Settings › Tunables:

As soon as an IPsec packet is received, decryption must be handled first before anything meaningful can be done with the actual data. This process can be parallelized with the command above, but because IPsec packets need to be processed in an explicit order, these have to be re-injected in the same one, often making performance gains negligible - especially when the machine is busy. The FreeBSD kernel queues decapsulated IPsec packets at all times, in contrast to regular ethernet frame handling (see technical note below). This also means that IPsec flows must adhere to the default single thread limitation after decryption. To fan out the traffic to multiple CPUs after decryption, you can increase the amount of threads bound to netisr:

You can also be more specific and set only N amount cores to be used for ipsec traffic processing.

In very busy environments, this change likely has the most impact.

Lastly, but less practical, you can scale up IPsec by ensuring distribution earlier on in the chain by spreading out more flows across multiple IKE SAs (different source/destination selectors). In this scenario you must ensure multiple ESP flows coming from different source IP addresses when configuring site-to-site tunnels. You must combine this with the net.isr.maxthreads setting to keep the traffic distributed while it is routed through the system.

Technical background

Netisr can be thought of as the conceptual equivalent of Receive Packet Steering (RPS) in Linux. How packets are distributed throughout the system is dependent on the packet protocol.

Netisr allows for two basic modes of operation: direct dispatch or deferred (queued) dispatch. Netisr is always single-threaded by default in deferred dispatch mode to make sure the system adheres to the strong ordering requirements in protocols.

In direct dispatch mode, the network driver handling the packet receive buffer moves the packets up the stack. If the packet is to be routed out via a different interface, this entire process is handled in the same thread (including decryption for IPsec, unless async crypto is set), which doesn’t exit until the packet is routed out. During this time, the receive buffer may fill with new packets. In this mode, the single-threaded limitation doesn’t apply as the execution context remains on the receiving thread, of which there are often multiple via the data channels provided by the NICs/drivers.

In deferred dispatch mode, the first protocol allowing deferred processing will queue the packet, at which point the driver input routine will exit immediately, so it can process new packets faster. This decoupling allows netisr to control which flows are executed on which threads (and by extension, CPU).

When it comes to IPsec, packets are always queued via netisr after decryption. Because netisr is single-threaded in this mode, it’s always queued to the same separate thread. The maxthreads option overcomes this limitation. Furthermore, the SPI value is used to calculate which thread the flow ends up on. This is true for both VTI- and policy-based configurations.

Miscellaneous variables

Path MTU Discovery

When trying to enforce path mtu discovery (PMTU), you need to make sure packets leave the network with the DF set. The kernel offers a tunable net.inet.ipsec.dfbit which offers 3 options, 0, clear the bit on packets leaving the firewall (default), 1, set the DF bit or 2 to copy the bit from the inner header.

Diagnostics

In order to keep track of the connected tunnels, you can use the VPN › IPsec › Status Overview to browse through the configured tunnels.

The VPN › IPsec › Security Policy Database is also practical to gain insights in the registered policies, when NAT is used, the additional SPD entries should be visible here as well.

When troubleshooting problems with your firewall, it is very likely you have to check the logs available on your system. In the UI of Ixolate, the log files are generally grouped with the settings of the component they belong to. The log files can be found in the “Log file” menu item.

Custom configurations

In some (rare) cases one might want to add custom configuration options not available in the user interface, for this reason we do support standard includes.

While the swanctl.conf and the legacy ipsec.conf configuration files are well suited to define IPsec-related configuration parameters, it is not useful for other strongSwan applications to read options from these files. To configure these other components, it is possible to manually append options to our default template, in which case files may be placed in the directory /usr/local/etc/strongswan.Ixolate.d/ using the file extension .conf

IPsec configurations are managed in swantcl.conf format (as of 23.1), merging your own additions is possible by placing files with a .conf extension in the directory /usr/local/etc/swanctl/conf.d/.

OpenVPN (SSL VPN)

One of the main advantages of OpenVPN in comparison to IPsec is the ease of configuration, there are fewer settings involved and it’s quite simple to export settings for clients.

General context

The OpenVPN module incorporates different functions to setup secured networks for roadwarriors and site to site connections. Since the start of our project we organized the openvpn menu section into servers and clients, which actually is a role for the same OpenVPN process. As our legacy system has some disadvantages which are difficult to fix in a migration, we have chosen to add a new component named Instances in version 23.7 which offers access to OpenVPN’s configuration in a similar way as the upstream documentation describes it. This new component will eventually replace the existing client and server options in a future version of Ixolate, leaving enough time to migrate older setups.

The following functions are available in the menu (as of Ixolate 23.7):

• Instances
  • New instances tool offering access to server and client setups
• Servers
  • Legacy server configuration tool
• Clients
  • Legacy client configuration tool
• Client Specific Overrides
  • Set client specific configurations based on the client’s X509 common name.
• Client Export
  • Export tool for client configurations, used for server type instances
• Connection Status
  • Show tunnel status
• Log File
  • Inspect log entries related to OpenVPN

Public Key Infrastructure (X.509)

OpenVPN is most commonly used in combination with a public key infrastructure, where we use a certificate authority which signs certificates for both server and clients (Also know as TLS Mode). More information about this topic is available in our Trust section.

Firewall rules

To allow traffic to the tunnel on any interface, a firewall rule is needed to allow the tunnel being established. The default port for OpenVPN is 1194 using protocol UDP.

After communication has been established, it’s time to allow traffic inside the tunnel. All OpenVPN interfaces defined in Ixolate are grouped as OpenVPN.

High availability [CARP]

When operating an OpenVPN server, there’s not much needed to allow an active/passive setup for your environment other then using a virtual (CARP) address. As the server will stop receiving traffic when the virtual address doesn’t it, the backup will eventually become out of service automatically.

In client mode, the OpenVPN instance needs to stop trying to reconnect when it’s not in MASTER mode, the legacy client module shutsdown all instances directly attached to the interface. Our new instances module allows to select the vhid to track. In most cases an explicit bind isn’t needed for a client, the default for a client is to use the nobind option.

Examples

This paragraph offers examples for some commonly used implementation scenarios.

Legacy (VPN › OpenVPN › Client|Server)

New (VPN › OpenVPN › Instances)

Client Specific Overrides

The mechanism of client overrides utilises OpenVPN client-config-dir option, which offer the ability to use specific client configurations based on the client’s X509 common name.

It is possible to specify the contents of these configurations in the gui under VPN › OpenVPN › Client Specific Overrides. Apart from that, an authentication server (System › Access › Servers) can also provide client details in special cases when returning Framed-IP-Address, Framed-IP-Netmask and Framed-Route properties.

A selection of the most relevant settings can be found in the table below.

Client Specific Overrides

Troubleshooting common issues

The most common causes for non functional overwrites are caused by mismatches, in order to debug these, make sure to check the logs for messages like the following:

• Locate overwrite for 'XXX' using server 'XXX' (vpnid: XXX) << trying to find an overwrite (user authentication))
  • Usually followed by user 'XXX' authenticated using 'XXX' XXX showing username, authenticator used and optionally the overwrite type and filename.
• client config created @ XXX << file written on client connect (without user authentication)
• unable to write client config for XXX, missing target filename << no matching overwrite found (without user authentication)

By default overwrites are matched by certificate common name, when Force CSO Login Matching (legacy) or Username as CN (instances) are set the username will be used instead.

Wireguard

General context

WireGuard® is a simple yet fast and modern VPN solution, which in some cases is more convenient than IPsec or OpenVPN, certainly in terms of options you need to configure. In our experience IPsec is the fastest solution for site-to-site connections, but Wireguard is the simplest option to setup.

A wireguard setup on our end exists of the following main components:

• Instances: in the wireguard configuration these are called “interfaces” and they describe how the virtual wgX device on our end is configured in terms of addressing and cryptography.
• Peers: these are the clients that are allowed to connect to us, described by their optional remote address including the networks that are allowed to pass through the tunnel. Peers belong to one or more instances.

Instances

In order to configure an instance, we start by adding one in the gui and generate a keypair. The public key is usually required for the other end of the tunnel (peer). An unused port to listen on is required as well. The tunnel addresses are configured on the wgX device (which is always visible in Interfaces › Overview).

By default, when “Disable routes” is not set, routes are created for each connected peer to the networks selected in “Allowed IPs”, optionally only a single gateway route might be configured as well.

Peers

Peers define the hosts that we exchange information with, which might be a road-warrior type or a static destination, in which case you either provide or omit an “Endpoint Address and Port”. At minimum you need the public key of the other party, optionally you may offer a pre-shared key as additional security measure. The “Allowed IPs” define the networks that are allowed to pass the tunnel.

Peer generator

When creating login credentials for multiple clients, the peer generator offers a convenient way to create and configure them.

The generator lets you select the WireGuard instance the new peer should belong to and automatically derives suitable client addresses from the networks configured on that instance. It also stores generator defaults on the selected instance so they can be reused for future peers.

Each newly created peer receives a key pair. By default, only the public key is stored on the firewall.

Attention

Attention

Storing the peer private key on the firewall is optional and disabled by default.

The private key normally only belongs on the client device. Enabling private key storage makes it possible to reconstruct the generated client configuration later, but also means that the client’s private key is retained in the firewall configuration.

Only enable this option when the convenience of being able to retrieve the client configuration again outweighs the additional exposure of storing the private key on the firewall.

After providing the relevant client information, the generated configuration can be copied from the Config field or imported using the displayed QR code.

When private key storage was enabled while creating the peer, the peer can later be opened again from the Peers page using the QR code action. This opens the peer generator in reference mode and reconstructs the client configuration and QR code from the stored peer and instance information.

Peers created without private key cannot have their original client configuration or QR code reconstructed later, because the required private key was never retained by the firewall.

The peer generator uses values stored on the selected instance as defaults for newly generated configurations. These include the client endpoint, DNS servers and allowed IPs and can be changed in the instance configuration.

Peer-specific values, such as the assigned tunnel address, preshared key and keepalive interval, remain part of the peer configuration.

High availability (using CARP)

When using wireguard on active/passive high availability clusters, only one instance at a time is allowed to communicate to the other party. In Ixolate this can be reached by selecting a vhid to track as instance dependency {Depend on (CARP)}.

If an instance depends on a CARP vhid, it will query the current status and determine if the interface should be usable (when MASTER), the interface status (up/down) will be toggled accordingly.

Diagnostics and debugging

In VPN › WireGuard › Status you can find the configured instances and peers including their last known handshake and the amount of data being exchanged. For Instances you are also able to see if the device underneath (wgX) is up or down, depending on the carp status described in the previous chapter.

Examples

This paragraph offers examples for some commonly used implementation scenarios.

Plugin VPN options

Via plugins additional VPN technologies are offered, including:

• OpenConnect - SSL VPN client, initially build to connect to commercial vendor appliances like Cisco ASA or Juniper.
• Stunnel - Provides an easy to setup universal TLS/SSL tunneling service, often used to secure unencrypted protocols.
• Tinc - Automatic Full Mesh Routing
• WireGuard - Simple and fast VPN protocol working with public and private keys.
• Zerotier - seamlessly connect everything, requires account from zerotier.com, free for up to 100 devices.

how-tos/openconnect how-tos/stunnel how-tos/zerotier