nguvu

pfSense multi-WAN failover

Last revised 7 March 2021.

Contents

Introduction

As more homes and business rely on the internet than ever, a reliable internet connection is pertinent to avoiding downtime in the event that internet connection is disrupted. Landlines are unfortunately vulnerable to disruption, and no matter how reliable your internet service provider is, having a backup, or failover, connection is useful to avoid disruptive downtime.

Failover is the ability to seamlessly switch to a reliable backup system when a primary system fails minimizing disruption to systems and users. Backup communication systems can be provided with additional broadband connections, satellite links, or cellular based systems. This guide will focus on configuring pfSense and supporting hardware to provide failover protection using a cellular 4G LTE connection.

Multi WAN failover options
Multi WAN failover options

Failover options

In considering failover options, I prioritised reliability and performance over cost or ease of installation. This does not mean this has to be expensive or difficult to install, but if going to the cost and effort of providing redundant communication channels, having them be available when needed, and fit for intended purpose is a requirement.

4G LTE based failover

4G LTE based cellular connections are available cheaply and can offer reliable and reasonable performance given suitable proximity and line of sight to cellular towers. There are a number of devices that can enable to use of 4G.

Onboard PCI / m-sata based modems

There are motherboards that support M.2 B-Key modems directly, for example the Supermicro’s X11SDV pictured below.

modem connection
X11SDV MODEM and SIM connectors

Although this may seem ideal, there are a couple of drawbacks to consider.

USB modems

USB modems suffer with similar drawbacks to onboard devices, i.e

4G router / Ethernet adapters

There are dedicated enterprise class 4G LTE routers available that can be configured to convert a 4G LTE signal to a standard ethernet RJ45 based connection, ideal for feeding into pfSense. I currently recommend the Cradlepoint CBA-850 for the following reasons:

Retail pricing on these routers new was in the region of $600 reflecting the enterprise nature, however, they are now available as cheaply as $50 refurbished from retailers or other popular online used sales portals.

Cradlepoint CBA-850 Configuration

The CBA850 has two primary modes of operation. It can perform primary routing functions, or as this guide will configure, function simply as a modem using IP Passthrough (bridge Mode) to the primary pfSense router, essentially acting as a ‘cellular-to-ethernet’ adapter.
In passthrough mode a public IP address will be passed through to the connected device so its important to connect to a secure network device such as a router or firewall and not direct to a laptop or other unprotected device. This mode also removes the need to use an unnecessary Network Address Translation (NAT) layer and also removes the need to configure any DMZ or port forwarding settings on the Cradlepoint.

First time setup

It may simplify problem resolution to configure the CBA850 in a physically easy to access location prior to any final external installation.
Connect your PC to the LAN 1 Ethernet port on the CBA850. LAN 2 POE is by default set to IP Passthrough and requires a valid internet connection before you can access the router by a different IP address than LAN 1.

Connect your PC directly to your CBA-850 and navigate to 192.168.10.1. You can login with initially with the default username of admin. The default password is last 8 digits of the units MAC address, also printed on underside of the modem.

Login
Login

Bypass NetCloud

From April 2019 Cradlepoint moved towards a cloud-based subscription service called NetCloud. This is not a requirement for our use case.

Netcloud username: blank
Netcloud password: blank
Dont show this window again
Click Start Nextcloud OS

Bypass Netcloud
Bypass Netcloud

Wizard setup

Step 1 of 3
Administrator password: somethingSuperSecure
Timezone: as per your locale
Click Next

Set Admin Password
Set Admin Password

Step 2 of 3
WAN verify enabled: off
Click Next

WAN verify
WAN verify

Step 3 of 3
Verify Settings and Click Finish

Wizard complete
Wizard complete

Update firmware

Cradlepoint recently moved (back) towards a cloud-based subscription service which was implemented in NCOS software releases post v7.0.50. Use NCOS v7.0.50 and avoiding updating further to retain functionality without needing to subscribe to the paid NCOS service.

Netcloud OS version
Netcloud OS version

Navigate to System > NetCloudOS > Manual NCOS Upload and upload firmware that can be obtained from the following links.

Modem configuration

If there is a valid SIM card and appropriate firmware installed your modem should connect.
If there is a need to configure the modem APN or bands, it can be performed through the device’s console at System > System Control > Device Options > Device Console.
Verify the modem is connected as indicated by a green icon in the header bar and in the connection manager dialogue.

Common carrier APN’s are

MODEM connection
MODEM connection

System

Logging

Once configured and functional, consider reducing the amount of logging by navigating to System > Administration > System Logging

Click Save

Networking

Administration

Set system identifier by navigating to System > Administration > Local Management

Click Save

Interfaces

Primary LAN1 is configured with NAT and administrator access. This enables a connection through LAN1 port for debugging purposes should LAN2 fail to function for any reason.
The IPPT Interface is configured to operate in IP Passthrough mode with administration access available at 192.168.13.1.

Navigate to Networking > Local Networks > Local IP Networks

IPPT interface:
Select Edit

Click Save

After saving, the Local IP Networks should look like

Configured Interfaces
Configured Interfaces

Cradlepoint CBA-850 Installation

Although it’s possible to use the CBA-850 inside a building with the included antennas, an optimal signal is available when using an external antenna and minimally attenuated by the use of short coaxial cables to an adjacently mounted CBA-850.

To protect the CBA-850 from the elements install it into a robust weather proof box such as one of those from WiFix. The image below is a medium enclosure (27.2 x 27.6 x 9.6cm / 10.7 x 10.8 x 3.8inch) for reference.

WiFix medium enclosure
WiFix medium enclosure

Although a number of official Cradlepoint CBA-850 modems are available, cost effective 3rd party modems are usable with an appropriate USB sled. The images below shows both a USB>M2 and USB>PCIe based modem sled that I used whilst testing various modems.

USB M2 adapter
USB M2 adapter
USB PCie adapter
USB PCIe adapter

Antennas

Antennas are a complicated topic and are outside the scope of this article. Optimal selection will be heavily influenced by your geographical location and relative proximity to the broadcast towers.

Two invaluable web sites that can help understand how to maximize signal reception include:

Cellmapper

Cellmapper offers insight as to what cellular towers are within your vacinity, and what carriers and bands are available on each.

Installation tools
Installation tools

airLink offers insight into the land topology between you and the cellular tower you are connected to.

Installation tools
Installation tools

pfSense configuration

This section will build on the pfSense baseline guide setup. This section aims to cover the major services that require configuring to enable failover functionality. However, there may be unique local settings that the reader will need to consider, for example, if they have followed other guides that implement redundant VPN connections.

Create failover WAN Interface

Navigate to Interfaces > Assignments

Select an available network port that the Cradlepoint modem will be connected to, click Add.
Click on the newly created OPTx label to configure the port as follows:

General Configuration

Reserved Networks

Create failover VPN

A second VPN connection will be required that originates its connection on the WAN3 interface previously created.
The originating interface can be edited within the VPN client configuration.

DNS

Add the ability to resolve system and DNS Forwarder based queries over the failover WAN3 connection.

Navigate to System > General Setup

Add two additional servers under the existing WAN_DHCP based primary and secondary servers. This example uses the Quad9 servers but can be substituted.

Set the gateway of these two servers to be the failover WAN3 interface. When complete your DNS server settings should look like this.

DNS Server settings
DNS Server settings

Routing

Gateway

Add a gateway for failover WAN3 traffic. This example uses a Quad9 server for monitoring latency.

Navigate to System > Routing > Gateways & click Add.

Default gateways

Navigate to System > Routing > Gateways

When complete the gateway configuration should look similar to this

Default gateways
Default gateways

Gateway Group - WAN

Create WAN_Group gateway group.

Navigate to System > Routing > Gateway Groups & click Add.

Click Save

Gateway Group
Gateway Group

Gateway Group - VPN

A gateway group will be required for the VPN failover also. When the primary WAN_DHCP connection drops, VPN1_WAN will also be dropped. This VPN failover group will failover the VPN1_WAN tunnel to the secondary VPN2_WAN gateway available via the WAN3 connection.

Navigate to System > Routing > Gateway Groups & click Add.

Click Save

Verify this is correct by navigating to Status > Gateways > Gateway Groups

Gateway Groups status
Gateway Groups status

Static routing for Cradlepoint CBA850 access

Once the modem connects pfSense will be allocated an external IP address. In order to be able to access the Cradlepoint CBA850 configuration page a static route is required to direct traffic out the appropriate port.

Navigate to System > Routing > Static Routes & click Add.

Save

NAT

Additional NAT rules will be required to enable traffic to egress failover gateways (WAN3 & VPN2_WAN) from the local subnets. Although this guide only provides an example for the LAN interface, you will need to consider all your subnets and external traffic flows for a fully robust system.

LAN

Navigate to Firewall > NAT > Outbound

Ensure ‘Manual outbound NAT rule generation` is selected.

Click ‘↴+’

Advanced Outbound NAT Entry

Translation

Miscellaneous

Click Save & Apply

Firewall rules

The firewall rules are configured as per the baseline guide for non-local traffic to egress to the internet via the default gateway. The rule responsible for this needs updating to egress traffic via the WAN_Group so traffic will egress the Tier 1 gateway, or if unavailable, egress out the failover tier2 connection. When the tier1 gateway is available again, traffic will revert to egress via the primary gateway once again.

WAN3

Navigate to Firewall > Rules > WAN3
Create default block for IPv4 (with logging) & IPv6 (sans logging)

WAN3 firewall rules
WAN3 firewall rules

VL10_MGMT egress

Navigate to Firewall > Rules > VL10_MGMT

Click on the pencil icon by the WAN egress rule and change the gateway to be the WAN_Group. The option is located under the advanced options section which is hidden by default.

Before rules

Original V10_MGMT default gateway
Original VL10_MGMT default gateway

and after editing

Revised VL10_MGMT WAN_Group gateway
Revised VL10_MGMT WAN_Group gateway

Adjust the non-local egress on your other subnet interfaces.

VL20_VPN egress

VL20_VPN supports traffic egress via both the regular and VPN gateways. Both rules require updating to support failover gateways.

Navigate to Firewall > Rules > VL20_VPN

Click on the pencil icon by the WAN egress rule and change the gateway to be the WAN_Group.
Click on the pencil icon by the VPN egress rule and change the gateway to be the WPN_Group.

Before rules

Original VL20_VPN default gateway
Original VL20_VPN default gateway

and after editing

Revised VL20_VPN WAN_Group gateway
Revised VL20_VPN WAN_Group gateway

Testing

Navigate to Status > Gateways > Gateway Groups and verify the primary gateway enters a degraded state when the primary connection is interrupted. Verify the primary gateway returns to service when they primary connection is restored. There is a smoothing function applied to the gateway status to prevent flapping that will prevent the gateway being restored immediately.

Failover tuning

If the gateway isn’t appropriately responsive to service levels, tuning gateway thresholds may be necessary.

Navigate to System > Routing and click the pencil icon next to a gateway.

Under the Advanced Gateway Settings adjust the thresholds for packet loss, latency, down time, and probing intervals that control when the gateway is considered up or down.

Member Down: Marks the gateway as down only when it is completely down, past one or both of the higher thresholds configured for the gateway. This catches worst case failures, when the gateway is completely unresponsive, but may miss more subtle issues with the circuit that can make it unusable before the gateway reaches that level.

Packet Loss: Marks the gateway as down when packet loss crosses the lower alert threshold (See Advanced Gateway Settings).

High Latency: Marks the gateway as down when latency crosses the lower alert threshold (See Advanced Gateway Settings).

Packet Loss or High Latency: Marks the gateway as down for either type of alert.

DNS leaks

I recommend running the full suite of DNS tests from the baseline guide to ensure full functionality and that no privacy compromising leaks have been introduced in both primary and failover states of operation.

References

FreeBSD 12.2 supported hardware