nguvu

UPS management with NUT

Last revised 14 September 2018.

Contents

Introduction
UPS Hardware
Recommended hardware
UPS Shutdown sequence
Shutdown plan
Configure APC UPS
Configure pfSense
Install Network UPS Tools (NUT)
Firewall rules config
Enable NUT Daemon access from hosts
Port Forward method
Listen Directive method
Verify communications
Testing
Additional steps
SNMP monitoring
Links & References

Introduction

Hosting ZFS file servers and virtualisation systems such as ESXi or Proxmox requires some precautions against less than perfect electrical supplies where brown or black outs can cause cause unexpected down time or worse, data corruption and loss. Adding a Uninterruptible Power Supply (UPS) to your network guards against these kind of events. This aim of this guide is to provide some recommendations for hardware, connectivity and configuration of pfSense’s NUT package to provide a central server capable of managing a number of connected devices and servers.

UPS Hardware

There are many manufacturers of UPS systems, the following details the criteria I used to determine which models would suit my needs.

Voltage
Ensure input and output voltages are as per your requirements, USA being primarily 110v and Europe being 240v.

Power rating
UPSs are typically rated in volt-amperes (VA) and watts. Watts measures real power and is the key rating. For example a UPS rated at 1000 VA / 900 watts provides 33% more power than one rated at 1000 VA / 600 watts.

Extensible power
My preference is for UPS’ that offer the ability to daisy-chain additional battery unit that enable you to easily extend your UPS system if your needs change.

Input plug
Verify the UPS’ power plug is suitable for you home, some larger models come with connections that require higher capacity circuits and it may be expensive to have your home or office wired for them. UPS’ with 1500 VA and below typically plug into a standard wall socket.

Output receptacles
Verify theres enough output receptacles of the right type for your needs. Daisy-chaining extension cords isn’t advisable.

Communications / Network card
Given the importance of a battery backup for the reasons outlined earlier, we need to provide a communication mechanism capable of 24x7 reliable operation. My preference is for a network connection which I’ve found more robust than USB connections. Verify your intended UPS comes with a network port or one can be installed.

Environmental sensors
If you desire the ability to monitor the local temperature and humidity, make sure your communication card has inputs for the environmental sensors.

Easily replaceable batteries
Most manufacturers will sell you a complete battery sled but they are usually expensive. A number of solutions take regular 12v, 9AH batteries which can be obtained cheaply and replaced fairly easily.

NUT compatibility
Check compatibility on the NUT hardware compatibility list to verify they are supported by NUT.

Recommended hardware

I’ve had good success with the following APC and Eaton units over several years now.

Eaton

APC

UPS Shutdown sequence

Its worth understanding how a NUT shutdown is managed.

  1. Power fails and UPS goes on battery (ups.status = OB)
  2. some time passes and UPS eventually reaches a low battery condition (ups.status = OB LB)
  3. upsmon master sets forced shutdown (ups.status = FSD OB LB) flag to inform all slave systems that it will soon power down the load (if there are no slaves, skip to step 6)
  4. upsmon slave systems notice the FSD flag and
    • generate NOTIFY_SHUTDOWN event
    • wait FINAL.DELAY seconds
    • call their SHUTDOWNCMD
    • disconnect from upsd
  5. upsmon master waits up to HOSTSYNC seconds for all salves to disconnect from upsd and then regardless of any remaining connections, upsmon stops waiting and proceeds with shutdown process.
  6. upsmon master
    • generates a NOTIFY_SHUTDOWN event
    • waits FINAL.DELAY seconds
    • if configured to, creates POWERDOWNFLAG file
    • calls the SHUTDOWNCMD
  7. init takes over, kills processes, syncs/unmounts file systems, and prepares for shutdown
  8. init runs shutdown script. This checks for the POWERDOWNFLAG file and if it finds it, and tells the UPS driver to power off the load.
  9. The system loses power.
  10. Some time passes, power returns, UPS switches back on.
  11. All systems reboot.

Shutdown plan

Although there is an option in my UPS’ to segment my hardware into three groups which would enable devices to be powered off in stages extending operating time for critical infrastructure such modems/routers during power outages, I haven’t needed to utilise this so far as I have enough battery capacity to provide sufficient run time to cover the typical power outages I experience and cover shut down needs. I currently set my UPS to initiate shutdown at 20% remaining battery time.

Configure APC UPS

I’ll cover the initial configuration of the APC Smart-UPS X 1500 UPS fitted with network communications card and dual environmental sensors.

UPS Configuration

Navigate to Configuration > UPS

Network Settings

Navigate to Configuration > Network > TCP/IP > IPv4 Settings

Add a reserved IP address in pfSense’s DHCP server on the approprite subnet to ensure the UPS is allocated the same address each time it boots, in my case this is 192.168.10.50 in my VL10_MGMT subnet.

## System Time

Navigate to Configuration > General > Date/Time > Mode

Apply

Navigate to Configuration > General > Date/Time > Daylight Savings

Apply

Power Settings

Ideally you want to use the highest sensitivity you can without causing too many transitions to/from battery power.

Navigate Configuration > Power Settings

For 120VAC output voltage

Apply

Shutdown Configuration

Navigate Configuration > Shutdown

On Battery shutdown behaviour

Apply

Self-Test Schedule Configuration

Navigate Configuration > Self-test schedule

Regular testing ensures your batteries are capable of covering any outages you may suffer and also calibrate runtime within the UPS too.

Apply

Universal IO

If you have connected environmental sensors to your network card, you’ll be able to configure them here.

Navigate Configuration > Universal IO > Temp & Humidity

Verify names are meaningful for your configuration. I use Port 1 to report my racks input temperature, and Port 2 as the exit temperature. .

Security

Navigate to Configuration > Security

Session Management

Allow concurrent Logins = [x]

Ping Response

IPv4 Ping Response = [x]

Local Users

Navigate to Configuration > Security > Local Users > Default Settings

Navigate to Configuration > Security > Local Users > Management

Configure pfSense

Install Network UPS Tools (NUT)

Navigate to System > Packages and select Available Packages, scroll down to the NUT package and click Install.

>>> Installing pfSense-pkg-nut... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 3 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	pfSense-pkg-nut: 2.7.4_7 [pfSense]
	nut: 2.7.4_8 [pfSense]
	neon: 0.30.2 [pfSense]

Number of packages to be installed: 3

The process will require 6 MiB more space.
[1/3] Installing neon-0.30.2...
[1/3] Extracting neon-0.30.2: .......... done
[2/3] Installing nut-2.7.4_8...
[2/3] Extracting nut-2.7.4_8: .......... done
[3/3] Installing pfSense-pkg-nut-2.7.4_7...
[3/3] Extracting pfSense-pkg-nut-2.7.4_7: .......... done
Saving updated package information...
done.
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Custom commands...
Executing custom_php_install_command()...done.
Executing custom_php_resync_config_command()...done.
Menu items... done.
Services... done.
Writing configuration... done.
>>> Cleaning up cache... done.
Success

Firewall rules config

Assuming you followed my pfSense baseline configuration add the following ports to our ALLOWED_LAN_OUT alias.

Enable NUT Daemon access from hosts

There are two methods to enable remote host access to the UPS daemon available, port forward and the LISTEN directive. My preference is for the port forward method rather than the listen directive method as it’s simpler to limit access to the daemon by adding a source address to the NAT rule. I’ll cover both methods here but remember to only implement one.

My UPS network card is assigned 192.168.10.50.

We define a user “slave” with password “mypassword”.

Different UPS’ declare low.battery at different level, some as low as 5% battery remaining, this is too low for my liking so I override how the master determines when to declare a low battery condition using the ignorelb command and configure low battery to be declared when the battery level falls below 20%. You may need to adjust these values to gives enough time to cleanly shutdown your system. In order for this to work, your UPS should be able to (reliably) report charge and/or runtime remaining on battery. Use with caution, test this thoroughly and don’t cut things too close, you don’t want the UPS running out of power before the shutdown is complete. For context 20% battery level on my system equates to around 40 minutes of runtime, plenty of time to shut everything down cleanly.

Port Forward method

Navigate to Services > UPS > UPS Settings

General Settings

Driver Settings

ignorelb
override.battery.charge.warning = 25
override.battery.charge.low = 20

Advanced Settings

Additional configuration options are set as follows:

[remoteuser]
password = mypassword
upsmon slave

Click Save

Skip down past the listen directive method to the Verify communications section.

Listen Directive method

Navigate to Services > UPS

General Settings

Driver Settings

ignorelb
override.battery.charge.warning = 25
override.battery.charge.low = 20

Advanced Settings

Additional configuration options are set as follows:

[remoteuser]
password = mypassword
upsmon slave

Click Save

Verify communication

Navigate to Services > UPS > UPS Status where you should see the UPS Stats.

UPS status
UPS status

Theres also a NUT dashboard widget to display key stats on the pfSense status page which is useful.

You can also test from the pfSense command line using /usr/local/bin/upsc <UPS NAME>
Specifically look for ups.status: OL indicating On-line. OB indicates ‘On Battery’ and LB ‘Low Battery’.

/root: /usr/local/bin/upsc UPS

ambient.1.humidity.alarm.high: 60.00
ambient.1.humidity.alarm.low: 30.00
ambient.1.temperature.alarm.high: 40.00
ambient.1.temperature.alarm.low: 10.00
ambient.humidity: 46.00
ambient.temperature: 23.0
battery.charge: 100.00
battery.charge.low: 30
battery.date: 11/15/2017
battery.packs: 1.00
battery.runtime: 8618.00
battery.runtime.low: 1200
battery.voltage: 54.10
device.mfr: APC
device.model: Smart-UPS X 1500
device.serial: AS1234567890
device.type: ups
driver.flag.ignorelb: enabled
driver.name: snmp-ups
driver.parameter.pollinterval: 2
driver.parameter.port: 192.168.10.50
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: apcc MIB 1.2
driver.version.internal: 0.97
input.frequency: 59.90
input.sensitivity: high
input.transfer.high: 127
input.transfer.low: 106
input.transfer.reason: selfTest
input.voltage: 120.60
input.voltage.maximum: 121.70
input.voltage.minimum: 120.30
output.current: 2.80
output.frequency: 59.90
output.voltage: 120.50
output.voltage.nominal: 120
ups.delay.shutdown: 20
ups.delay.start: 10
ups.firmware: UPS 09.1 (ID20)
ups.id: UPS
ups.load: 28.10
ups.mfr: APC
ups.mfr.date: 06/30/2012
ups.model: Smart-UPS X 1500
ups.serial: AS1234567890
ups.status: OL
ups.temperature: 26.50
ups.test.date: 09/12/2018
ups.test.result: Ok

Testing

Its vital to comprehensively test the shutdown sequence on your systems before leaving it unattended.

You can test a forced shutdown sequence (FSD) using the following command from the pfSense command line.

/usr/local/sbin/upsmon -c fsd

Its also worth a final check by simulating a real power out event by pulling the plug and letting the UPS run down. This final check also verifies the power-up sequence for your UPS and connected devices.

A successful shut down sequence is defined as one where the OS halts before the battery runs out, and the system restarts to proper operation when power returns.

Power Off vs Halt

When shutdown time arrives, the NUT package uses the following shutdown command:

SHUTDOWNCMD "/sbin/shutdown -p +0"

This command will power off the pfSense system. This is generally appropriate for systems such as firewalls that are configured to always turn on when power is applied. If there is a power setting in the BIOS this is generally referred to as “always on”.

If you have a system that does not support the always on mode, and instead always returns to the prior (last) state when power is applied, then you probably want to override the shutdown command so that the system is halted but not powered off.

You can do this by placing the following in upsmon.conf section of the advanced settings:

SHUTDOWNCMD "/sbin/shutdown -h +0"

If you have an option, the default “always on” approach is preferable.

Additional steps

SNMP monitoring

I make use of some LibreNMS to monitor my network devices, this provides great insight on many aspects of my infrastructures performance, for example, here are some graphs generated over the last 12 months or so.

UPS temperatures
UPS temperatures

and here’s some useful data that shows how my electricity suppliers performance can vary

UPS voltage
UPS voltage

Links & References

[NUT homepage]((http://networkupstools.org)
NUT mail list
NUT HCL
APC AP9631 network card manual
NUT ups.conf man pages