FreeBSD 12 Jail Host - Part 3 - Networking (2021)

Estimated reading time: Seal yourself in a dark room for a day or two.

This is where things start to get a little bit hairy. We're going to get vnet set up, ipfw set up and nat implemented, etc.

The Journey

All the parts in this series.

The Network

For the purposes of this document we're going to need to reference several devices and subnets. I'll lay those out here, and you can adjust for whatever your own setup looks like or whatever you choose to use.

  • External Interface: ena0
  • External Network: 10.0.10.0/24
  • Primary External IP (DHCP): 10.0.10.100/24
  • Secondary External IP (Static): 10.0.10.200/24
  • Bridge: bridge0
  • Bridge Network: 10.0.20.0/24
  • Host's Bridge IP (Static): 10.0.20.1/24

The Bridge

First we're going to create the virtual network device to act as our bridge. In /etc/rc.conf add:

cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.20.1/24"

We need to make some other configuration changes, so let's do that first before we reboot to bring up the interface.

The Router

Let's make a few more changes to enable ipfw, packet forwarding, and configure some other things around that. In /etc/rc.conf add:

# Enable ipfw firewall
firewall_enable="YES"
# Enable kernel NAT
firewall_nat_enable="YES"
# Run a script to initialize firewall
firewall_script="/etc/ipfw.rules"
# Enable packet forwarding
gateway_enable="YES"

We'll also make some changes to how the firewall handles NAT'd packets. Add to /etc/sysctl.conf:

# After processing packets through NAT, reinject them into the firewall
# rather than stopping and forwarding them.
net.inet.ip.fw.one_pass=0
# Disable TCP segmentation offloading as in-kernel NAT does not support it.
net.inet.tcp.tso=0

Let's also take this opportunity to configure the secondary IP address on our interface. In /etc/dhclient.conf add:

interface "ena0" {
}

alias {
	interface "ena0";
	fixed-address 10.0.10.200;
	option subnet-mask 255.255.255.0;
}

Finally, before we restart and see if we blew everything up, let's put a basic firewall script in place to just make the firewall wide open for now so we can connect when the machine comes back up. Put the following in /etc/ipfw.rules:

#!/bin/sh
IPFW="/sbin/ipfw"
$IPFW -q -f flush
$IPFW add allow all from any to any

Now reboot!

When the machine comes back up, you should see your secondary static IP assigned to your external interface and a bridge0 interface with the appropriate IP assigned. Running ipfw list should show your allow rule. If so, then everything's in place to move forward.

The NAT

Now that we've got everything enabled and (ostensibly) working. It's time to actually configure it to actually do what we want. For the purposes of this section, I'll be using a script like the one below for testing firewall configuration because I'm working on this server remotely:

#!/bin/sh
TIMEOUT="30"
SCRIPT_NEW="${1:-/etc/ipfw.rules.new}"
SCRIPT_OLD="${2:-/etc/ipfw.rules}"

echo "Running new ruleset: $SCRIPT_NEW"
$SCRIPT_NEW

echo "Waiting $TIMEOUT seconds"
echo "Press Ctrl+C to cancel rollback."

i=0
while /usr/bin/true; do
	echo "."
	sleep 1
	i=$((i+1))
	if [ "$i" == "$TIMEOUT" ]; then
		echo "Timeout exceeded. Rolling back."
		break
	fi
done

echo "Running old ruleset: $SCRIPT_OLD
$SCRIPT_OLD

echo "Waiting $TIMEOUT*2 seconds"
echo "Press Ctrl+C to cancel failsafe."

i=0
while /usr/bin/true; do
	echo "."
	sleep 2
	i=$((i+1))
	if [ "$i" == "$TIMEOUT" ]; then
		echo "Timeout exceeded. Implementing failsafe."
		break
	fi
done

echo "Clearing rules"
/sbin/ipfw -q -f flush
echo "Allowing all traffic"
/sbin/ipfw add allow all from any to any

echo "Done. Hopefully you find your way back to me."

What this does is runs a new ruleset at /etc/ipfw.rules.new then waits TIMEOUT seconds (default 30). If you haven't killed the script by pressing Ctrl+C before then, then the script assumes you're unable to connect to the instance and will re-run the old/current ruleset at /etc/ipfw.rules then waits double TIMEOUT seconds (default 60). If you still haven't been able to connect in and kill the script, it will then flush the entire ruleset and replace it with a rule allowing all traffic. This allows you to test out new rulesets with relatively little risk—worst case scenario should be your instance becoming unavailable for 90 seconds. Even if this fails completely (e.g., if you forgot to run with nohup), as long as /etc/ipfw.rules contains a working ruleset, then you should be able to recover with a reboot.

Note that in order for this to work you'll need to do two things:

  • Ensure both your /etc/ipfw.rules and /etc/ipfw.rules.new have the execute bit set. E.g., chmod 0755 /etc/ipfw.rules*
  • Run this test script such that it won't be killed if your SSH session is disconnected. You can run it inside of something like screen, or run it with something like nohup sh ipfw_test.sh & to detach it from your shell if it's killed. I prefer screen as it makes it simpler to reattach and kill the script before rollback if your rules worked but just caused your firewall to lose state and drop the connection.

ipfw.rules

A copy of the ipfw script I'm using is below. I won't add much explanation as most of the purpose and context is in the comments.

#!/bin/sh
IPFW="/sbin/ipfw"

################################################################################
# CONFIG
################################################################################

##########
# Loopback
NET_LOOPBACK_INTF="lo0"
NET_LOOPBACK_CIDR="127.0.0.0/8"

##########
# External
NET_EXTERNAL_INTF="ena0"

# Our default/primary IP for services
# TODO: Pull this ip from ifconfig or something as it's not static.
NET_EXTERNAL_DEFAULT_IP="10.0.10.100"
# Port forwards on default IP
NET_EXTERNAL_DEFAULT_NAT_CONFIG="
        redirect_port tcp 10.0.20.5:80 80
        redirect_port tcp 10.0.20.5:443 443
"

# Our secondary IP that we only use for one service
NET_EXTERNAL_SECONDARY_IP="10.0.10.200"
# What range on the internal bridge we want to
# forward out via this IP
NET_EXTERNAL_SECONDARY_CIDR="10.0.20.40/32"
# Port forwards on the secondary IP
NET_EXTERNAL_SECONDARY_NAT_CONFIG="
        redirect_port tcp 10.0.20.40:22 22
        redirect_port tcp 10.0.20.40:80 80
        redirect_port tcp 10.0.20.40:443 443
"

##########
# Bridge
NET_BRIDGE_INTF="bridge0"
NET_BRIDGE_CIDR="10.0.20.0/24"

################################################################################
# RULES
################################################################################

# Flush all existing rules
$IPFW -q -f flush

# Disable one_pass -- not sure if this is needed as it's disabled in sysctl.
# After a packet runs through NAT, we reinject it back into the firewall at
# the next rule rather than just allowing it to let us do further filtering
$IPFW disable one_pass

# Allow all local traffic passing loopback->loopback or bridge->bridge so
# containers can communicate among themselves and the host can talk to itself.
$IPFW add allow all from any to any via $NET_LOOPBACK_INTF
$IPFW add allow all from $NET_BRIDGE_CIDR to $NET_BRIDGE_CIDR via $NET_BRIDGE_INTF

##########
# NAT

# NAT 1: For the default external IP
$IPFW nat 1 config if $NET_EXTERNAL_INTF ip $NET_EXTERNAL_DEFAULT_IP $NET_EXTERNAL_DEFAULT_NAT_CONFIG
# NAT 2: For the secondary external IP
$IPFW nat 2 config if $NET_EXTERNAL_INTF ip $NET_EXTERNAL_SECONDARY_IP $NET_EXTERNAL_SECONDARY_NAT_CONFIG

##########
# External - Incoming

# Apply NAT to incoming packets on external interface.
# We decide which NAT instance to use based on which IP the traffic has
# been sent to.
$IPFW add nat 1 ip4 from any to $NET_EXTERNAL_DEFAULT_IP/32 in via $NET_EXTERNAL_INTF
$IPFW add nat 2 ip4 from any to $NET_EXTERNAL_SECONDARY_IP/32 in via $NET_EXTERNAL_INTF

# After NAT is performed, any packet destined for anything on the bridge
# should now have the destination address rewritten. We can do some
# filtering on traffic destined to the host itself here.

# AWS security groups are per-interface, not per-ip. Port 22 is open for
# the service on the secondary IP, but we have the host system's SSH moved
# to a different port so we can limit the source IPs. That means that port 22
# is still passed through the AWS security group to us on the default ip. This
# rule just drops all traffic to the host's IP on port 22 so it will not reject
# the connection but silently drop it.
# NOTE: Remove this if you kept your SSH daemon on port 22.
$IPFW add drop tcp from any to $NET_EXTERNAL_DEFAULT_IP 22 in via $NET_EXTERNAL_INTF

##########
# External - Outgoing

# Perform NAT on outgoing traffic forwarded from the bridge.
# We select which NAT instance to use based on the jail's IP address.

# Send traffic from our service we want on the secondary IP through the secondary
# IP's NAT.
$IPFW add nat 2 ip4 from $NET_EXTERNAL_SECONDARY_CIDR to any out via $NET_EXTERNAL_INTF
# Send all other traffic through the primary IP.
$IPFW add nat 1 ip4 from $NET_BRIDGE_CIDR to any out via $NET_EXTERNAL_INTF

##########
# Catch-all

# Allow everything
# Normally I'd go for a whitelist-based firewall, but since we've got the
# benefit of the AWS security group in front of us already doing that I'm
# not _too_ worried about it.
$IPFW add allow all from any to any

Run this firewall script (I highly recommend using the test script!). Your connection to the server may drop temporarily, but you have 30 seconds to connect back in, reattach to your screen, and kill the script before it rolls back to your old (wide open) rules. If everything's working, you can copy the /etc/ipfw.rules.new file over top of /etc/ipfw.rules to persist the changes through a restart.

The Conclusion

At this point everything should be in place for our next steps—actually starting up some jails. We'll do that in Part 4.