FreeBSD 12 Jail Host - Part 3 - Networking (2021)
Estimated reading time: Seal yourself in a dark room for a day or two.
This is where things start to get a little bit hairy. We're going to get vnet set up, ipfw set up and nat implemented, etc.
The Journey
All the parts in this series.
- Part 1—Basic System Setup
- Part 2—Jail Manager
- Part 3—Networking
- Part 4—Jails
- Part 5—Usage
The Network
For the purposes of this document we're going to need to reference several devices and subnets. I'll lay those out here, and you can adjust for whatever your own setup looks like or whatever you choose to use.
- External Interface: ena0
- External Network: 10.0.10.0/24
- Primary External IP (DHCP): 10.0.10.100/24
- Secondary External IP (Static): 10.0.10.200/24
- Bridge: bridge0
- Bridge Network: 10.0.20.0/24
- Host's Bridge IP (Static): 10.0.20.1/24
The Bridge
First we're going to create the virtual network device to act as our bridge. In /etc/rc.conf
add:
cloned_interfaces="bridge0" ifconfig_bridge0="inet 10.0.20.1/24"
We need to make some other configuration changes, so let's do that first before we reboot to bring up the interface.
The Router
Let's make a few more changes to enable ipfw, packet forwarding, and configure some other things
around that. In /etc/rc.conf
add:
# Enable ipfw firewall firewall_enable="YES" # Enable kernel NAT firewall_nat_enable="YES" # Run a script to initialize firewall firewall_script="/etc/ipfw.rules" # Enable packet forwarding gateway_enable="YES"
We'll also make some changes to how the firewall handles NAT'd packets. Add to
/etc/sysctl.conf
:
# After processing packets through NAT, reinject them into the firewall # rather than stopping and forwarding them. net.inet.ip.fw.one_pass=0 # Disable TCP segmentation offloading as in-kernel NAT does not support it. net.inet.tcp.tso=0
Let's also take this opportunity to configure the secondary IP address on our interface. In
/etc/dhclient.conf
add:
interface "ena0" { } alias { interface "ena0"; fixed-address 10.0.10.200; option subnet-mask 255.255.255.0; }
Finally, before we restart and see if we blew everything up, let's put a basic firewall script
in place to just make the firewall wide open for now so we can connect when the machine
comes back up. Put the following in /etc/ipfw.rules
:
#!/bin/sh IPFW="/sbin/ipfw" $IPFW -q -f flush $IPFW add allow all from any to any
Now reboot!
When the machine comes back up, you should see your secondary static IP assigned to your
external interface and a bridge0
interface with the appropriate IP assigned.
Running ipfw list
should show your allow rule. If so, then everything's in place
to move forward.
The NAT
Now that we've got everything enabled and (ostensibly) working. It's time to actually configure it to actually do what we want. For the purposes of this section, I'll be using a script like the one below for testing firewall configuration because I'm working on this server remotely:
#!/bin/sh TIMEOUT="30" SCRIPT_NEW="${1:-/etc/ipfw.rules.new}" SCRIPT_OLD="${2:-/etc/ipfw.rules}" echo "Running new ruleset: $SCRIPT_NEW" $SCRIPT_NEW echo "Waiting $TIMEOUT seconds" echo "Press Ctrl+C to cancel rollback." i=0 while /usr/bin/true; do echo "." sleep 1 i=$((i+1)) if [ "$i" == "$TIMEOUT" ]; then echo "Timeout exceeded. Rolling back." break fi done echo "Running old ruleset: $SCRIPT_OLD $SCRIPT_OLD echo "Waiting $TIMEOUT*2 seconds" echo "Press Ctrl+C to cancel failsafe." i=0 while /usr/bin/true; do echo "." sleep 2 i=$((i+1)) if [ "$i" == "$TIMEOUT" ]; then echo "Timeout exceeded. Implementing failsafe." break fi done echo "Clearing rules" /sbin/ipfw -q -f flush echo "Allowing all traffic" /sbin/ipfw add allow all from any to any echo "Done. Hopefully you find your way back to me."
What this does is runs a new ruleset at /etc/ipfw.rules.new
then waits TIMEOUT
seconds (default 30). If you haven't killed the script by pressing Ctrl+C before then, then
the script assumes you're unable to connect to the instance and will re-run the old/current
ruleset at /etc/ipfw.rules
then waits double TIMEOUT seconds (default 60). If
you still haven't been able to connect in and kill the script, it will then flush the entire
ruleset and replace it with a rule allowing all traffic. This allows you to test out new
rulesets with relatively little risk—worst case scenario should be your instance
becoming unavailable for 90 seconds. Even if this fails completely (e.g., if you forgot to run
with nohup), as long as /etc/ipfw.rules
contains a working ruleset, then you
should be able to recover with a reboot.
Note that in order for this to work you'll need to do two things:
- Ensure both your
/etc/ipfw.rules
and/etc/ipfw.rules.new
have the execute bit set. E.g.,chmod 0755 /etc/ipfw.rules*
- Run this test script such that it won't be killed if your SSH session is disconnected. You
can run it inside of something like
screen
, or run it with something likenohup sh ipfw_test.sh &
to detach it from your shell if it's killed. I prefer screen as it makes it simpler to reattach and kill the script before rollback if your rules worked but just caused your firewall to lose state and drop the connection.
ipfw.rules
A copy of the ipfw script I'm using is below. I won't add much explanation as most of the purpose and context is in the comments.
#!/bin/sh IPFW="/sbin/ipfw" ################################################################################ # CONFIG ################################################################################ ########## # Loopback NET_LOOPBACK_INTF="lo0" NET_LOOPBACK_CIDR="127.0.0.0/8" ########## # External NET_EXTERNAL_INTF="ena0" # Our default/primary IP for services # TODO: Pull this ip from ifconfig or something as it's not static. NET_EXTERNAL_DEFAULT_IP="10.0.10.100" # Port forwards on default IP NET_EXTERNAL_DEFAULT_NAT_CONFIG=" redirect_port tcp 10.0.20.5:80 80 redirect_port tcp 10.0.20.5:443 443 " # Our secondary IP that we only use for one service NET_EXTERNAL_SECONDARY_IP="10.0.10.200" # What range on the internal bridge we want to # forward out via this IP NET_EXTERNAL_SECONDARY_CIDR="10.0.20.40/32" # Port forwards on the secondary IP NET_EXTERNAL_SECONDARY_NAT_CONFIG=" redirect_port tcp 10.0.20.40:22 22 redirect_port tcp 10.0.20.40:80 80 redirect_port tcp 10.0.20.40:443 443 " ########## # Bridge NET_BRIDGE_INTF="bridge0" NET_BRIDGE_CIDR="10.0.20.0/24" ################################################################################ # RULES ################################################################################ # Flush all existing rules $IPFW -q -f flush # Disable one_pass -- not sure if this is needed as it's disabled in sysctl. # After a packet runs through NAT, we reinject it back into the firewall at # the next rule rather than just allowing it to let us do further filtering $IPFW disable one_pass # Allow all local traffic passing loopback->loopback or bridge->bridge so # containers can communicate among themselves and the host can talk to itself. $IPFW add allow all from any to any via $NET_LOOPBACK_INTF $IPFW add allow all from $NET_BRIDGE_CIDR to $NET_BRIDGE_CIDR via $NET_BRIDGE_INTF ########## # NAT # NAT 1: For the default external IP $IPFW nat 1 config if $NET_EXTERNAL_INTF ip $NET_EXTERNAL_DEFAULT_IP $NET_EXTERNAL_DEFAULT_NAT_CONFIG # NAT 2: For the secondary external IP $IPFW nat 2 config if $NET_EXTERNAL_INTF ip $NET_EXTERNAL_SECONDARY_IP $NET_EXTERNAL_SECONDARY_NAT_CONFIG ########## # External - Incoming # Apply NAT to incoming packets on external interface. # We decide which NAT instance to use based on which IP the traffic has # been sent to. $IPFW add nat 1 ip4 from any to $NET_EXTERNAL_DEFAULT_IP/32 in via $NET_EXTERNAL_INTF $IPFW add nat 2 ip4 from any to $NET_EXTERNAL_SECONDARY_IP/32 in via $NET_EXTERNAL_INTF # After NAT is performed, any packet destined for anything on the bridge # should now have the destination address rewritten. We can do some # filtering on traffic destined to the host itself here. # AWS security groups are per-interface, not per-ip. Port 22 is open for # the service on the secondary IP, but we have the host system's SSH moved # to a different port so we can limit the source IPs. That means that port 22 # is still passed through the AWS security group to us on the default ip. This # rule just drops all traffic to the host's IP on port 22 so it will not reject # the connection but silently drop it. # NOTE: Remove this if you kept your SSH daemon on port 22. $IPFW add drop tcp from any to $NET_EXTERNAL_DEFAULT_IP 22 in via $NET_EXTERNAL_INTF ########## # External - Outgoing # Perform NAT on outgoing traffic forwarded from the bridge. # We select which NAT instance to use based on the jail's IP address. # Send traffic from our service we want on the secondary IP through the secondary # IP's NAT. $IPFW add nat 2 ip4 from $NET_EXTERNAL_SECONDARY_CIDR to any out via $NET_EXTERNAL_INTF # Send all other traffic through the primary IP. $IPFW add nat 1 ip4 from $NET_BRIDGE_CIDR to any out via $NET_EXTERNAL_INTF ########## # Catch-all # Allow everything # Normally I'd go for a whitelist-based firewall, but since we've got the # benefit of the AWS security group in front of us already doing that I'm # not _too_ worried about it. $IPFW add allow all from any to any
Run this firewall script (I highly recommend using the test script!). Your connection to the server
may drop temporarily, but you have 30 seconds to connect back in, reattach to your screen, and
kill the script before it rolls back to your old (wide open) rules. If everything's working, you
can copy the /etc/ipfw.rules.new
file over top of /etc/ipfw.rules
to
persist the changes through a restart.
The Conclusion
At this point everything should be in place for our next steps—actually starting up some jails. We'll do that in Part 4.