FreeBSD 12 Jail Host - Part 5 - Usage (2021)

Estimated reading time: Seal yourself in a dark room for a day or two.

At this point everything should be working. This document covers some general advice and points you towards a few commands that should be handy.

The Journey

All the parts in this series.

It really depends heavily on what sort of workload you plan on running, but some general thoughts:

You can't have an external port forwarding to more than one jail, and you can't have a jail receiving traffic from more than one external IP (i.e., primary and secondary). In this day and age, that probably means you're going to want to start up an instance of something like nginx per external IP to act as a reverse proxy for all the stuff that's going to want to glob onto 80/443.
You can share state between jails by mounting folders from the host system. For instance, I run acme.sh in its own jail. It retrieves and renews SSL certificates from LetsEncrypt and writes them out to a shared folder which is then mounted read-only in my proxy jails and the jails for other services which need to present a SSL certificate directly. (Anything that is purely internal just uses certificates signed by my private trusted root.)

Some extra handy commands.

To configure your jails to start at boot, run iocage set boot=on myjail. This also requires that iocage be set to start on boot by adding iocage_enable="YES" to /etc/rc.conf.
You can configure jails to depend on other jails with iocage set depends="otherjail1 otherjail2" myjail to ensure that those jails are started first.
You can mount host folders in a jail with something like iocage fstab -a myjail /path/to/host /path/in/jail nullfs rw 0 0 (replace rw with ro if the jail doesn't need to write to the folder).
You can set up "templates" in iocage to avoid some of the repetitive work (and save some disk space). Create a jail, set up and configure whatever services you want (e.g., your reverse HTTP proxy), then stop the jail and run iocage set template=yes mytemplate. You can then create new jails based on this jail with iocage create -t mytemplate -n myjail.

Some things you might want to run (and where available/necessary, FreeBSD-specific instructions for doing so).

No idea if you'll run into the same problems I did, but a couple of outstanding issues that I just haven't gotten around to resolving yet.

NAT dies every few minutes—For some reason every time dhclient ran all NAT would stop. I think it was due to it detaching and reattaching the configured IP addresses, causing the NAT configurations to lose track of which IP they were supposed to be using. For the time being I've just worked around this by shutting down dhclient after the system is up and running since the assigned IP will never be changing anyway.
iocage doesn't start on boot—No idea on this one, found a few people online running into the same problem with no real solution. It just doesn't seem to actually start on boot and I haven't been bothered enough to track this down yet. Running /usr/local/etc/rc.d/iocage start after booting brings everything up so it's something in the system startup process that's not actually trying to start this service.

Hopefully that helps, whoever you are! (Future me. It's likely future me.)