From Firewall to Homelab Server: WatchGuard Firebox M200/M300 on Gentoo and Linux 6.18 LTS

Second-hand WatchGuard Firebox M200 and M300 appliances are cheap. Once their Fireware subscriptions have run out, nobody wants them. Inside each one is a quad-core 64-bit PowerPC SoC, eight gigabit ports and an SD card that holds the whole operating system. That’s a good base for a small, always-on box that keeps DNS, DHCP and NTP running when everything else in the rack is down for maintenance.

This post covers the whole path. It starts with the stock WatchGuard SD card and its 3.12 kernel. Next comes a rebuilt 3.12 kernel that can run a modern userland. It ends with an unpatched mainline Linux 6.18 LTS kernel running Gentoo on both models, with every port working at line rate.

Hardware	WatchGuard Firebox M200 (NXP T1042, 4 × e5500) and M300 (NXP T2081, 4 × e6500)
Operating system	Gentoo Linux ppc64 (big-endian, OpenRC)
Software	WatchGuard U-Boot 2014.07, GCC 12 cross (Debian bookworm), NXP QorIQ SDK 1.9 (GCC 4.9.2)
Kernel	Linux 6.18.56 LTS (mainline, unpatched); stock 3.12.19-rt30 rebuilt with getrandom()

---

Why bother?

My homelab runs on a hypervisor cluster. That works well until the cluster itself needs maintenance, and then DNS, DHCP and time go down with it, and so does everything that depends on them. I wanted those three services on hardware that:

• uses little power and can stay on all the time;
• doesn’t depend on the hypervisor, shared storage or anything else in the rack;
• has plenty of Ethernet ports for a management network;
• boots from removable media, so a broken system is fixed by swapping an SD card.

A retired Firebox meets all four. The harder part is getting a current Linux onto it.

---

The hardware

	M200	M300
SoC	NXP T1042 (QorIQ T1)	NXP T2081 (QorIQ T2)
Cores	4 × e5500, 64-bit, no AltiVec	4 × e6500, 2 threads each (8 CPUs), AltiVec
Endianness	big-endian ppc64	big-endian ppc64
RAM	2 GB	4 GB DDR3L ECC
Storage	SD card	SD card
Bootloader	WatchGuard U-Boot 2014.07 in NOR flash	U-Boot in NOR flash
Ethernet	8 × 1 GbE: 3 direct PHYs + 5 via a Marvell 88E6171R switch	same
Console	RJ45 serial, 115200 8N1	same

Other things on the board: a Ricoh RS5C372A RTC, two Winbond W83793 hardware monitors, an Atmel I²C TPM, the SEC/CAAM crypto engine (which includes a hardware RNG), a watchdog, and 4 MB of CFI NOR flash. The boards are made by Lanner for WatchGuard.

What big-endian ppc64 means in practice

This affects everything that follows:

• Almost no binary distros and no container images. Nearly all current 64-bit PowerPC builds are ppc64le (little-endian). The options that still work are Gentoo ppc64, Debian ports ppc64 (sid only) and OpenWrt (M300).
• Go programs won’t run. Go’s ppc64 port has required POWER8 or newer since Go 1.9. That rules out CoreDNS, Blocky, Vault/OpenBao, most Prometheus exporters and similar tools. C and C++ software is fine.
• musl assumes AltiVec on ppc64, and the M200’s e5500 doesn’t have it, so musl-based systems don’t work on the M200.

The FMan microcode lives in NOR, so leave the NOR alone

The SoC’s Frame Manager (FMan), its packet engine, needs microcode (version 107.4.1 on the T1042). On these boxes the microcode is stored in NOR flash, and U-Boot loads it at power-on, before any kernel runs. As a result, even a blank SD card with your own kernel gets working Ethernet. You can see it in U-Boot:

=> printenv fman_ucode
fman_ucode=eff20000

Make a backup copy (shown later) and never erase or rewrite the NOR.

---

The serial console

All of this work depends on the serial console. Any USB-to-serial adapter with a Cisco-style RJ45 console cable will do. I use a cheap PL2303.

sudo picocom -b 115200 --logfile firebox-$(date +%F-%H%M).log /dev/ttyUSB0
# leave picocom: Ctrl-A then Ctrl-X
# alternative: sudo screen /dev/ttyUSB0 115200   (leave with Ctrl-A k)

To avoid sudo, add yourself to the dialout group. Keep a log of every session. When a kernel panics during boot, the log file is usually the only record of what happened.

The boot menu

At power-on you get the U-Boot banner, FMan microcode loading, and then a menu:

WatchGuard (SYSA)
WatchGuard (SYSB)
WatchGuard (SAFE MODE)
WatchGuard (SYSA BOOTDRIVE)
Hit any key to stop autoboot (except ENTER): 10

Press Space (Enter is ignored) to stop the countdown. Pick an entry with the arrow keys and press Enter. If you don’t press anything, it boots SYSA.

Looking around in U-Boot (read-only)

printenv                          # the whole environment; save a copy of this first
printenv bootcmd WGKernelfile
mmc rescan
mmc part                          # SD card partition table
ext2ls mmc 0:1 /                  # files on partition 1 (also 0:2, 0:3)

On my M200 the relevant parts of the environment were:

Variable	Value
bootcmd	run wgBootSysA
WGKernelfile / fdtfile	uImage_m200 / m200.dtb
SysARoot / SysBRoot	mmcblk0p3 / mmcblk0p2
wgBootSysA	ext2load mmc 0:3 of the DTB and kernel, then bootm, root on p3
wgBootSysB	the same from mmc 0:2, root on p2
loadaddr / fdtaddr	1000000 / fc0000

Boot a test kernel once, without saving anything

setenv WGKernelfile uImage_test
run wgBootSysA

Or by hand:

setenv bootargs root=/dev/mmcblk0p3 rw rootdelay=2 console=ttyS0,115200
ext2load mmc 0:3 1000000 uImage_test
ext2load mmc 0:3 fc0000 m200.dtb
bootm 1000000 - fc0000

The next power cycle boots the normal kernel again.

---

The SD card: an A/B layout you get for free

The stock 8 GB card has seven partitions:

Part	Size	FS	Content
p1	16 MB	ext2	boot: uImage_m200, uImage_m300 (the same file), m200.dtb, m300.dtb
p2	128 MB	ext2	SysB: Fireware recovery
p3	1.2 GB	ext2	SysA: the Fireware root filesystem
p5	760 MB	ext3	configuration and data: config.xml, certificates, backups
p6, p7	2–3 GB	ext2/3	temporary files and cache

The boot path is simple. U-Boot ext2loads the kernel and DTB from the root directory of p3 (not /boot) and passes root=/dev/mmcblk0p3. SYSB does the same from p2. That gives you an A/B scheme for free:

• SysA (p3) is production and the default boot;
• SysB (p2) is the test system, which you pick from the menu;
• a reboot without touching the console goes back to SysA.

My layout (on a larger card)

Image the stock card first (dd to a file, then compress) and keep it as your way back to stock. Then partition a new card so it matches what U-Boot expects:

Part	Size	Label	Content
p1	16 MB	M200_BOOT	stock uImage_m200 + m200.dtb (fallback)
p2	16 GB	M200_SysB	test system
p3	16 GB	M200_SysA	production system
p4	the rest	M200_DATA	/data: build output, binary packages, backups

sudo sfdisk /dev/sdX <<'EOF'
start=2048, size=32768, type=83, bootable
size=33554432, type=83
size=33554432, type=83
type=83
EOF

Filesystems a 3.12 kernel can actually mount

If you stay on the stock kernel, even for a while, a default mkfs.ext4 won’t mount. The boot ends with unsupported optional features (2c0). Turn off the features that 3.12 doesn’t support:

mkfs.ext2 -b 1024 -I 128 -O ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file \
  -L M200_BOOT /dev/sdX1
mkfs.ext4 -b 4096 -I 256 \
  -O ^extent,^64bit,^flex_bg,^huge_file,^dir_nlink,^extra_isize,^metadata_csum,^uninit_bg,has_journal,ext_attr,resize_inode,dir_index,filetype,sparse_super,large_file \
  -L M200_SysA /dev/sdX3

You end up with ext3 that ext4 can still mount. Mainline kernels mount it without problems, so the same partitions work with both kernels.

---

Stage 1: Gentoo on the stock 3.12 kernel

The first working system used WatchGuard’s own kernel (3.12.19-rt30) and DTB, plus a Gentoo ppc64 OpenRC stage3 unpacked onto SysA. You can pull the stock kernel’s config out of uImage with extract-ikconfig, or later from /proc/config.gz. WatchGuard’s kernel modules live under /lib/drivers/*.ko, not /lib/modules. The most important one is wg_dsa.ko, which drives the switch and creates eth3–eth7.

It worked, but it needed a lot of workarounds. The main cause is that 3.12 doesn’t have getrandom(2), which was added in 3.17. Current glibc’s arc4random() aborts when it gets ENOSYS. Here is what that meant in practice:

Wanted	Had to use on stock 3.12
OpenSSH	Dropbear (OpenSSH crashed)
Unbound	dnsmasq forwarding only (Unbound crashed)
keepalived	a shell-script VRRP replacement
devtmpfs	device nodes created by hand at boot (including ttyS0 in inittab)
normal OpenRC services	critical services started from local.d

It was usable, but brittle. The next step was to fix the kernel.

---

Stage 2: rebuilding 3.12 with getrandom()

WatchGuard publishes the GPL source for the Firebox when you ask for it, and it includes the kernel tree and their board device trees (mb_m200.dts, mb_m300.dts). Ask their support for the GPL source for your model. Those device trees later turned out to be the most valuable part of the package.

The toolchain problem

Building a 2014 kernel in 2026 isn’t easy:

• Current Ubuntu gcc (7, 9, 13) is too new. 3.12 only knows compiler-gcc3.h/compiler-gcc4.h, the in-tree dtc fails with modern flex/gcc (yylloc multiple definition), and there are ____ilog2_NaN link errors.
• The right compiler is the NXP QorIQ SDK 1.9 toolchain (gcc 4.9.2 for e5500). Its installer needs Python 2, which current Debian no longer ships.
• My first container used a “Linaro gcc 4.9” tarball that turned out to be GCC source, not a binary toolchain. Check what you downloaded before you spend an evening on it.

What worked: a debian:buster container (EOL; point apt at archive.debian.org). Buster still has python2.7, has a host gcc 8 that builds the old dtc, and has u-boot-tools. The NXP SDK toolchain is installed into /opt/fsl-qoriq/1.9. Work in a Linux filesystem, not on an NTFS or Windows-mounted path: the kernel tree contains files whose names differ only in case.

W=~/firebox-build; mkdir -p $W/docker $W/configs && cd $W
# Dockerfile: debian:buster + python2.7 + build-essential, flex, bison, bc, u-boot-tools,
#             then runs the NXP fsl-qoriq-glibc-i686-ppc64e5500-toolchain-1.9.sh installer
docker build -t wg-firebox-build --build-arg SDK=fsl-qoriq-glibc-i686-ppc64e5500-toolchain-1.9.sh docker/

tar xzf linux-3.12.19.tar.gz && chmod -R u+w linux-3.12.19   # WatchGuard's tarball is read-only
cd linux-3.12.19
patch -p1 < 0001-powerpc-binutils-2.24-athigh.patch   # binutils >= 2.24 rejects @h in LOAD_REG_IMMEDIATE
patch -p1 < 0002-getrandom-syscall.patch
cd ..
docker run --rm -u $(id -u):$(id -g) -v $W/linux-3.12.19:/build -v $W/configs:/configs:ro \
  wg-firebox-build build.sh /configs/my.config      # make uImage modules, modules_install into out/

Milestone 1: reproduce the stock kernel first

Before changing anything, build the GPL tree with the running config from /proc/config.gz and boot it from SysB. Check Ethernet, eth3–eth7, the RTC and the SD card. Only then start making changes.

Milestone 2: the actual changes

• getrandom() backport from upstream 3.17 (commit c6e9d6f38894) in drivers/char/random.c. It accepts GRND_NONBLOCK and GRND_RANDOM, and also treats GRND_INSECURE (from 5.6) as non-blocking, because current userland passes it.
• powerpc syscall wiring: getrandom at number 359 in systbl.h, with 355–358 as ni_syscall; __NR_getrandom 359, __NR_syscalls 360.
• Config: DEVTMPFS=y, DEVTMPFS_MOUNT=y, bridge/STP/LLC built in, and a LOCALVERSION suffix so the build can’t be mistaken for the stock kernel.

Quick test after booting:

python3 -c 'import os; print(os.getrandom(16))'

Result: OpenSSH and Unbound both run. Device nodes appear by themselves. The hand-made mknod workarounds could be deleted.

Entropy on an old kernel

3.12 doesn’t feed hardware RNGs into the entropy pool, and right after boot the pool sat around 130/4096. Use rngd and point /dev/hwrng at the CAAM engine rather than the TPM:

emerge sys-apps/rng-tools && rc-update add rngd default
# /etc/local.d/05-hwrng.start
grep -qw rng-caam /sys/class/misc/hw_random/rng_available && \
  echo -n rng-caam > /sys/class/misc/hw_random/rng_current

(echo -n matters, because 3.12 includes the trailing newline when it compares the name.) CAAM delivered about 800 kB/s from /dev/random. The TPM managed 2.5 kB/s.

Moving from Dropbear to OpenSSH without changing the host keys

for t in ed25519 ecdsa rsa; do
  dropbearconvert dropbear openssh /etc/dropbear/dropbear_${t}_host_key /etc/ssh/ssh_host_${t}_key
  ssh-keygen -y -f /etc/ssh/ssh_host_${t}_key > /etc/ssh/ssh_host_${t}_key.pub
  chmod 600 /etc/ssh/ssh_host_${t}_key
done
sshd -t
# switch over with a fallback, all in one line on the box:
rc-service dropbear stop; if rc-service sshd start; then rc-update del dropbear default; rc-update add sshd default; else rc-service dropbear start; fi

Once SysB had been stable for a while, I promoted it:

mount /dev/mmcblk0p3 /mnt/p3
rsync -aHAX --numeric-ids --delete -x / /mnt/p3/
# then set the root entry in /mnt/p3/etc/fstab to /dev/mmcblk0p3

---

Stage 3: mainline Linux 6.18 LTS

The 3.12 rebuild was a stopgap. The real goal was a current kernel with no vendor patches.

Earlier failed attempts

Before any of this worked, I spent a couple of weeks on 4.14, 5.10 and 6.19 kernels with hand-edited device trees based on the NXP T1042RDB reference board. None of them booted to a usable system. Looking back, the cause is clear: the reference board’s DTS describes different PHY addresses, a different switch wiring and different MAC ordering. The DTS was the weak point every time.

What changed this time was writing the device tree from WatchGuard’s own mb_m200.dts and their wg_dsa.c switch driver, on top of mainline’s SoC include files (t104xsi-pre.dtsi, t1042si-post.dtsi).

Build

The kernel is stock linux-6.18.x with no patches. The config is corenet64_smp_defconfig plus a fragment. A Debian bookworm container provides the big-endian cross compiler:

FROM debian:bookworm
RUN apt-get update && apt-get install -y --no-install-recommends \
      gcc libc6-dev gcc-powerpc64-linux-gnu binutils-powerpc64-linux-gnu make bc bison flex libssl-dev libelf-dev \
      u-boot-tools device-tree-compiler cpio kmod xz-utils python3 rsync ca-certificates git file

# build.sh, run inside the container with the kernel tree at /build and the config/DTS at /cfg
export ARCH=powerpc CROSS_COMPILE=powerpc64-linux-gnu-
cp /cfg/watchguard-m200.dts arch/powerpc/boot/dts/fsl/
make corenet64_smp_defconfig
scripts/kconfig/merge_config.sh -m .config /cfg/m200.config
make olddefconfig
make -j"$(nproc)" vmlinux modules
powerpc64-linux-gnu-size vmlinux                       # watch this number, see below
make fsl/watchguard-m200.dtb DTC_FLAGS="-p 8192"
make uImage
make INSTALL_MOD_PATH="$PWD/out" INSTALL_MOD_STRIP=1 modules_install

The build takes about three minutes on 16 cores. Copy uImage → uImage_m200 and the DTB → m200.dtb to the root of the SysB partition, and the modules to its /lib/modules/. Keep the old kernel next to them under a different name.

What mattered

1. Kernel size. U-Boot decompresses the kernel to address 0, and the DTB sits at 0xfc0000, so the kernel’s text and data must stay under about 15.7 MB or it overwrites its own device tree. corenet64_smp_defconfig produces 22.3 MB. Trimming it (NFS, UBIFS/JFFS2/NTFS, ftrace, full kallsyms, other NIC vendors, wireless; SATA/SCSI/USB as modules) and setting CONFIG_CC_OPTIMIZE_FOR_SIZE=y brought it down to 10.7 MB (uImage 4.9 MB). Check size vmlinux on every build.

2. CPU options. corenet64 enables AltiVec, which the e5500 doesn’t have:

CONFIG_ALTIVEC=n
CONFIG_E5500_CPU=y
CONFIG_PPC_E500MC=y
CONFIG_NR_CPUS=8

An e5500 build also runs on the M300’s e6500, so one kernel binary boots both models.

3. The switch. The 88E6171R sits on the FMan’s MDIO bus at SMI address 0x10 (multi-chip mode). It is handled by mainline’s mv88e6xxx DSA driver. Its port 6 is the CPU port, wired over RGMII to FMan MAC e8000. WatchGuard’s driver writes 0xc03e to the switch’s CPU ports, which means the switch adds the RX and TX RGMII delays. So the switch side gets rgmii-id and the MAC side gets plain rgmii with a fixed 1 Gbit link:

fm_e8000: ethernet@e8000 {
    phy-connection-type = "rgmii";
    fixed-link { speed = <1000>; full-duplex; };
};

mdio@fc000 {
    phy_port0: ethernet-phy@0 { reg = <0x0>; };   /* front port 0 -> MAC e4000 */
    phy_port1: ethernet-phy@1 { reg = <0x1>; };   /* front port 1 -> MAC e0000 */
    phy_port2: ethernet-phy@2 { reg = <0x2>; };   /* front port 2 -> MAC e2000 */

    switch@10 {
        compatible = "marvell,mv88e6085";
        reg = <0x10>;
        ethernet-ports {
            port@0 { reg = <0>; label = "lan3"; };
            /* ... port@1-3 = lan4-lan6 ... */
            port@4 { reg = <4>; label = "lan7"; };
            port@6 {
                reg = <6>;
                ethernet = <&fm_e8000>;
                phy-mode = "rgmii-id";
                fixed-link { speed = <1000>; full-duplex; };
            };
        };
    };
};

4. Frame size. With the switch up, the conduit interface reported an MTU 1508 error: a 1500-byte frame plus the 8-byte EDSA tag doesn’t fit the FMan’s default 1522-byte maximum. The fix goes into the kernel’s built-in command line, so U-Boot’s environment stays untouched:

CONFIG_CMDLINE="fsl_dpaa_fman.fsl_fm_max_frm=1600"
CONFIG_CMDLINE_EXTEND=y

5. MAC addresses. U-Boot sets them through the ethernetN aliases. List the aliases in WatchGuard’s order (on the M200, ethernet0 = e4000) and every port keeps its factory MAC.

6. Interface names. Mainline names FMan ports in address order and switch ports by their DTS label, so the names don’t match the 3.12 ones. A small OpenRC script in the boot runlevel (before net) renames them:

M200 hardware	mainline name	renamed to
MAC e4000 (front port 0)	eth2	eth0
MAC e0000 (front port 1)	eth0	eth1
MAC e2000 (front port 2)	eth1	eth2
switch ports, front 3–7	lan3–lan7	eth3–eth7

The script reads /proc/device-tree/model to pick the M200 or M300 mapping, and does nothing on 3.12. The same root filesystem boots either kernel, and /etc/conf.d/net didn’t change at all.

7. Smaller issues:

• NOR flash not probed: mainline’s SoC include marks the IFC bus as fsl,ifc only. Add simple-bus back in the board DTS, and mark every NOR partition read-only.
• ip_tables not found: since 6.17, legacy iptables needs CONFIG_NETFILTER_XTABLES_LEGACY and CONFIG_IP_NF_IPTABLES_LEGACY.
• PAMU (IOMMU) call traces at boot: CONFIG_FSL_PAMU=n. It isn’t needed here.
• FMan microcode: nothing to do. U-Boot already loaded it.
• Per-kernel module lists: OpenRC uses the most specific modules_<version> line, so both kernels can share one /etc/conf.d/modules.

# /etc/conf.d/modules: the 3.12 list stays as it is, this line is used by 6.x kernels only
modules_6="ip_tables iptable_filter iptable_nat ipt_MASQUERADE caam caam_jr tpm_i2c_atmel w83793"

Harmless messages you can ignore: PCIe BAR 0 … failed to assign (no PCIe devices are attached), an unrecognised SPI-NOR JEDEC ID, and the dummy0/tunl0/sit0 interfaces from the defconfig.

The M300

The M300 device tree started from OpenWrt’s working watchguard-firebox-m300.dts and was checked against WatchGuard’s mb_m300.dts and wg_dsa.c. The differences from the M200:

• front port 0 = MAC f2000 (PHY 0), port 1 = e0000 (PHY 1), port 2 = e2000 (PHY 2);
• the switch is the same 88E6171R at 0x10, but its CPU port 6 connects to MAC e6000;
• U-Boot loads uImage_m300 + m300.dtb from the same places and addresses.

I put both models’ kernel and DTB files on p1/p2/p3, so one card boots in either box. The M300 came up on its first boot: 8 CPUs, 3.9 GB RAM, the switch, all ports, CAAM, TPM, sensors and RTC all working. The only new messages were extra PCIe window warnings and a second sensor chip that complains about bank selection.

Rescue without a power cycle

If a test kernel boots but networking doesn’t, you can recover over serial. Send a BREAK followed by s, u and b (magic SysRq: sync, remount read-only, reboot). The box falls back to SysA without anyone pulling the plug. In picocom, Ctrl-A Ctrl-\ sends a BREAK. I scripted it with pyserial so it works unattended.

---

Results

Internet speed (box as the endpoint, 900+ Mbit/s FTTP line, 15 s HTTP tests)

	down, 1 stream	down, 4 streams	up, 1 stream	up, 4 streams
desktop PC (reference)	533 Mbit/s	–	109 Mbit/s	–
M200 (4 × e5500)	533 (CPU 41 %)	904 (CPU 68 %)	106 (11 %)	110 (32 %)
M300 (8 × e6500)	533 (CPU 21 %)	910 (CPU 26 %)	105 (4 %)	110 (14 %)

Both boxes fill the line. The single-stream limit comes from the path to the test server, since the desktop gets the same number. One caveat: HTTPS on the M200 is much slower (about 104 Mbit/s for one stream), because userspace TLS without hardware AES is limited by the e5500 CPU.

LAN throughput (to a VM on a 10 Gbit uplink, 10 s TCP tests)

	box receives, 1 / 4 streams	box sends, 1 / 4 streams	CPU busy (rx1 / rx4 / tx1 / tx4)
M200	825 / 939 Mbit/s	939 / 941 Mbit/s	51 / 59 / 21 / 27 %
M300	936 / 941 Mbit/s	941 / 941 Mbit/s	24 / 20 / 4 / 6 %

941 Mbit/s is the maximum TCP payload on a 1 Gbit port with 1500-byte frames, so both boxes run at line rate. The only exception is a single receive stream on the M200, where one core does all the work.

Everything else

Both models boot from power-on to a login prompt in about 40 seconds. All eight ports, the switch, SSH, dnsmasq (DNS and DHCP), chrony, tftpd, rngd, iptables NAT, the CAAM RNG, the TPM, the sensors, the RTC, the watchdog and read-only NOR all work on stock 6.18.

---

Keeping Gentoo up to date

The boxes never run emerge themselves. A 1.2 GHz e5500 compiling a toolchain would take days, and the SD card would take the wear. Instead the system is built and updated on a desktop PC, then shipped to a box as a complete root filesystem:

• a ppc64 Gentoo chroot running under QEMU user emulation: this is the “master” copy of the Firebox system;
• C and C++ compiles are sent over distcc to a native x86 cross-compiler in a container;
• the result is copied to a box’s SysB, tested, then promoted to SysA as before.

Piece	What it is
Build root	a copy of a box’s root filesystem in a directory on the PC, kept between updates
Emulator	static qemu-ppc64 (QEMU 10), registered with binfmt_misc
Chroot helper	a script that mounts everything and runs a command in the build root
distcc container	native Gentoo with a crossdev powerpc64-unknown-linux-gnu toolchain, serving distccd on port 3632
Caches	distfiles and binpkgs directories on the PC, shared with the chroot

The emulated chroot

The kernel runs ppc64 binaries through QEMU once the format is registered with binfmt_misc. The F flag loads the interpreter at registration time, so it also works inside the chroot without copying QEMU into it:

Q=/usr/local/bin/qemu-ppc64-static
printf '%s' ":qemu-ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:$Q:F" \
  > /proc/sys/fs/binfmt_misc/register

The helper script then mounts what Portage needs and keeps the PC-only settings out of the box’s own make.conf, by passing them as environment variables:

R=~/firebox-build/gentoo-root
for d in proc sys dev dev/pts; do mountpoint -q $R/$d || mount --bind /$d $R/$d; done
# /dev/shm is a separate mount on the host: give the chroot its own, world-writable one
mount -t tmpfs -o mode=1777,nosuid,nodev tmpfs $R/dev/shm
mount --bind ~/firebox-build/distfiles $R/var/cache/distfiles
mount --bind ~/firebox-build/binpkgs   $R/var/cache/binpkgs
mount -t tmpfs -o size=16G tmpfs $R/var/tmp/portage          # build in RAM
mount --bind /etc/resolv.conf $R/etc/resolv.conf             # the box's own file stays untouched

export FEATURES="buildpkg distcc -pid-sandbox -network-sandbox -ipc-sandbox" \
       DISTCC_HOSTS="127.0.0.1:3632/24" MAKEOPTS="-j24" EMERGE_DEFAULT_OPTS="--jobs=3" \
       PKGDIR=/var/cache/binpkgs DISTDIR=/var/cache/distfiles
chroot $R /bin/bash -l

Use 127.0.0.1, not localhost, in DISTCC_HOSTS: distcc treats localhost as “compile locally”, which means compiling under emulation again. After installing distcc in the chroot, run distcc-config --update-masquerade once, or the compiler wrappers in /usr/lib/distcc/bin never get created.

The distcc cross-compiler

The container is a normal amd64 Gentoo with a crossdev toolchain for the boxes’ target. The versions are pinned to exactly the chroot’s gcc, binutils and glibc:

FROM gentoo/stage3:latest
ARG SNAPSHOT=20261012
ARG GCC=16.2.1_p20260926
ARG BINUTILS=2.47
ARG GLIBC=2.43-r4
ARG TARGET=powerpc64-unknown-linux-gnu
RUN emerge-webrsync --revert=${SNAPSHOT} --quiet
RUN getuto && emerge --quiet --getbinpkg --noreplace sys-devel/crossdev sys-devel/distcc app-eselect/eselect-repository
RUN eselect repository create crossdev
RUN MAKEOPTS="-j$(nproc)" crossdev --stable -t ${TARGET} \
      --binutils "${BINUTILS}" --gcc "${GCC}" --libc "${GLIBC}" --kernel 6.18 --portage --quiet
EXPOSE 3632
CMD ["distccd", "--no-detach", "--daemon", "--log-stderr", "--allow", "127.0.0.1", "--allow", "172.16.0.0/12", "--jobs", "16"]

Only allow the networks that need it: here, localhost and Docker’s bridge networks.

A routine update

# as root on the PC, with the distcc container running
fbx-chroot /bin/bash -lc 'emerge --sync && eselect news read new'
fbx-chroot /bin/bash -lc 'emerge -puDN --getbinpkg --with-bdeps=y @world'    # read the list first
# if sys-apps/portage or the default Python changes, update Portage on its own first:
fbx-chroot /bin/bash -lc 'emerge -1 --getbinpkg sys-apps/portage'
fbx-chroot /bin/bash -lc 'emerge -uDN --getbinpkg --with-bdeps=y @world'
fbx-chroot /bin/bash -lc 'emerge --depclean -p'                              # review, then without -p
fbx-chroot /bin/bash -lc 'emerge @preserved-rebuild; dispatch-conf'

(fbx-chroot is my name for the helper script above.) Gentoo publishes official binary packages for ppc64, so most of an update arrives prebuilt and signed. Run getuto once per build root to set up the signing keyring; without it, every binary package fails with “GnuPG verification failed”. Only the rest is compiled through distcc.

Run long updates in the background with a log, and watch for stalls (a package with no progress for 20 minutes), not just for errors.

Shipping it to a box

1. Unmount everything the helper mounted in the build root.
2. Copy the build root to the box’s SysB partition (rsync -aHAX --numeric-ids --delete, or tar over SSH). Keep the box’s own identity: hostname, /etc/conf.d/net, the SSH host keys, /etc/fstab (root = its own partition), the DNS/DHCP listen address and the random seed. The kernel files at the partition root and /lib/modules/6.18* come from the kernel build, not from the chroot.
3. Boot SYSB from the U-Boot menu, test, and promote it to SysA.

When gcc changes

distcc only works if the container’s cross-compiler produces exactly what the chroot’s gcc would. After an update that changes gcc, binutils or glibc: switch the chroot to the new version (gcc-config, binutils-config), change the ARGs in the Dockerfile, rebuild the container (about 30 minutes), and check once that both sides agree:

# run in the chroot and in the container, and compare the output
gcc -Q --help=target -O2 | grep -E "mabi|mcpu|mtune|maltivec|mlong-double|mcmodel"
gcc -O2 -c t.c && objdump -d t.o

What the first big update taught me

The first update covered 233 packages after five and a half months without one. Most of the time went on these:

Problem	Cost	Fix
Binary package keyring missing: every official binary rejected	~15 min	getuto once per build root
Everything compiled under emulation before distcc was set up	~5 h	distcc from the start
Ubuntu 22.04’s QEMU 6.2 lost a SIGCHLD: ninja install waited forever for a child that had already exited	1 h	QEMU 10 (static qemu-user from Debian trixie)
Old Portage (Python 3.13) crashed while installing the new Portage, as Python 3.14 became the default	10 min	update Portage on its own first
distcc’s compiler wrappers were never created in the chroot	–	distcc-config --update-masquerade
/dev/shm in the chroot was a root-only directory: pillow’s parallel build failed with [Errno 13] Permission denied	30 min	mount a mode-1777 tmpfs on /dev/shm
Five and a half months of updates in one batch	the rest	update monthly

The gain from distcc: cmake 4.3.5 took about 70 minutes emulated and 17 minutes with distcc. Portage itself, configure scripts, linking and Python/Perl packages still run under emulation, so expect roughly 3–5× for C/C++-heavy packages, not more.

Two options for later: run Portage natively on the M300 (8 threads, 4 GB, build in tmpfs) with the same distcc container, so nothing is emulated at all; or move both containers to a small build VM, so the PC doesn’t need to be on.

---

Lessons learned

1. Start from the vendor’s device tree, not the reference board’s. Two weeks of failed mainline boots came down to a DTS based on the T1042RDB. Once I wrote it from WatchGuard’s own mb_m200.dts and switch driver, the first mainline boot came up with working networking.
2. Ask for the GPL source. It was free and took a few days. It contained the device trees, the switch driver that showed the RGMII delay setup, and the U-Boot environment.
3. Reproduce before you modify. Rebuilding the stock kernel unchanged first meant that every later failure had one cause. It also exposed the EDSA/DSA mismatch before it got mixed up with my own changes.
4. Use the A/B partitions. Every risky change went to SysB first, and a reboot without touching the console undid it. I never needed to reflash anything or touch the NOR.
5. Don’t touch the U-Boot environment. Everything here, including the frame-size fix, works with saveenv never run once. Kernel parameters you need can go into CONFIG_CMDLINE.
6. Watch the kernel size. The ~15.7 MB limit doesn’t produce a helpful error. The kernel just dies silently or loses its device tree.
7. Old toolchains: check what you downloaded. “gcc 4.9” turned out to be source code, and the “right” NXP toolchain needed Python 2. An EOL Debian container from archive.debian.org is a reasonable way to get an old build environment.
8. Keep one root filesystem that boots both kernels. Renaming interfaces to the old names and using per-version module lists made the switch to mainline a kernel swap, with no userland changes.
9. Big-endian ppc64 limits your software choices. Check before you plan services: no Go, no containers, few binary distros. C and C++ software, which covers most of the classic infrastructure daemons, runs fine.
10. Log every serial session. Almost every fix above came from reading a log of a boot that failed.
11. Don’t build on the box. An emulated chroot on a PC, a matching cross-compiler over distcc and official binary packages turn a days-long job into an evening, and the box only ever receives a tested image.

---

Next steps on my side: a soak test before promoting 6.18 to SysA, Unbound as a validating resolver, keepalived between the two boxes (now possible on a modern kernel), and moving from iptables to nftables.