I have a firewalla purple. it’s idiot mode networking and I love it, but I have never been too thrilled with it’s cloud shit and really don’t to rely on it as my only option right now.
A while back I tried spinning up a VM with opnsense and never got good performance off my home ryzen server. I tried multiple NICs and even bare metal installs and while bare metal was a little more performant, it was never able to reach gigabit on WAN. the firewalla falls just a hair short of gigabit WAN but its still way ahead of my more muscular server. I notice the CPU load spikes high. it seems nothing I do can bring down that CPU load for opnsense. openwrt performed a bit better but still never hit gigabit speeds and was still below the firewalla’s performance. bare metal was again a bit better but still not matching the firewalla.
The firewalla is a heavily optimized amlogic based pi. it’s not special. but it works right and my crap doesnt. I have other SBCs I can use if folding into the home server as a VM just isnt practical but the server is always on anyway and already has extra resources I can throw into this so I’d like to just throw it all in there, snapshot a working config and be done with it if I can.
I walked away from this a while back thinking I would have a fix if I took a break and came back to it later but I’m still stumped. How are other people doing this?
Running it as a VM or even on a server that is running other services and potentially competing for I/O or memory bandwidth also introduces many other potential sources of inefficiency. I always recommend running a firewall on dedicated bare metal hardware, it is a very specialized task with very particular requirements on behalf of both the hardware and the software and it has very little tolerance for other sources of latency or delays. That doesn’t mean you need to use a pre-built appliance, but it does explain why it’s so common, and running it on a VM on a server that is doing other stuff is likely contributing to your issues significantly.
Personally, I run my firewall/router on a very stripped-down Debian with almost no non-essential services and a custom built kernel. I hand-picked a multi-port PCIe x4 Intel NIC with good Linux compatibility and drivers, and I’m using foomuuri to handle the routing and kea to handle DHCP/DNS for my internal network. This is a very minimal, bare-bones configuration and I wouldn’t really recommend it unless you really know what you’re doing, and it’s absolutely not “idiot mode networking” and if that’s what you want you’re going to have a real bad time if you try to follow in my footsteps, because I am a very different kind of idiot. But it works for me, so it’s proof that it is possible.