08
Apr
2018

FreeBSD: pfSense and OPNSense on Apollo Lake – Issues

Time to dig into some of the gremlins I came across when setting up my shiny new router and ultimately why I settled with OPNSense (and not pfSense).

20180330_170235

I intended to install pfSense since it was the one I had heard about the most as being the “bread-and-butter” of routing platforms. I was aware of other softwares such as m0n0wall or even going through and using a VM running the Unify controller software from Ubiquity on it – but ultimately I wanted to run pfSense as a starting point.

The system I am using is as follows:

  • Gigabyte GA-J3455N-D3H motherboard
  • Intel Celeron J3455 1.5-2.3GHz (Apollo Lake)
  • A SanDisk 32GB USB3 drive that I had lying around for the OS
  • 4GB DDR3L-1600

20180330_170435

I’ve read that you should only use Intel network interface cards (NICs) when setting up any FreeBSD-based box – I had a lot of issues finding reasonably priced boards featuring AES-NI, Intel NICs and a low power SoC – it’s like I had to pick two out of the three. I ended up taking a punt and trying out the J3455N to see if it’s Realtek-based NICs would at least handle my 80/20 connection.

I remember seeing somewhere that AES-NI was going to become mandatory for new versions of pfSense – so this was a feature I really wanted. Older boards (a lot based on the J1900) don’t support it – so I needed to go with something new. The low cost was also quite important since I had RAM and a USB install drive on hand. The main downside of this board is for some inexplicable reason, the one expansion slot is PCI. Not PCI-E – meaning I can’t use it for a dual gigabit Intel NIC in case the Realtek ones don’t work.

As with Ryzen, new hardware typically has “growing pains” – and this is exactly what I had with FreeBSD. pfSense and OPNsense both were very sensitive to BIOS settings – in a way I had not quite seen before. For starters, the board would not boot either pfSense or OPNsense with its default BIOS settings. This is a problem – meaning when (not if) the BIOS battery goes flat it’ll render my box unbootable and I’ll have to put it right again.

Okay, so I found that I still couldn’t boot pfSense – the system would freeze and the last line would always be:

Timecounter “HPET” frequency 19200000 Hz

This was weird – I had to select option 3 on the pfSense/OPNsense boot menu and enter the following commands:

set hint.hpet.0.clock=0
boot

And then it would boot normally. Except then I had a lot of other problems trying to shoehorn pfSense onto the install USB drive – the system would randomly hang.

I tried both the ISO and USB installer images of pfSense – I could not get it to install. I even tried an older version of pfSense to see if that would be any better – it wasn’t. I ended up stumbling across OPNsense as being a viable alternative (since a lot of other people also encounter issues installing pfSense from the looks of things).

So then I used the ISO version of OPNSense and “burned” it to a USB drive for booting – which worked just fine except I still encountered the HPET error and enter the same commands as before into the boot screen to get it to boot. I was finally able to install the base system to the boot USB and actually have the router booting the install. Except for the fact that each time the system was booted it would hang on the same HPET error – then I would have to reboot and go through the whole sequence of pressing 3, and entering the commands as above.

Clearly not a viable way of moving forwards with this – I hadn’t even got to setting up the network yet! I finally discovered that in order for FreeBSD-based OS’s to boot on Apollo Lake, you need to edit /boot/loader.conf.local and add hint.hpet.0.clock=0 into the config file. Once that was done, the system could then boot without intervention and I could then setup my network.

Opensense

It turns out that the Realtek 8111G NICs work fine – at least with my internet connection. I don’t see very much CPU usage (averages 0.2% over a period of months).

20180204_172404

Then I noticed something odd. The throughput readings were always through the roof on the router – for instance, I would be downloading a game through Steam at say, 7MB/s (55-60 Megabits) and the router would report it as being way over 200 megabits! This was very weird – but maybe it was adding up the values of multiple interfaces. I didn’t think too much of it – then I noticed that the clock was running really really fast on the router. It would boot and be almost correct, but after ten minutes it would be minutes fast.

This was clearly an issue with the “HPET” so to speak – from what I can tell it seems to influence the clock in certain systems. Something was clearly out of whack and I couldn’t work it out. The network worked fine – I was getting good performance and low ping times so I really wasn’t sure what was going on.

I did some more digging, and found a post online about Legacy Mode in the BIOS. I hadn’t touched this yet, and thought it was worth a shot. And then, bingo. Clock works properly, and now the BIOS buzzer actually beeps when OPNsense is booted! It’s like all the weird issues I was noticing had disappeared, and I wished I had spotted it sooner. The throughput readings also reported correctly too! Maybe pfSense would have worked now, but I was getting quite impressed with OPNSense at this point and I couldn’t be bothered going back to trying to install pfSense again.

So now it seems to be rock solid! There still lies the issue of the BIOS battery going flat and rendering it unbootable, but at least now I know (and I can refer to this blog post in the future) how to fix it and get it working again. The system keeps up well with my 80/20 connection, and I may have to find a way of benchmarking it to see how it could handle way faster speeds.

WhatsApp Image 2018-03-30 at 16.32.39

CPU temperatures are kept well in check – never exceeding 50 degrees even! You can tell it’s passively cooled since you can see the cycling of the central heating and when I left the heating off when I went on holiday.

WhatsApp Image 2018-03-30 at 16.28.35

And from the looks of it, the average CPU usage seems to be below 0.2% – even when I’m hammering the internet it doesn’t seem to even reach 0.5%! So yeah – a quad core is total overkill for routing jobs. 4GB RAM is also ridiculous… only 15% of it is actually used. I have enabled the tmpfs options in OPNSense for things like log files to keep the USB drive from being constantly pummelled.

This took me about two weeks to get going properly – with many late nights of googling and trying many different things. The main things to do for Apollo Lake are:

  • Enable Legacy Mode in the BIOS
  • Press 3, and type “set hint.hpet.0.clock=0” into the console for installation
  • add “hint.hpet.0.clock=0” into /boot/loader.conf.local

And then it works great! It’s time to delve into how to actually set up the bloody thing…

An Overview of my Home Network
The 2016 NAS: two years in!

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.