29
Oct
2018

OPNSense – Dark Theme!

OPNSense is great and all, however it’s quite the strain on the eyes to look at:

Phowar… blindin!

The UI is lovely and all, however when you’re messing with it late at night (like me), you might want to grab the Cicada theme:

Aaah… that’s better!

To enable the theme after getting it from System > Firmware > Plugins, go to System > Settings > General and select it from the Theme dropdown:

And that is it! There are other themes too, but I do like the black and orange look of Cicada!

19
Oct
2018

Adding another 4TB to the NAS

So, I managed to make it around two-and-a-half years before I started to get low on space in the 2016 NAS. It was time to add 4 more terabytes and expand the storage!

The NAS (top) sitting happy in the “rack”

The three 4TB Seagate Barracuda NAS drives I got in 2016 are still running perfectly, so I had no concerns in getting another to throw into the fourth bay.

The design of the drive has changed a little bit since they have now moved from ST4000VN000 being the model number to ST4000VN008. The drive still runs at 5900 RPM and is physically very similar to the others.

I am using RAID5 in the CentOS-based Rockstor – although it is based on BTRFS and isn’t “production ready” it has been working just fine for me. Than again, I haven’t suffered any drive failures.

I left the NAS running and installed the drive. It span up, then I went into Rockstor and used the “Resize/ReRaid Pool” option to add the drive into the main storage pool. It added successfully and informed me that it had started a re-balance to spread the data across the new drive.

The new storage space was immediately accessible:

14.5TB is what SMB reports before the RAID5 overhead is accounted for – really the drive pool is around 12TB. They really should fix this.

The rebalance took around two days to complete, but the pool was slow but usable until it was done. Now, lets wait until I run out of space again and then I will have to replace the entire drive set… that might be expensive! Let’s hope 8+TB drives come down in price before then!

18
Oct
2018

OPNSense 18.7: Higher CPU usage?

Just a short one – I made an observation going from OPNSense 18.1 to 18.7 that the average CPU usage has gone up slightly:

The first half of this year showed about 0.05% CPU usage on average, and after the update this has sprung up to 0.20-0.25%. 

The CPU is the Intel J3455, a 10W TDP chip with a Passmark score of 2144 points. This is a low power chip that would be more sensitive to the changes in background usage than for a desktop chip for instance – however the actual CPU usage is still tiny on average and I’m not worried about it (0.25% is still a tiny, tiny amount).

Zooming in on the data, you can see why:

Every 30 minutes there is a spike (though sometimes there isn’t one) – these were not there before the update. Some routine task has probably been added – though I have no idea what it is looking at the update notes.

I hadn’t changed anything about the network or setup between the updates, just to be clear. The spikes are quite a high usage – on the order of up to 6% CPU usage. This is still negligible though, and I can’t tell when it’s happening during online games or anything like that.

If someone knows what this is about or if they are getting the same behaviour – please feel free to let me know!

17
Oct
2018

OPNSense – Installing On Intel Apollo Lake

So, I’ve had some issues getting OPNSense (or pFsense, anything based on FreeBSD) to install and boot properly on my shiny new router. The router is based on the Intel J3455 CPU, which uses the Apollo Lake architecture.

The Gigabyte (Gigabyte GA-J3455N-D3H) board’s BIOS is rather basic – it does not allow you to configure ACPI or HPET settings; these are what cause issues in FreeBSD 11. To get around this, some stuff is needed to be done:

Booting the USB:

  • Ensure the CSM is enabled and set “Other Devices” to “Legacy”
  • Boot the USB installer in UEFI mode
  • Press “3” 
  • Type: set hint.hpet.0.clock=0
  • Type: set hint.acpi.0.msi=2 – repeat four times for “0” to “3”
  • Type: set machdep.disable_msix_migration=1
  • Type: boot

Install OPNSense/pFsense as normal.

Once installed:

  • When the machine reboots, repeat the steps above
  • Login to the console
  • Select the “Shell” option
  • Type: cd /boot
  • Use vi to add the following lines to loader.conf.local:
    • hint.hpet.0.clock=”0″
    • hint.ahci.0.msi=”2″ (add 4 lines from “0” to “3”)
    • machdep.disable_msix_migration=”1″
  • Reboot the machine

Now, the device should boot up without hanging or intervention required. This guide is what worked for me on this particular system installing to an SSD connected via SATA – hopefully FreeBSD 12 fixes this problem.

Hope this helps – good luck!

16
Oct
2018

Configuring DNS-over-TLS with OPNSense

Privacy is becoming more and more important in the ever-developing world of the Internet. Of course, you have the ongoing issues with security on the Internet – a lot of the main workings are ancient. And the other thing we now have more and more is targeted advertisement – where companies exist solely to gather as much information about an individual as possible and present to them adverts that they think would matter to them.

One mechanism that is used for this is the ancient DNS system – used to translate something like “google.com” to it’s IP address “216.58.212.110”. Typically, DNS operates unencrypted using UDP – where a request is sent in plain-text and is open to being interrupted and replaced with something else (called a “man-in-the-middle” attack).

maninthemiddleattack

The main problem here is that there is no way with traditional DNS of knowing if the returning answer is from the DNS server you originally requested. The effect of leaving everything unencrypted is allowing companies (like your ISP) to be able to log your DNS traffic (and maybe sell it to advertisers).

So we need to do two things:

  • Verify connections back from the DNS server are legit
  • Encrypt the connection

This is where DNS-over-TLS comes in! TLS uses a TDP connection to form a two-way conversation with the DNS server and then also encrypts the traffic to prevent snooping. There is also DNS-over-HTTPS which employs a similar method to “hide” your DNS traffic as standard HTTPS requests to websites.

TLS is an encrypted protocol that makes it extremely difficult (although not impossible) to read the DNS requests sent from your browser to the DNS server. A handshake is made using keys to ensure all the traffic is between the router and DNS server is legitimate. Mozilla is currently trialling built in DNS-over-TLS functionality in Firefox, however this still limits you to one browser on your desktop PC – you may also want to use your phone for instance. The best way to go about this is to perform the encryption of DNS requests in the router.

Fortunately, pfSense and OPNsense support the new 1.1.1.1 service provided by CloudFlare. There are alternatives (such as Quad9 – 9.9.9.9) that can also be used – infact you can use these as backups in case CloudFlare goes down (or use Quad9 as your primary). Either way it is recommended to use a secondary DNS service as a backup, and then a tertiary service such as 8.8.8.8 can be used as a final backup to ensure you never go offline.

I am running OPNsense – though the setup for pfSense is very similar to this. The bulk of the configuration is achieved through the Unbound DNS > General page:

Above is my configuration – you should add multiple “forward-addr” lines to allow for multiple DNS servers to be used if one or more are offline. You will want “DNS Query Forwarding” to be disabled.

And that is it! Once that is set, make sure that Unbound is restarted using the restart icon at the top of the screen. You can test if it works using the “Packet Capture” tool under Interfaces > Diagnostics and looking at data transmitted over port 853:

You will see your public IP connecting to the DNS service using TCP over port 853! 

And that’s it! Your DNS requests will now be encrypted. Safe browsing!

26
Apr
2018

Should I upgrade to Ryzen 2700X?

I’ve written about Flammenwerfer – my AMD Ryzen 1700X build that I put together last year. As is apparent on every tech YouTube channel at the moment, the Ryzen 2000 series has now launched. The question is – is it worth it for me to sell the 1700X and get a 2700X?

There are some things I have to think about with this.

  • Is the 1700X enough?

Yeah, probably actually. The most I tend to do is transcoding on it – and it’s a beast when it comes to that. Gaming-wise, the online benches do show an improvement in the 2000 series, but it’s only minor.

  • Would I need a new motherboard?

No, actually. From what I can tell the only main difference is StoreMI which allows you to combine slower and faster storage into a single drive – but I prefer keeping control over my storage so I wouldn’t use it anyway. The performance difference between X370 and X470 is negligible.

  • Would memory support be better?

Definitely – I’ve only just got DDR4-2933 working on my system (for a long while I was stuck at 2666). Even now the system does fail memory training every so often and boot loops a couple of times so that would be an improvement for sure. I’m already close to the maximum rated memory speed of my DDR4-3000 kit though so I wouldn’t see any real benefit.

  • Overclocking?

With the newer BIOS versions I have got the system to 3.95GHz on safe voltages on the 1700X – from the looks of things the 2700X does that kind of all-core speed out of the box and can reach the 4.2-4.3GHz mark. Would dropping £299 on ~250-300MHz be worth it?

  • Selling the 1700X?

I can find the 1700X on eBay (at the moment) for £150 ish – and I did keep the box. So that could half the investment of the 2700X.

Ultimately though, I bought the 1700X (pre-ordered, even) for full price – at the time around the £400 mark. That’s quite a decent investment I made – and I probably wouldn’t make back half its value today. Personally in my experience there’s never been a lot of return to me made with selling CPUs though – they depreciate in value extremely quickly. I sold my Phenom II X4 965 for practically a tenth of what I paid for it, so maybe making just under half back wouldn’t be so bad.

I’ve thought about it for a while – I’m not going to bother this time. The socket is expected to be supported for a good while – it may make more sense to wait another year or more for the next Ryzen to be released (or whatever it’s called).

Then I’d be happy to just keep the 1700X as a spare and as a sentimental item – it marks a return to the game after a ten year gap for AMD. I mean, have you seen how much Intel has had to pull their socks up to compete? You can absolutely thank AMDs Ryzen for your 6-core (and rumoured) 8-core desktop consumer Intel chips that have recently come out. We need competition – no matter which side of the fence you sit on.

I’d go for a new CPU and motherboard combination next time – X570 might have some actual new features I might care about and a new CPU architecture means more than a modest performance increase that we got this time around (~3%). Plus, the 1700X isn’t exactly slow anyway! Bring on next year!

25
Apr
2018

IPv6 bogons – OPNsense errors?

I must admit I am still getting used to a more complex firewall setup, as detailed in earlier posts. I’m constantly finding new things in the settings panel which have absolutely no applicable use to me, but absolutely do to those in the larger enterprise space.

That’s all well and good, and I’ve been tootling along just fine leaving most of the more advanced settings alone. Until…

bogonsmessage

[ There were error(s) loading the rules: /tmp/rules.debug:15: cannot define table bogonsv6: Cannot allocate memory – The line in question reads [15]: table persist file /usr/local/etc/bogonsv6]

This was weird. As far as I’m aware, my ISP doesn’t have IPv6 support yet – but I can’t ignore it since it will come in the future. I was having problems getting ports forwarded – I would apply the settings but I still couldn’t get through, unless I rebooted the router. This didn’t make sense to me – this was all working before. One nice thing about OPNsense is the ability to live view packets being allowed or blocked by the firewall – this way I could still tell the default rule was blocking packets that should be allowed through.

That message appeared at the top of the screen each time I applied the firewall settings. Hmm… What actually does it mean? I did some digging and discovered that the IPv6 bogon list was recently expanded – and this has now tipped over the default maximum number of firewall entries.

Since IPv6 is still relatively new, the bogon list (which specifies things like loopback addresses to block on the incoming side of the WAN – those sorts of things should never be on the incoming side so they’re blocked for security purposes) is still being updated.

The solution to this is to bump up the maximum size from the default.

firewallentries

I’ve set mine to 1 million since I have plenty of memory and CPU and didn’t want to have another bogon update trip the limit again. 200,000 is the default limit for versions around the time of writing this (though I’ve read that 400,000 is a decent setting to use). Maybe a million is too many but I’ll see!

Another option is to disable IPv6 bogon filtering in the Interfaces settings page – though IPv6 support may drop at any time I’m going to leave it enabled and just increase the firewall entry table size instead.

blockbogons

I wouldn’t recommend turning off bogon filtering since they are actually quite important, especially with IPv6 covering a much larger set.

Anyway – now that’s been dealt with I can move onto DNS-over-TLS (since it’s all over the web at the moment). Stay tuned!

10
Apr
2018

OPNsense/pfSense: What it brings to the table

I’ve just finished my 2018 router and am enjoying the experience so far. I’ve been wondering… what does it give me over a standard ISP-issued router, and what am I sacrificing? Well – I thought I’d list a few things I have come to appreciate over the last little while…

20180330_171401

I am running OPNsense – and so far it has been great. As for a routing platform it seems quite similar to pfSense – though I do prefer the look of the OPNsense UI (subjectively, mind). But this post isn’t about the software running – it’s about why I did this project at all. I mean, I could have just used the standard router the ISP issued me – and that was working very well before this project.

The first reason: fun! There’s something about setting up your own stuff to work the way you want it to – having ultimate control over everything to the point where you mess everything up and spend hours and hours troubleshooting. That, for me at least, is fun! It’s why I like Linux, it’s why I build my own PCs, its why I built my network from scratch. I believe it’s fine to break stuff by fiddling, as long as you can work out how to put it right again and learning along the way!

The second reason: security. OPNsense is updated on a regular basis with bug fixes and security patches:

alltheupdates.png

I bet there isn’t a single router anywhere on the planet that gets updated 4-5 times a month! You even get a full description of what goes into each update, rather than a generic “security fixes” reason that you would get from the likes of, for instance, Netgear.

The third reason: configuration. There are lots and lots of settings in this thing… many of which I don’t know what they do. That adds to the fun aspect of reason #1, but also gives you the ability to fine tune how you want things to behave. There is a very steep learning curve, and it took me a good while to figure out how to forward ports even! It gives you a level of control you won’t get from any off-the-shelf router.

The fourth reason: privacy. This seems to be getting more and more important as time goes on, and VPN’s are becoming more and more ubiquitous. You can use this to tunnel everything on your network through a VPN provider transparently, or only certain parts of your network. One big privacy-centred feature for me is being able to use DNS-over-TLS – where DNS requests that are typically unencrypted become unreadable by, for instance, your ISP or government. Future privacy-related content to come soon!

The fifth reason: reliability. I’ve had this setup running for over two months and not once have I had a problem with connectivity (except for ones I caused!). I’ve had many different routers in the past from ISPs and third-party companies in the past – not one hasn’t required me to reboot it every-so-often. Virgin Media routers especially are notorious for being unstable (in my own experience). Instead of having one box with a combined router, switch and WiFi access point, I have separated everything.

20180331_015340(0)

This separation means that if one bit stops working (often the WiFi bit, for instance) then that part can be rebooted without affecting other parts of the network.

Those are the top five reasons I did this – and the reward is so worth the effort and headache. Let’s go through what disadvantages I can think of:

  • Cost. The ISP-router was free. All in, this cost me just shy of £200 (though I did have a fair few bits lying around that I could use).
  • Power consumption (more detail here). It’s low – but not as low as a normal router.
  • Not being something that “just works”. This took a lot of setting up, and didn’t “Just work” for me.
  • The steep learning curve. I’m not a newbie when it comes to networking, and even for me this was a challenge. If you’re new to networks and all you’ve ever done is forward a port through a Belkin router, then this is not for you (unless you like a challenge and want to learn something – then it is definitely for you).

Would I do this again? Absolutely. I don’t think I’ll need to for a long while – this thing is so overkill its hilarious. I’ve yet to properly benchmark it but just based on the CPU and RAM usage – it’s barely even lifting a finger.

To conclude – I am very happy with this router. It’s far exceeded my expectations with what it is capable of, and I am slowly learning about what it can do. It feels like I discover something new each time I go into the admin panel and poke around – there’s a reason there are courses people go on to learn all this stuff. I mentioned before there will be future content on stuff it can do – so stay tuned!

09
Apr
2018

The 2016 NAS: two years in!

Time really does fly… Its been two years now since I set up the 2016 NAS.

The NAS is in its new home in the equipment rack (TV stand edition) – and fortunately for me and perhaps unfortunately for you I don’t have much to report.

The hard drives have been absolutely fantastic. They haven’t even batted an eyelid – still serving files just as well as the day I got them. The underlying OS, Rockstor, has been okay. I’ve had to redo some configs with regards to scheduled tasks but other than that its been grand!

I’ve been scrubbing the volume once a month and so far there hasn’t been any corruption. Maybe the usage of ECC RAM has helped – but there are no signs of any of the drives dying just yet.

The volume itself is getting a bit full now so I am tempted to expand the storage to 12TB by throwing in another 4TB drive and balancing the array. The prices of hard drives have come down slightly in price for 4TB drives but now 8TB drives are better value.

Its tempting to replace them all with 8TB drives to double the space but there isn’t anything wrong with the 4TBs. I’ve decided I don’t need so much on the NAS, so I have been removing/compressing old data and recovering space that way.

The snapshot functionality of btrfs has been a lifesaver too – I was running a Minecraft server and the save file got corrupted. Usually you’re stuffed, but I was able to roll back a snapshot from a few hours before and it was recovered!

Just a short one – I’m looking out for 4TB Seagate NAS drives on sale… Maybe there will be an upgrade post coming!

08
Apr
2018

FreeBSD: pfSense and OPNSense on Apollo Lake – Issues

Time to dig into some of the gremlins I came across when setting up my shiny new router and ultimately why I settled with OPNSense (and not pfSense).

20180330_170235

I intended to install pfSense since it was the one I had heard about the most as being the “bread-and-butter” of routing platforms. I was aware of other softwares such as m0n0wall or even going through and using a VM running the Unify controller software from Ubiquity on it – but ultimately I wanted to run pfSense as a starting point.

The system I am using is as follows:

  • Gigabyte GA-J3455N-D3H motherboard
  • Intel Celeron J3455 1.5-2.3GHz (Apollo Lake)
  • A SanDisk 32GB USB3 drive that I had lying around for the OS
  • 4GB DDR3L-1600

20180330_170435

I’ve read that you should only use Intel network interface cards (NICs) when setting up any FreeBSD-based box – I had a lot of issues finding reasonably priced boards featuring AES-NI, Intel NICs and a low power SoC – it’s like I had to pick two out of the three. I ended up taking a punt and trying out the J3455N to see if it’s Realtek-based NICs would at least handle my 80/20 connection.

I remember seeing somewhere that AES-NI was going to become mandatory for new versions of pfSense – so this was a feature I really wanted. Older boards (a lot based on the J1900) don’t support it – so I needed to go with something new. The low cost was also quite important since I had RAM and a USB install drive on hand. The main downside of this board is for some inexplicable reason, the one expansion slot is PCI. Not PCI-E – meaning I can’t use it for a dual gigabit Intel NIC in case the Realtek ones don’t work.

As with Ryzen, new hardware typically has “growing pains” – and this is exactly what I had with FreeBSD. pfSense and OPNsense both were very sensitive to BIOS settings – in a way I had not quite seen before. For starters, the board would not boot either pfSense or OPNsense with its default BIOS settings. This is a problem – meaning when (not if) the BIOS battery goes flat it’ll render my box unbootable and I’ll have to put it right again.

Okay, so I found that I still couldn’t boot pfSense – the system would freeze and the last line would always be:

Timecounter “HPET” frequency 19200000 Hz

This was weird – I had to select option 3 on the pfSense/OPNsense boot menu and enter the following commands:

set hint.hpet.0.clock=0
boot

And then it would boot normally. Except then I had a lot of other problems trying to shoehorn pfSense onto the install USB drive – the system would randomly hang.

I tried both the ISO and USB installer images of pfSense – I could not get it to install. I even tried an older version of pfSense to see if that would be any better – it wasn’t. I ended up stumbling across OPNsense as being a viable alternative (since a lot of other people also encounter issues installing pfSense from the looks of things).

So then I used the ISO version of OPNSense and “burned” it to a USB drive for booting – which worked just fine except I still encountered the HPET error and enter the same commands as before into the boot screen to get it to boot. I was finally able to install the base system to the boot USB and actually have the router booting the install. Except for the fact that each time the system was booted it would hang on the same HPET error – then I would have to reboot and go through the whole sequence of pressing 3, and entering the commands as above.

Clearly not a viable way of moving forwards with this – I hadn’t even got to setting up the network yet! I finally discovered that in order for FreeBSD-based OS’s to boot on Apollo Lake, you need to edit /boot/loader.conf.local and add hint.hpet.0.clock=0 into the config file. Once that was done, the system could then boot without intervention and I could then setup my network.

Opensense

It turns out that the Realtek 8111G NICs work fine – at least with my internet connection. I don’t see very much CPU usage (averages 0.2% over a period of months).

20180204_172404

Then I noticed something odd. The throughput readings were always through the roof on the router – for instance, I would be downloading a game through Steam at say, 7MB/s (55-60 Megabits) and the router would report it as being way over 200 megabits! This was very weird – but maybe it was adding up the values of multiple interfaces. I didn’t think too much of it – then I noticed that the clock was running really really fast on the router. It would boot and be almost correct, but after ten minutes it would be minutes fast.

This was clearly an issue with the “HPET” so to speak – from what I can tell it seems to influence the clock in certain systems. Something was clearly out of whack and I couldn’t work it out. The network worked fine – I was getting good performance and low ping times so I really wasn’t sure what was going on.

I did some more digging, and found a post online about Legacy Mode in the BIOS. I hadn’t touched this yet, and thought it was worth a shot. And then, bingo. Clock works properly, and now the BIOS buzzer actually beeps when OPNsense is booted! It’s like all the weird issues I was noticing had disappeared, and I wished I had spotted it sooner. The throughput readings also reported correctly too! Maybe pfSense would have worked now, but I was getting quite impressed with OPNSense at this point and I couldn’t be bothered going back to trying to install pfSense again.

So now it seems to be rock solid! There still lies the issue of the BIOS battery going flat and rendering it unbootable, but at least now I know (and I can refer to this blog post in the future) how to fix it and get it working again. The system keeps up well with my 80/20 connection, and I may have to find a way of benchmarking it to see how it could handle way faster speeds.

WhatsApp Image 2018-03-30 at 16.32.39

CPU temperatures are kept well in check – never exceeding 50 degrees even! You can tell it’s passively cooled since you can see the cycling of the central heating and when I left the heating off when I went on holiday.

WhatsApp Image 2018-03-30 at 16.28.35

And from the looks of it, the average CPU usage seems to be below 0.2% – even when I’m hammering the internet it doesn’t seem to even reach 0.5%! So yeah – a quad core is total overkill for routing jobs. 4GB RAM is also ridiculous… only 15% of it is actually used. I have enabled the tmpfs options in OPNSense for things like log files to keep the USB drive from being constantly pummelled.

This took me about two weeks to get going properly – with many late nights of googling and trying many different things. The main things to do for Apollo Lake are:

  • Enable Legacy Mode in the BIOS
  • Press 3, and type “set hint.hpet.0.clock=0” into the console for installation
  • add “hint.hpet.0.clock=0” into /boot/loader.conf.local

And then it works great! It’s time to delve into how to actually set up the bloody thing…