Lessons my DIY file server taught me about the 'Out-of-the-loop performance problem'

tl;dr: Roll your own file server.

In my recent post I uncovered a bunch of settings needed to fix problems with MacOS clients unable to properly browse my in-home file server. It wasn’t a complicated fix, but it required a deep dive of MacOS-specific settings in Samba. And a lot of trial and error over multiple days.

As I reflect back on this issue, it kept reminding me of this post on The Jurassic Park Effect by Scott Alan Miller. In the article, the author explains that NAS OS’s allow novices to create a production storage environment that’s “dangerously easy” to set up. And in doing so, these OS’s add unecessary complexity, functional limitations, upstream update lag, a possibility of abandonment, and… they shield their operators from understanding basic system fundamtenals necessary to triage and repair problems with the system. This is a perfect illustration of the out-of-the-loop performance problem. All of this has been on my mind recently.

I first found Miller’s article in 2021. And after these 3 years, I can tell you: He’s absolutely right.

My home file server (and, homelab in general) became a great case study for this topic.

A home file server is a relatively straightforward appliance. Running a simple *nix box and a Samba share isn’t tough. But when ZFS came along, I knew I wanted to use it, but I was worried I didn’t know how to use it properly, so I turned to NAS OS’s for expert guidance.

TrueNAS went through a total rewrite shortly after I started using it, and it got too bloated and “dashboardy.” The streamlined simplicity of XigmaNAS (née NAS4Free) caught my attention, and I used it for several years. It worked smoothly and the team didn’t load it up with flashy updates. I fixed up parts of their documentation site as a show of thanks to the project.

But after a while, I found that I needed to grow past what that OS could provide for me. Features in the OS started to turn into roadblocks. My education around ZFS was stunted because I was letting someone else make decisions for me. And I wanted to do more with the server than the pared-down image was built to allow.

Miller was absolutely correct when he said that a NAS OS is “a more complex and less functional version” of the underlying OS it’s built on top of. It’s true for any and every NAS OS out there. New ones are coming and going all the time, and none of them, even the ones built with simplicity and freedom in mind, can escape this truth.

I decided that the training wheels needed to come off.

Hunkering down

Over the course of a few months I taught myself slowly and carefully about all the things needed to replace these NAS OS’s. I spun up a barebones FreeBSD server, tested building ZFS pools, running Samba shares, FTP servers, playing with snapshots and transferring datasets around, and a lot more. It was challenging, and fun! And my tinkering didn’t stop there.

Just two weeks ago I completed a migration and consolidation of my entire home lab onto a single FreeBSD server. My dev environments, experiements, and various services are running in jails. I finally mastered bhyve to the point where I was able to retire my Proxmox host so I can virtualize different OS’s like Debian and Windows.

Not only has my setup becomed streamlined and simplified, but my personal knowledge and confidence around these topics has grown tremendously.

Looking back

I can’t ever imagine using a NAS OS ever again at this point. There’s no feature one of these systems could ever offer that would draw me in. Ultimately, I don’t want to be disconnected and kept out-of-the-loop.

I want to be in-the-loop.

I want to know how a vital system like a file server works. I want to know how to build it, configure it, maintain it, and most importantly — how to repair it.

But that takes time, effort, aptitude, and space. I can understand why people turn to NAS OS’s. Some people don’t want to bother with all of this work. And that’s totally fine! It’s just a decision that imposes some limitations and a certain amount of risk.

In my case, I didn’t roll my own because I was simply scared of what I didn’t know. And like the saying goes, I didn’t know what I didn’t know. When it comes to my data, I wanted to be conservative and err on the side of caution. I wanted expert help and guidance. But after a while, once I warmed up to the idea and had time and space to learn, I was able to “earn the knowledge for myself” (Ian Malcom, Jurassic Park). I’m truly glad I made this decision.