Hi guys!
I have read a lot about SLOG and special vdev and think it is fascination technology. Thank you again @HoneyBadger for explaining it so well.
Anyway I wanted to write a post about my future homelab and hoped that you guys could give me some advice. Maybe someone else already kinda made this setup or maybe this could help someone else looking for a home lab. This is gonna touch on a lot of topics and probably is gonna be complicated to discuss. Because of that I try to give the answers or statements numbers hoping this helps to structure the discussion.
But of course you are also very welcome to critique other points of my setup. Main main reason for upgrading is noise! Icing on the cake would be to bring down the 250W base load. Next year that will cost me 650$ per year.
To start with, I think it is probably best to take a look at my current use case and setup and why I wanna switch hardware and the architecture. Most of my hardware I got for free and has grown over time...
OPNsense: I have a old Intel NUC as a "router-on-a-stick" OPNsense. 4GB RAM, single SSD. Works fine. Only downside is missing IPMI and SFP+. My ISP offers a free upgrade to 10Gbit, so yeah, SFP+ would be nice.
TrueNAS: Supermicro Board, Pentium G4560, 64GB ECC. Case is a Rosewill 4500. I hate this case, it is the cheapest case I ever owned, with sharp corners. Two old SSDs for
boot mirror. Two 5bay hotswap caddies. My data pool consists of three vdevs that are mirrors. 4x 8TB disks and 2x 16TB disks. I have three datasets and NFS shares, one is for nextcloud data, the other one is for Proxmox Backups and last one is for family videos and raw footage of my recordings. Rsync backups Proxmox and Nextcloud to an off-site old QNAP NAS. The hot swap caddies turned out to be less usefull than I thought they would. Only one drive die on me and even then I could have handled the downtime by shutting down TrueNAS. I mostly used it to wipe disks from old devices I recycled.
The biggest problem I have with this setup is the noise. The caddies don't have sound dampening and the case itself does not have it either.
Gaming PC: Has a 500GB SSD and a 2TB HDD for games that don't need fast storage. Would be cool to remove the HDD and use iSCSI.
Windows PC: Old Lenovo SFF. Has Plex installed, because some family members have not made the switch to Jellyfin. Is my Windows client to play around and test things. Has a 8TB disk that runs Storj just for fun.
Ubuntu PC: Old Lenovo SFF. Has Jellyfin installed. Can hardware transcode thanks to Intel Quick sync. Wanted to migrate this host to Proxmox but never managed to passtrough the GPU. Now all devices can direct play and transcoding is no longer needed.
Proxmox: Supermicro Board, 32GB ECC, two nvme SSDs mirror for boot and VM storage. There are 10 VMs running selfhosted services. They don't need to much CPU power and also not a lot of disk io. For example, one VM is hosting Mumble, a Teamspeak alternative. That VM is mostly idling, the sharpest IO increase is during updates. A old 4TB single HDD is for unimportant stuff, like a Linux Gameserver and a qbittorrent. I did not put torrents on ZFS, because someone in the forums said, that this will badly fragment ZFS pools.
1: Is that still true if you use SLOG? SLOG does not help with fragmentation at all.
2: Why don't I use docker? When this whole setup started 5 years ago, I have heard a lot of people talk negative about Docker. Some said it is not secure (not that important to me, also I think these issues are gone nowadays). Also performace should not be great and it is bloated. I think a lot of that has changed over the years and a lot of selfhosted services I use offer Docker support, for some this has even become the recommended way to install. On the other hand, with VM templates and Cloud init, I am pretty happy with the way Proxmox has different VMs for different use cases. Would you recommend me looking deeper into Docker and Kubernetes and make the switch? Maybe I have become a lazy old man and I should make the switch to containers? Can you bluebill me on containers? ;)
Hardware wise, I am currently looking into buying a Fractal Define R7 XL. The main reason is, that I think this will bring down HDD noise.
3: The sound dampened case and the rubbers on the disk hopefully help with that.
So now I am thinking about where to go next.
4: Go all in on TrueNAS! Replace all devices with one single TrueNAS system. Replace most of my services with docker and use a few VMs for stuff that does not work in docker (like OPNsense). It would be so cool to have only one system. On the other hand, passtrough NICs to OPNsense sounds like a complexity nightmare :)
5: Still have multiple devices. TrueNAS, a Firewall and a Hypervisor but they both use iSCSI for boot disks and NFS share for VMs. I would only have to manage a single Storage System.
6: Boring option I currently have. Fast local SSDs for the hypervisor and VMs and TrueNAS for "slow" storage.
If I decide to go for 4 or 5, how bad will the performance penalty be? I haven't touched OS with HDDs for over 7 years :) Also I assume that by
7: using SLOG and three mirrored vdevs, performance would be still decent?
8: Read performance should be pretty good if I have 128GB RAM for ARC?
9: How about using one fast nvme SSD for SLOG and two SSDs partitioned into two partitions, one for boot, the other for vdev?
10: Percentage of used storage will sooner or later be 80%. A friend will backup his TrueNAS to mine. So I will not be able to follow the "only 50% full for VMs" rule. Will using 80% have a big impact?
I have read a lot about SLOG and special vdev and think it is fascination technology. Thank you again @HoneyBadger for explaining it so well.
Anyway I wanted to write a post about my future homelab and hoped that you guys could give me some advice. Maybe someone else already kinda made this setup or maybe this could help someone else looking for a home lab. This is gonna touch on a lot of topics and probably is gonna be complicated to discuss. Because of that I try to give the answers or statements numbers hoping this helps to structure the discussion.
But of course you are also very welcome to critique other points of my setup. Main main reason for upgrading is noise! Icing on the cake would be to bring down the 250W base load. Next year that will cost me 650$ per year.
To start with, I think it is probably best to take a look at my current use case and setup and why I wanna switch hardware and the architecture. Most of my hardware I got for free and has grown over time...
OPNsense: I have a old Intel NUC as a "router-on-a-stick" OPNsense. 4GB RAM, single SSD. Works fine. Only downside is missing IPMI and SFP+. My ISP offers a free upgrade to 10Gbit, so yeah, SFP+ would be nice.
TrueNAS: Supermicro Board, Pentium G4560, 64GB ECC. Case is a Rosewill 4500. I hate this case, it is the cheapest case I ever owned, with sharp corners. Two old SSDs for
boot mirror. Two 5bay hotswap caddies. My data pool consists of three vdevs that are mirrors. 4x 8TB disks and 2x 16TB disks. I have three datasets and NFS shares, one is for nextcloud data, the other one is for Proxmox Backups and last one is for family videos and raw footage of my recordings. Rsync backups Proxmox and Nextcloud to an off-site old QNAP NAS. The hot swap caddies turned out to be less usefull than I thought they would. Only one drive die on me and even then I could have handled the downtime by shutting down TrueNAS. I mostly used it to wipe disks from old devices I recycled.
The biggest problem I have with this setup is the noise. The caddies don't have sound dampening and the case itself does not have it either.
Gaming PC: Has a 500GB SSD and a 2TB HDD for games that don't need fast storage. Would be cool to remove the HDD and use iSCSI.
Windows PC: Old Lenovo SFF. Has Plex installed, because some family members have not made the switch to Jellyfin. Is my Windows client to play around and test things. Has a 8TB disk that runs Storj just for fun.
Ubuntu PC: Old Lenovo SFF. Has Jellyfin installed. Can hardware transcode thanks to Intel Quick sync. Wanted to migrate this host to Proxmox but never managed to passtrough the GPU. Now all devices can direct play and transcoding is no longer needed.
Proxmox: Supermicro Board, 32GB ECC, two nvme SSDs mirror for boot and VM storage. There are 10 VMs running selfhosted services. They don't need to much CPU power and also not a lot of disk io. For example, one VM is hosting Mumble, a Teamspeak alternative. That VM is mostly idling, the sharpest IO increase is during updates. A old 4TB single HDD is for unimportant stuff, like a Linux Gameserver and a qbittorrent. I did not put torrents on ZFS, because someone in the forums said, that this will badly fragment ZFS pools.
2: Why don't I use docker? When this whole setup started 5 years ago, I have heard a lot of people talk negative about Docker. Some said it is not secure (not that important to me, also I think these issues are gone nowadays). Also performace should not be great and it is bloated. I think a lot of that has changed over the years and a lot of selfhosted services I use offer Docker support, for some this has even become the recommended way to install. On the other hand, with VM templates and Cloud init, I am pretty happy with the way Proxmox has different VMs for different use cases. Would you recommend me looking deeper into Docker and Kubernetes and make the switch? Maybe I have become a lazy old man and I should make the switch to containers? Can you bluebill me on containers? ;)
Hardware wise, I am currently looking into buying a Fractal Define R7 XL. The main reason is, that I think this will bring down HDD noise.
3: The sound dampened case and the rubbers on the disk hopefully help with that.
So now I am thinking about where to go next.
4: Go all in on TrueNAS! Replace all devices with one single TrueNAS system. Replace most of my services with docker and use a few VMs for stuff that does not work in docker (like OPNsense). It would be so cool to have only one system. On the other hand, passtrough NICs to OPNsense sounds like a complexity nightmare :)
5: Still have multiple devices. TrueNAS, a Firewall and a Hypervisor but they both use iSCSI for boot disks and NFS share for VMs. I would only have to manage a single Storage System.
6: Boring option I currently have. Fast local SSDs for the hypervisor and VMs and TrueNAS for "slow" storage.
If I decide to go for 4 or 5, how bad will the performance penalty be? I haven't touched OS with HDDs for over 7 years :) Also I assume that by
7: using SLOG and three mirrored vdevs, performance would be still decent?
8: Read performance should be pretty good if I have 128GB RAM for ARC?
9: How about using one fast nvme SSD for SLOG and two SSDs partitioned into two partitions, one for boot, the other for vdev?
10: Percentage of used storage will sooner or later be 80%. A friend will backup his TrueNAS to mine. So I will not be able to follow the "only 50% full for VMs" rule. Will using 80% have a big impact?