SLOG Device: Kioxia CM5 vs Intel Optane P4800X?

[helloadam]

We are building out a high end ZFS storage array for our VMware datastore to be backed by NFS in our colo facility.

For NFS sync writes, what is the community experience between

• Kioxia CM5 drive - KCM5XVUG1T60
• .. vs ...
• Intel Optane P4800X for a SLOG device?

Or would you recommend something totally different/newer? Regardless of which we choose, we would run both in a mirror.

Our goal is to have the SLOG be as close to system memory (sync=none).

• Is a stripe mirror of SLOG devices possible / make sense?

Our setup includes:

• Network connection is 10Gbps Fiber with jumbo frames in private vlan/L2
• Pool is: 10 vdevs of 3-way mirror of HDD (30 drives total) in a 4u / 44 bay Supermicro JBOD.
• Each HDD is 14TB Seagate Enterprise drives (Exos X18 7200RPM) SAS 12Gbps ST14000NM004J with native 4K sector
• TrueNAS Core
• NFS v3 for exposing dataset to VMWare cluster

Head Node is

• 2u Supermicro
• 256GB DDR4 ECC Memory
• 32 Core AMD CPU (for the PCI lanes)
• 12 bays for SAS/NVMe
• 6x PCIe Gen 4x 16 slots

 

[awasb]

I'm not quite sure, I understand you correctly. If you set sync=none (what I wouldn't recommend for VMware data store via NFS) you don't need any slog. If it's just the performance level you are aiming at, while in fact using sync=auto or sync=all everything is fine.

Concerning latency I'd prefer Optanes anytime. (Though I'd choose some of those P1600X 118GB mirrored.They yield almost identical performance for a fraction of the price.)

I don't expect striping to give you much of a gain. It's more of a "load balancing" feature. A mirrored pair will only guarantee that the SLOG is available even if one of the two Optanes dies. But it's easy to test all this, since you can add and remove SLOG vdev any time without damaging pool data.

 

[HoneyBadger]

Hello @helloadam

I edited your post to remove the color formatting, as it made the post unreadable on the dark theme. Hope you don't mind.

In brief, I would expect the Optane P4800X to be much, much faster than the Kioxia CM5 in sync writes. Neither will be as fast as sync=disabled as approaching those speeds requires things like NVDIMM or other battery-backed RAM technologies.

A striped-mirror of four SLOGs is possible, but won't necessarily be a linear doubling of performance versus two drives.

See the thread in my signature regarding community benchmarks of various SLOG devices, nothing that for VMware you'll generally want to look at the results for lower recordsizes to represent the speed for running/active VMs.

 

[Davvo]

Looking at the specs, the optane also seems to have a greater endurance.

 

[helloadam]

I'm not quite sure, I understand you correctly. If you set sync=none (what I wouldn't recommend for VMware data store via NFS) you don't need any slog. If it's just the performance level you are aiming at, while in fact using sync=auto or sync=all everything is fine.

Concerning latency I'd prefer Optanes anytime. (Though I'd choose some of those P1600X 118GB mirrored.They yield almost identical performance for a fraction of the price.)

I don't expect striping to give you much of a gain. It's more of a "load balancing" feature. A mirrored pair will only guarantee that the SLOG is available even if one of the two Optanes dies. But it's easy to test all this, since you can add and remove SLOG vdev any time without damaging pool data.

Sorry if I was not clear. We are going to run sync=always with our NFS dataset. But we want a SLOG that can achieve performance as if we disabled sync so the write was acknowledged as soon its in system memory.

Looking at the specs, the optane also seems to have a greater endurance.

Yeah, between CM5 and Optane we are leaning towards Optane.

Hello @helloadam

I edited your post to remove the color formatting, as it made the post unreadable on the dark theme. Hope you don't mind.

In brief, I would expect the Optane P4800X to be much, much faster than the Kioxia CM5 in sync writes. Neither will be as fast as sync=disabled as approaching those speeds requires things like NVDIMM or other battery-backed RAM technologies.

A striped-mirror of four SLOGs is possible, but won't necessarily be a linear doubling of performance versus two drives.

See the thread in my signature regarding community benchmarks of various SLOG devices, nothing that for VMware you'll generally want to look at the results for lower recordsizes to represent the speed for running/active VMs.

Do not mind the edit, I posted this on mobile and must have messed up the formatting.

What recommendations do you have for NVDIMM or battery-backed RAM?

RMS-200/300?

• The RMS-200 looks promising with unlimited write endurance but at the low record sizes (4-16KB) is slow.
• RMS-300 looks better but not game changer
• Maybe my concerns are not valid given our setup? 10vdevs in 3-way mirror of HDD?

NVDIMM?

• I have to admit, I am not familiar with NVDIMM technology. We are using the Supermicro 2014CS-TR as our head node and I am not 100% sure if our motherboard supports NVDIMM based on public posting
• Our CPU is AMD EPYC 7543P (7003 / Malian based)
• https://www.supermicro.com/en/aplus/system/2u/2014/as-2014cs-tr.cfm

Diving our motherboard manual (https://www.supermicro.com/en/products/motherboard/H12SSW-AN6) we see one line in the 97 page PDF that says

So maybe our motherboard supports NVDIMM? I am not sure where the super capacitor/battery will be installed. Or if this is just a typo in our manual.

 

[HoneyBadger]

Do not mind the edit, I posted this on mobile and must have messed up the formatting.

What recommendations do you have for NVDIMM or battery-backed RAM?

RMS-200/300?

• The RMS-200 looks promising with unlimited write endurance but at the low record sizes (4-16KB) is slow.
• RMS-300 looks better but not game changer
• Maybe my concerns are not valid given our setup? 10vdevs in 3-way mirror of HDD?

NVDIMM?

• I have to admit, I am not familiar with NVDIMM technology. We are using the Supermicro 2014CS-TR as our head node and I am not 100% sure if our motherboard supports NVDIMM based on public posting
• Our CPU is AMD EPYC 7543P (7003 / Malian based)
• https://www.supermicro.com/en/aplus/system/2u/2014/as-2014cs-tr.cfm

Diving our motherboard manual (https://www.supermicro.com/en/products/motherboard/H12SSW-AN6) we see one line in the 97 page PDF that says

View attachment 68796

So maybe our motherboard supports NVDIMM? I am not sure where the super capacitor/battery will be installed. Or if this is just a typo in our manual.

Given that a one-liner in the FAQ section is the only reference to NVDIMMs, I have my doubts that your board supports them.

The RMS-200 and RMS-300 are options for unlimited endurance but they are slower than the Optane P4800X at small record sizes. They do come in at a lower-cost in the second-hand market though vs. the Optane.