Hi Everyone,
Summary: on ESXI host itself (not a vm) read performance over NFSv3 from Freenas, when reading a file completely from ARC, is very slow (13-17MB/s) over 1GbE link.
Hoping to pick the brains of some experts on this one. I'm relatively new to the forums, but not to FreeNAS. Basically, new build in testing, FreeNAS 11 RC3 on baremetal Xeon E1230v6, 32GB. NFS share from SSD mirror (samsung 850 EVO) with Intel S3520 SLOG, and mounted on both ESXI 6.5 and 5.5 (ie, tried this from 2 different servers).
Read performance of a file off this share is very slow at 13-17MB/s. The file is in ARC, there is no disk activity when reading the file and dumping it to /dev/null. Same test from a vm client on the same esxi boxes gives close to line speed at 111MB/s (ie, NFS mounting within the VM and dumping to /dev/null, so no ESXI NFS mounted datastore involved, but exactly the same network/NIC/path etc). ESXI 5.5 performance slightly worse. Nothing moves much past idle on either ESXI host, or the freenas server in terms of CPU/mem etc. Boxes are not doing much of anything else.
I could move to ISCSI (have not tested performance in comparison yet), but was hoping NFS would perform better than this, at least for reads. Has me a bit baffled. This is not a sync/async issue. I've not done definitive test writes. They are slow as expected with sync, but I suspect they are around 30-60MB/s (which was maybe double what they were with no SLOG) - so they may even be faster than reads. I'll test write performance if I can solve the read issues!
Why would reading a file over NFS be slow, especially when the file is cached in freenas memory? I've not done any tuning and the NFS server on freens is pretty much out of the box, esxi hosts just mounted with defaults and NFSv3. ESXI is accessing the NFS server over dedicated NIC (in failover with another) with only storage network traffic on it on its own VLAN. Just one switch connecting boxes, so low latency. this VLAN is on dedicated NIC on freenas also. Clearly the problem is with ESXI and how its NFS client implementation interacts with freenas.... just not sure if this is solvable. I've googled and found similar problems, although complaints seems to be centered around performance closer to the 60-70MB/s mark.
Any ideas or advice appreciated. Ideas for tuning or further testing to narrow down problem welcome, although if someone else has done the hard yards and advises that FreeNAS NFS server and ESXI NFS client don't go well together for some reason, I'll consider moving on to ISCSI.
Thanks for your time reading.
Regards
Dave
Summary: on ESXI host itself (not a vm) read performance over NFSv3 from Freenas, when reading a file completely from ARC, is very slow (13-17MB/s) over 1GbE link.
Hoping to pick the brains of some experts on this one. I'm relatively new to the forums, but not to FreeNAS. Basically, new build in testing, FreeNAS 11 RC3 on baremetal Xeon E1230v6, 32GB. NFS share from SSD mirror (samsung 850 EVO) with Intel S3520 SLOG, and mounted on both ESXI 6.5 and 5.5 (ie, tried this from 2 different servers).
Read performance of a file off this share is very slow at 13-17MB/s. The file is in ARC, there is no disk activity when reading the file and dumping it to /dev/null. Same test from a vm client on the same esxi boxes gives close to line speed at 111MB/s (ie, NFS mounting within the VM and dumping to /dev/null, so no ESXI NFS mounted datastore involved, but exactly the same network/NIC/path etc). ESXI 5.5 performance slightly worse. Nothing moves much past idle on either ESXI host, or the freenas server in terms of CPU/mem etc. Boxes are not doing much of anything else.
I could move to ISCSI (have not tested performance in comparison yet), but was hoping NFS would perform better than this, at least for reads. Has me a bit baffled. This is not a sync/async issue. I've not done definitive test writes. They are slow as expected with sync, but I suspect they are around 30-60MB/s (which was maybe double what they were with no SLOG) - so they may even be faster than reads. I'll test write performance if I can solve the read issues!
Why would reading a file over NFS be slow, especially when the file is cached in freenas memory? I've not done any tuning and the NFS server on freens is pretty much out of the box, esxi hosts just mounted with defaults and NFSv3. ESXI is accessing the NFS server over dedicated NIC (in failover with another) with only storage network traffic on it on its own VLAN. Just one switch connecting boxes, so low latency. this VLAN is on dedicated NIC on freenas also. Clearly the problem is with ESXI and how its NFS client implementation interacts with freenas.... just not sure if this is solvable. I've googled and found similar problems, although complaints seems to be centered around performance closer to the 60-70MB/s mark.
Any ideas or advice appreciated. Ideas for tuning or further testing to narrow down problem welcome, although if someone else has done the hard yards and advises that FreeNAS NFS server and ESXI NFS client don't go well together for some reason, I'll consider moving on to ISCSI.
Thanks for your time reading.
Regards
Dave