jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are.
1) An HBA is a Host Bus Adapter.
This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort, and even in "JBOD" or "HBA mode" generally hide the physical device. If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA. If you cannot get the output of "smartctl" for a device, you DO NOT HAVE A TRUE HBA. A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.
2) FreeBSD has incredibly robust support for the LSI HBA's.
FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The FreeNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.
3) You must crossflash to IT/IR firmware
If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT-20.00.07.00 HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this is probably no longer relevant.
The LSI 9240 (etc) default MFI firmware is apparently being sold on eBay as "IR" by clueless sellers. The MFI firmware is unsuitable for FreeNAS and may cause your pool to get eaten.
The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware 20.00.07.00.
The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware ... 13? I don't have one of these online right now so my appreciation to whoever follows up with the current correct answer.
4) FreeBSD may or may not have good support for other HBA's/RAID controllers.
FreeBSD does support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.
It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.
If you have come to FreeNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.
5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.
In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.
6) A RAID controller with write cache is particularly bad.
A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.
For more information on related topics, please see:
Confused about that LSI card? Join the crowd...
Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA
1) An HBA is a Host Bus Adapter.
This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort, and even in "JBOD" or "HBA mode" generally hide the physical device. If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA. If you cannot get the output of "smartctl" for a device, you DO NOT HAVE A TRUE HBA. A true HBA passes communications through itself directly to a drive without further processing. No amount of marketing department wishful thinking can change that technical reality.
2) FreeBSD has incredibly robust support for the LSI HBA's.
FreeBSD's LSI HBA (mps/mpr) drivers are authored by LSI and carefully designed to work with their HBA firmware. The FreeNAS userbase has installed many thousands of these cards which have, in aggregate, BILLIONS of problem-free run-hours. Not only are they known to work very well during normal operations, but they're also known to work correctly during ABNORMAL operations, such as when a disk times out or throws an error. SMART is properly supported. Forum members are incredibly familiar with all the variations on these and can provide useful assistance. Cards such as the LSI 9240-8i, IBM ServeRAID M1015, Dell PERC H200 and H310, and others are readily available on the used market and can be converted to LSI 9211-8i equivalents.
3) You must crossflash to IT/IR firmware
If you don't crossflash, then a lot of the remainder of this ALSO applies to LSI non-IT-20.00.07.00 HBA's!! The IR firmware is also fine but is a few percent slower. It is not clear there is any value to doing this as you would never want to use an IR virtual device with FreeNAS. We used to do this in the old days for boot devices, but with ZFS boot this is probably no longer relevant.
The LSI 9240 (etc) default MFI firmware is apparently being sold on eBay as "IR" by clueless sellers. The MFI firmware is unsuitable for FreeNAS and may cause your pool to get eaten.
The LSI 9211-8i (PCIe 2.0 based on LSI 6Gbps SAS2008) and LSI 9207-8i (PCIe 3.0 based on LSI 6Gbps SAS2308) both require firmware 20.00.07.00.
The LSI 9300-8i (PCIe 3.0 based on LSI 12Gbps SAS3000) requires firmware ... 13? I don't have one of these online right now so my appreciation to whoever follows up with the current correct answer.
4) FreeBSD may or may not have good support for other HBA's/RAID controllers.
FreeBSD does support a wide range of HBA's and RAID cards, but in many cases the drivers have been reverse-engineered, or may cause various problems when there is an underlying issue with a component disk. ZFS has the capability to place immense I/O workloads on your HBA, and you don't really want the controller chip crashing and taking a bunch of your disks offline with it (known to happen). ZFS expects that its ability to talk with correctly functioning HDD's is 100.000% and bad things start to happen when controllers or drivers freak out.
It might seem very tempting to use some alternative brand of HBA or RAID controller that you happen to "have handy", but the problem is that you will be on your own private little adventure. Things might seem to work fine, until one day something bad happens. And then you might be outta luck. Some RAID cards do things like encapsulating their JBOD VD's in a partition, making it effectively impossible to plug the drives into a SATA port or LSI HBA.
If you have come to FreeNAS with the intention of making your NAS into a guinea pig for testing of an unknown and untested controller, then, by all means, go ahead. Just please be aware that the measure of success isn't "I got it to make a pool." It is possible for things to work for weeks, months, even years before something adverse happens. We promote the LSI controllers because they are proven to work, with an aggregate of billions of run-hours. You will not be able to duplicate that sort of rigorous testing.
5) A RAID controller that supports "JBOD" or "HBA mode" isn't the same.
In these devices, you are relying on the RAID card driver to communicate from the host to the controller. As previously noted, the LSI HBA drivers have billions of proven run-hours, but in many cases, RAID drivers aren't as solid. Some of FreeBSD's RAID drivers have been tweaked to cope better with device error handling on the theory that you have redundancy (JBOD isn't), many do not allow you to poll the drive's SMART status, and in many cases you can inadvertently set up bad situations with write caching, etc. ZFS has the ability to generate immense amounts of I/O traffic that can be a crushing workload for the weedy little CPU's on a RAID controller, can totally flood the cache on a RAID card, etc. As mentioned in the previous section, many RAID cards also do things such as encapsulation of JBOD within a partition, which effectively locks you into having to use that RAID card. This is super-bad for error recovery. With SATA ports or LSI HBA ports, SATA drives are completely interchangeable.
6) A RAID controller with write cache is particularly bad.
A RAID controller with a write cache is likely to get swamped by the massive I/O ZFS is pushing. These devices are typically sized to cope with the sorts of I/O a standard server would push around, update a file here, read an executable from there, do some database updates... but ZFS will perform operations such as scrubs and resilvers that will maintain massive I/O pressure for hours or days. Even the normal I/O is demanding, as a ZFS transaction group can easily be a gigabyte, every five seconds. Hiding the actual performance of devices behind a tiny RAID card cache is not a good idea as it leads to less-predictable performance.
For more information on related topics, please see:
Confused about that LSI card? Join the crowd...
Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA
Last edited: