jgreco
Resident Grinch
- Joined
- May 29, 2011
- Messages
- 18,680
Posted in response to a staff request, this is intended to help answer the "certificate is expired" issues.
SSL underpins most network session security on the Internet. Historically, this was implemented with self-signed certificates (a bad idea), or through commercial certificate authorities (a pricey proposition). In a bid to see the Internet default to securing everything (which is a bad idea of a different sort), several industry players cobbled together a free, automatic certificate authority called LetsEncrypt, and released software to make it easy to get valid SSL certificates for your website (generally a good idea).
To get this bootstrapped, they initially signed this with the IdenTrust DST Root X3 certificate, which expired at the end of September 2021. LetsEncrypt has experienced wild success, and is now signing with their own ISRG Root X1 certificate, now trusted in most browsers and SSL trust stores. I'm not making a full accounting or description of the bootstrap and cross-signing processes used to bring this online. You can Google all that.
However, OpenSSL 1.0.2 has a "bug" that causes it to prefer the expired certificate. Without going into a lot of detail and contentious debate over how this kind of thing should work, the fact of the matter is that you're probably reading this because you've encountered it.
FreeBSD (and therefore FreeNAS) 11 and before all use OpenSSL 1.0.2, and FreeNAS uses the Mozilla NSS port (ports/ca_root_nss) to define its default trust list. This list includes the expired DST Root X3 certificate, so after September 30th, FreeNAS clients attempting to connect to a LetsEncrypt-based web site are likely to hit the "certificate is expired" error.
This does not mean the remote server's certificate. It is, instead, an expired certificate in your own trust store.
To resolve it, you need to locate and eliminate the old DST Root X3 certificate. For the base FreeBSD/FreeNAS system, this is the Netscape NSS list of CA's, which is located in /usr/local/share/certs/ca-root-nss.crt as a single file. Jails and other applications may have their own trust stores, and may need custom adjustment of various types.
You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.
Note that the decoded (human-readable) text version of the certificate comes BEFORE the encoded (machine-readable) certificate, so you would want to make a backup of the file, then try deleting the text version starting at
where you will eventually see a BEGIN CERTIFICATE and a bunch of ASCII encoded binary garbage. Delete all of this, all the way through the following
The total deleted section should be 81 lines.
The ca-root-nss on older FreeNAS systems is probably a bit out of date, and the file that we're using for our installs here is nss-3.71, which has been manually edited to remove the "bad" DST certificate. You can get a copy at
if you wish. I do not recommend doing this. SSL is the foundation of trust on the Internet, and you SHOULD NOT blindly accept my assurances that this is nss-3.71 with a single certificate deleted. You are better off acquiring nss-3.71 yourself and removing the certificate in question yourself. Because this is certainly newer than any shipped FreeNAS ca-root-nss file, it will have a number of differences, and this represents some risk in that you have effectively allowed some rando Internet crank define your SSL trust. On the other hand, I'm not actually a rando Internet crank, I do infrastructure engineering for NTP.ORG and if you've got your NTP servers set to pool.ntp.org, then you are already relying on services I've engineered. ;-)
I am absolutely an Internet crank, just not a rando one.
Note that you would need to move the downloaded file into place once you've inspected and approved of it:
You are strongly encouraged to instead download a fresh copy of FreeBSD, update your ports, install the security/ca_root_nss port, and then copy that file to your FreeNAS after editing out the DST Root X3 certificate.
This hasn't solved your problem? Need to update some other trust store? There's already some hints in the thread over here, or feel free to ask below. I fully understand how frustrating this is. As someone who is experienced with this stuff, it's exasperating to me too. Let me know how and where this topic needs to be improved. I write these things for you, and the community exists to help each other through these challenging bits.
SSL underpins most network session security on the Internet. Historically, this was implemented with self-signed certificates (a bad idea), or through commercial certificate authorities (a pricey proposition). In a bid to see the Internet default to securing everything (which is a bad idea of a different sort), several industry players cobbled together a free, automatic certificate authority called LetsEncrypt, and released software to make it easy to get valid SSL certificates for your website (generally a good idea).
To get this bootstrapped, they initially signed this with the IdenTrust DST Root X3 certificate, which expired at the end of September 2021. LetsEncrypt has experienced wild success, and is now signing with their own ISRG Root X1 certificate, now trusted in most browsers and SSL trust stores. I'm not making a full accounting or description of the bootstrap and cross-signing processes used to bring this online. You can Google all that.
However, OpenSSL 1.0.2 has a "bug" that causes it to prefer the expired certificate. Without going into a lot of detail and contentious debate over how this kind of thing should work, the fact of the matter is that you're probably reading this because you've encountered it.
FreeBSD (and therefore FreeNAS) 11 and before all use OpenSSL 1.0.2, and FreeNAS uses the Mozilla NSS port (ports/ca_root_nss) to define its default trust list. This list includes the expired DST Root X3 certificate, so after September 30th, FreeNAS clients attempting to connect to a LetsEncrypt-based web site are likely to hit the "certificate is expired" error.
This does not mean the remote server's certificate. It is, instead, an expired certificate in your own trust store.
To resolve it, you need to locate and eliminate the old DST Root X3 certificate. For the base FreeBSD/FreeNAS system, this is the Netscape NSS list of CA's, which is located in /usr/local/share/certs/ca-root-nss.crt as a single file. Jails and other applications may have their own trust stores, and may need custom adjustment of various types.
You may remove the offending certificate from /usr/local/share/certs/ca-root-nss.crt with an editor.
Note that the decoded (human-readable) text version of the certificate comes BEFORE the encoded (machine-readable) certificate, so you would want to make a backup of the file, then try deleting the text version starting at
Code:
Certificate: Data: Version: 3 (0x2) Serial Number: 44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b Signature Algorithm: sha1WithRSAEncryption Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Validity Not Before: Sep 30 21:12:19 2000 GMT Not After : Sep 30 14:01:15 2021 GMT
Code:
-----END CERTIFICATE-----
The total deleted section should be 81 lines.
The ca-root-nss on older FreeNAS systems is probably a bit out of date, and the file that we're using for our installs here is nss-3.71, which has been manually edited to remove the "bad" DST certificate. You can get a copy at
Code:
# fetch --no-verify-peer https://extranet.www.sol.net/files/misc/ca-root-nss.crt.src
if you wish. I do not recommend doing this. SSL is the foundation of trust on the Internet, and you SHOULD NOT blindly accept my assurances that this is nss-3.71 with a single certificate deleted. You are better off acquiring nss-3.71 yourself and removing the certificate in question yourself. Because this is certainly newer than any shipped FreeNAS ca-root-nss file, it will have a number of differences, and this represents some risk in that you have effectively allowed some rando Internet crank define your SSL trust. On the other hand, I'm not actually a rando Internet crank, I do infrastructure engineering for NTP.ORG and if you've got your NTP servers set to pool.ntp.org, then you are already relying on services I've engineered. ;-)
I am absolutely an Internet crank, just not a rando one.
Note that you would need to move the downloaded file into place once you've inspected and approved of it:
Code:
# cp /usr/local/share/certs/ca-root-nss.crt /usr/local/share/certs/ca-root-nss.crt.old # cp ca-root-nss.crt.src /usr/local/share/certs/ca-root-nss.crt
You are strongly encouraged to instead download a fresh copy of FreeBSD, update your ports, install the security/ca_root_nss port, and then copy that file to your FreeNAS after editing out the DST Root X3 certificate.
This hasn't solved your problem? Need to update some other trust store? There's already some hints in the thread over here, or feel free to ask below. I fully understand how frustrating this is. As someone who is experienced with this stuff, it's exasperating to me too. Let me know how and where this topic needs to be improved. I write these things for you, and the community exists to help each other through these challenging bits.