i have setup a new system with freenas 9.10 and currently i`m testing a little bit
i have 2 pools raidz2 9+2 (9 disks + 2 hotspares)
zil/slog is mirrored across enterprise ssd`s, cache/l2arc is striped across the same disks (as i do not want to add extra ssd i partitioned the disks appropriately)
i came across two issues:
1.
- when pulling one of the ssd`s, zil mirror goes to degrade state as expected and cache device goes unavail. as expected
- when re-inserting the disk zill mirror automatically resilvers - but cache device remain unavail.
- i cannot bring back the cache device online. i`m getting:
[root@fnask-bnkw] ~# zpool online zfspool_sas_hitachi /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150
cannot online /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150: cannot expand cache devices
[root@fnask-bnkw] ~# glabel status|grep ed8c
gptid/ed8c9680-ee27-11e6-a3b4-00259004b150 N/A da21p3
- what do do here ?
2.
- when pulling an unused hotspare disk, it`s not being correctly detected as missing nor is it being correctly detected as
available by zfs when it was offline and disk being re-inserted.
- i cannot bring it back online manually to zfs again. any clues, why ?
[root@fnask-bnkw] ~# zpool online zfspool_sas_seagate /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
cannot online /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150: device is reserved as a hot spare
[root@fnask-bnkw] ~# glabel status | grep 9b9
gptid/9b95873a-ee2b-11e6-a3b4-00259004b150 N/A multipath/disk24p2
[root@fnask-bnkw] ~# zpool status|grep 9b95
1682681741557551889 UNAVAIL was /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
regards
roland
i have 2 pools raidz2 9+2 (9 disks + 2 hotspares)
zil/slog is mirrored across enterprise ssd`s, cache/l2arc is striped across the same disks (as i do not want to add extra ssd i partitioned the disks appropriately)
i came across two issues:
1.
- when pulling one of the ssd`s, zil mirror goes to degrade state as expected and cache device goes unavail. as expected
- when re-inserting the disk zill mirror automatically resilvers - but cache device remain unavail.
- i cannot bring back the cache device online. i`m getting:
[root@fnask-bnkw] ~# zpool online zfspool_sas_hitachi /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150
cannot online /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150: cannot expand cache devices
[root@fnask-bnkw] ~# glabel status|grep ed8c
gptid/ed8c9680-ee27-11e6-a3b4-00259004b150 N/A da21p3
- what do do here ?
2.
- when pulling an unused hotspare disk, it`s not being correctly detected as missing nor is it being correctly detected as
available by zfs when it was offline and disk being re-inserted.
- i cannot bring it back online manually to zfs again. any clues, why ?
[root@fnask-bnkw] ~# zpool online zfspool_sas_seagate /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
cannot online /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150: device is reserved as a hot spare
[root@fnask-bnkw] ~# glabel status | grep 9b9
gptid/9b95873a-ee2b-11e6-a3b4-00259004b150 N/A multipath/disk24p2
[root@fnask-bnkw] ~# zpool status|grep 9b95
1682681741557551889 UNAVAIL was /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150
regards
roland
Code:
pool: zfspool_sas_hitachi state: ONLINE status: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0 in 0h0m with 0 errors on Thu Feb 9 04:31:13 2017 config: NAME STATE READ WRITE CKSUM zfspool_sas_hitachi ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/c907b699-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/c98a0ca1-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/ca0ba737-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/ca8ee926-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/cb0fc998-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/cb9c7ef9-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/cc310971-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/ccafe07d-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/cd33f6db-ee29-11e6-a3b4-00259004b150 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 gptid/713f0662-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/78d3e971-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 cache gptid/e9d3a97b-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 242793518834828308 REMOVED 0 0 0 was /dev/gptid/ed8c9680-ee27-11e6-a3b4-00259004b150 spares gptid/841cb167-ee2a-11e6-a3b4-00259004b150 AVAIL gptid/849e25a9-ee2a-11e6-a3b4-00259004b150 AVAIL errors: No known data errors pool: zfspool_sas_seagate state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Thu Feb 9 04:55:42 2017 config: NAME STATE READ WRITE CKSUM zfspool_sas_seagate ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/6f6df9f4-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/701396d5-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/7102205b-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/71ac1e3e-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/7254274d-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/72fd6fee-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/73a6dc80-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/744ee081-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/74f7524d-ee2b-11e6-a3b4-00259004b150 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 gptid/98bdd972-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/9d75100f-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 cache gptid/f4becdc9-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 gptid/f94b4bc7-ee27-11e6-a3b4-00259004b150 ONLINE 0 0 0 spares 1682681741557551889 UNAVAIL was /dev/gptid/9b95873a-ee2b-11e6-a3b4-00259004b150 gptid/9aef2b1b-ee2b-11e6-a3b4-00259004b150 AVAIL errors: No known data errors
Last edited: