SSH access does not work .... apart for root :( :) !

[Louis2]

I own a TrueNas Core system and try to access the system via SSH and that does not work .... apart for the root user .....

I have spend many hours trying to find out why

I principle I did this:
- generate a key pairs using Bitvise SSH Client
- save them in open ssh format as <username>.pub and <username>.key
- added / copied the <username>.pub into the 'Users / Accounts / Autentication' field
- saving the <username>.key in <home>/<username>/.ssh/
- chmod 600 <home>/<username>/.ssh/<username>.key
Where copying <username>.key had to be done by root via ssh (a small crime)

It does not work .....
- the users can not login using ssh
- rsync does not work under user accounts

Long ago I decided to create the user home directory's on an NVME-SSD and the situation is like this:
- the home for root is of course /root
- the home for users is /mnt/NVME-drive/USERS/<username>
In principle the standard for users is /home/<username> ...... where /home does not exist on core .....


As a result of this weirded behavoir and a lots of headache and lost hours, I do have some verdict that it has to with the path to the user home directory's
Some feeling !! that the ssh deamon can not access the user directory's ...

Does any one has clue's / solutions !?


Note that I did

 

[danb35]

- saving the <username>.key in <home>/<username>/.ssh/

Why? There's no reason for the private key to be on the remote system, and a strong reason for it not to be.

- chmod 600 <home>/<username>/.ssh/<username>.key

Was this done as <username>, and did it succeed? Because one common reason for issues here is that your home directory is using ACLs rather than standard Unix permissions. And the bigger issue would be the permissions for authorized_keys, which should be handled by the middleware, but once again can be trouble if the dataset in question isn't using normal Unix permissions.

 

[Louis2]

I have been looking every where for documentation how to setup SSH. But it must be my stupidity, but I could not find any really not any !!

It is completely clear that the pubic key has to be added in the user definition screen, its just silly that you have to copy that with cut and past ... but ala that works

However where to put the private key, is completely misty ....
On the internet here and there some remarks that
- it should be in the user directory
- probably in subfolder .ssh (that turns out to work for scale and core root)
- and that the key file should be named ir_rsa or something like that (turned out to be nonsense, can be <name>.key)

And from the security point of view I do understand that the private.key should be controlled by the user and not by root, and should be in the user dir ...

However you have a point saying that the private key should (mostly) not be on a remote system. The reason I where this drama started ....rync between my new TrueNas scale NAS and the other NAS my core based NAS. And scale is in this case the client from core and as such need a private key ... I suppose. Perhaps I should create a separate user for rsync, that apart.

And ssh access should of course work for all users

 

[danb35]

its just silly that you have to copy that with cut and past

Why is that silly?

However where to put the private key, is completely misty ....

The private key should be, well, private. It should ordinarily never leave the machine on which it's created. Where it should be stored on that machine depends on the SSH client in use there. If you're using normal CLI SSH (as in *nix, macOS, and Windows since Win10), it goes in the user's ~/.ssh directory by default. If you're using (as you are) a third-party SSH client, you'd need to consult the documentation for that client.

The reason I where this drama started ....rync between my new TrueNas scale NAS and the other NAS my core based NAS.

Ah, then yes, in that case, you'd need a private key on whichever of those systems is the client. The way to do that is from the shell on that system as the user in question, where you'd run ssh-keygen. The public key that's generated there goes into the authorized_keys file (or pasted into the appropriate field of the user edit screen in the GUI) of the corresponding user on the other system.

 

[Samuel Tai]

A quick & dirty tutorial for SSH access.

1. Create a user in the UI. Make sure this user has a valid home directory and shell.
2. Copy the user's SSH public key into the SSH Public Key field.

On the system from which you SSH to TrueNAS.

1. Create an .ssh directory with permissions 700 in your home directory
2. Copy the TrueNAS user's private SSH key file to .ssh, and give it permissions 600.
3. Within the .ssh directory, create a text file named config with permissions 600. Add these lines to config to set up your SSH access:
   
   Code:

   Host <short label for TrueNAS server>
     Hostname <FQDN of TrueNAS server>
     User <TrueNAS user you use for SSH>
     IdentityFile ~/.ssh/<name of private key file>

You should now be able to SSH to TrueNAS as the configured user via ssh <short label>.

Some rules of thumb:

• The private key always goes on the SSH originator. It should never be put on the SSH target.
• The public key always goes on the SSH target.
• You can generate the SSH keys anywhere, so long as they get moved to the correct locations
• Note the permissions above. SSH won't work with other permissions.

 

[danb35]

Copy the TrueNAS user's private SSH key to .ssh, and give it permissions 600.

Why on earth would you do that? Leaving aside that there won't be such a file unless you create it, why would you want to copy it?

 

[Samuel Tai]

Why on earth would you do that? Leaving aside that there won't be such a file unless you create it, why would you want to copy it?

Not on the TrueNAS server, on the host from which you originate the SSH session.

 

[Patrick M. Hausen]

@Samuel Tai To address someone inexperienced you forgot to mention that .ssh is supposed to be a directory and the key is a file named id_rsa or id_ed25519. And this file should have permissions 600.

 

[Louis2]

I solved the not working SSH-login ...... but not yet the rsync problem

The users accounts with the not working SSH-login, where related to my actual older nas upgraded many times.
That problem was gone after deleting those accounts and recreate them .......
Apart from the fact that that did solve part of the problem, it did make the situation less weird

Never the less, very strange, rsync still does not work. I try to pull data from the old nas (which should stay operational by the way), towards the new NAS.

Among other things I did create a RsyncTask on the new Nas (scale) using SSH. And when I save that definition a lot seems ok since the path verification on the remote machine (the old nas) does work. If I change the path to something non existing ... there is a error and also the message that there was a login.

However, if I start the RsyncTask ..... it starts and directly stops .....

Since there are no errors given, I tried to fetch errors by opening cmd shells on both computers and starting 'tail -f /var/log/messages'
But no clue at all

By the way, a proper message/alarm widget on the dashboard would be highly appreciated .....

I did not yet dig into Samuel his suggestions, I will read those in detail later on

 

[Louis2]

I have put a lot of investigation in this problem and I tend more an more to think that there is a bug.

- I did deep down testing using tcpdump
tcpdump -nvX -e 'vlan 18' and src net 192.168.18.0/24 and dst net 192.168.18.0/24 (the communication between the two TrueNas systems)
=> if I define the job, the remote path is tested, that works and I see that with tcpdump
- If I start the related Rsysnc tast => it starts and directly after fails
=> and with the tcpdump ..... I see ..... nothing !!
=> and if download the log from the 'failed button' the log is empty (very help full )

So do not know what to investigate further, but IMHO something is not OK

 

[Louis2]

A quick & dirty tutorial for SSH access.

1. Create a user in the UI. Make sure this user has a valid home directory and shell.
2. Copy the user's SSH public key into the SSH Public Key field.

On the system from which you SSH to TrueNAS.

1. Create an .ssh directory with permissions 700 in your home directory
2. Copy the TrueNAS user's private SSH key file to .ssh, and give it permissions 600.
3. Within the .ssh directory, create a text file named config with permissions 600. Add these lines to config to set up your SSH access:
   
   Code:

   Host <short label for TrueNAS server>
     Hostname <FQDN of TrueNAS server>
     User <TrueNAS user you use for SSH>
     IdentityFile ~/.ssh/<name of private key file>

You should now be able to SSH to TrueNAS as the configured user via ssh <short label>.

Some rules of thumb:

• The private key always goes on the SSH originator. It should never be put on the SSH target.
• The public key always goes on the SSH target.
• You can generate the SSH keys anywhere, so long as they get moved to the correct locations
• Note the permissions above. SSH won't work with other permissions.

@Samuel Tai

Samual, I am becoming nuts:
- My TrueNas core system is the source it has a defined rsync module, the test directory is open for every one
- My TrueNas scale system is the destination and has also a defined rsync module
- the test destination folder is again open for every body
- both systems have/know the same user
- I am using SSH
- the pk of that user is in the home folder of that user on the scale system
- the scale system rsync definition is capable of verifying the source path on the remote system, so the communication is OK is it not !!
- I can monitor that check using tcpdump
- then I start the rsync task:
* it starts
* it stops
* there is no data communication at all, I can see from tcpdump

The only thing I did not do is adding the config code under "3", I do not understand at all

My user home folders are on /mnt/<somessd>/home/<user>, where they should be conform the standard in /home/<user>
However:
- root => /root
- admin => /home/admin !!??
- other users can not be added to /home/<user> they are in my case in /mnt/somedataset/home/<user>
Might that be a clue ???

 

[danb35]

where they should be conform the standard

On what basis do you say they should do this?

 

[WI_Hedgehog]

I have put a lot of investigation in this problem and I tend more an more to think that there is a bug.

In my experience with many different operating systems, if root (or an admin account) "works" and a user-level account doesn't it's probably an overlooked security issue. Sometimes turning off as much security as possible and creating a secondary administration account with elevated privileges allows things to "work," then removing privileges/using a non-admin account along with re-enabling security slowly helps identify the issue. (remember some security settings are "remembered" and need to be flushed, depending--Windows likes to do this)

 

[Louis2]

On what basis do you say they should do this?

I think that is at least the proposed structure

04-E.12.1: Linux Directory Structure - Hierarchy

eng.libretexts.org

However note that given the ridiculous fact that the bootdrive has to be a separate drive (big waist !), I prefer to put the user directory's on a nvme based dataset (and not on a usb-stick or so). But .... given all the problems I am facing and the fact that the admin account is at the proposed position, I thought .... this might be a clue ...

 

[WI_Hedgehog]

However note that given the ridiculous fact that the bootdrive has to be a separate drive (big waist !), I prefer to put the user directory's on a nvme based dataset (and not on a usb-stick or so). But .... given all the problems I am facing and the fact that the admin account is at the proposed position, I thought .... this might be a clue ...

A datacenter rackmount system (which is what TrueNAS is designed to be) generally keeps the OS information separate from the data.

Imagine a 4U rack-mount server case holding 96 data multi-terabyte drives spinning away, with a small 32 GB SSD boot drive. It's Network Attached Storage, not a Virtual Machine Server, or Internet Gateway, or Workstation, it's one of 12 industrial high-capacity high-availability storage servers.

I realize LTT and Serve The Home along with other 'Tubers talk about setting up Home Servers using TrueNAS, and of course many such systems exist. The thing is it is actually more involved than a Windows Desktop share, which I'm sure you realized beforehand, though how involved may be been a bit of a gray area.

We're here to help you and other succeed with TrueNAS, it really is a well-thought product.

 

[Louis2]

A datacenter rackmount system (which is what TrueNAS is designed to be) generally keeps the OS information separate from the data.

Imagine a 4U rack-mount server case holding 96 data multi-terabyte drives spinning away, with a small 32 GB SSD boot drive. It's Network Attached Storage, not a Virtual Machine Server, or Internet Gateway, or Workstation, it's one of 12 industrial high-capacity high-availability storage servers.

I realize LTT and Serve The Home along with other 'Tubers talk about setting up Home Servers using TrueNAS, and of course many such systems exist. The thing is it is actually more involved than a Windows Desktop share, which I'm sure you realized beforehand, though how involved may be been a bit of a gray area.

We're here to help you and other succeed with TrueNAS, it really is a well-thought product.

I understand that for large professional NAS-systems it is not an issue. But for a home or small office system it is. So one way I understand, but the other way why o why (!) is it such a big deal to have the option to have the boot partition on a data drive !???

But that is another discussion, not the problem discussed here.

PS. I opened another thread about that long ago, which some recent backing :)
See elsewhere!

To note, yesterday I bought a high quality usb-stick to use as boot drive. Not that I like that, but that was the alternative.

 

[WI_Hedgehog]

Quality (name-brand) SSDs are inexpensive ($16 to $25) if you have a free SATA port....

 

[Louis2]

Quality (name-brand) SSDs are inexpensive ($16 to $25) if you have a free SATA port....

That is exactly the problem!! I do not have free SATA or NVME ports !!!
It is exactly that it is not the money !

I could have used a NVME-SSD placed in an USB-enclosure, but that are physical big things :( I did opt for a Samsung 256 GB Fit Plus mini USB Drive, which is fast and small. I dis choose for 256 GB, much bigger than needed because of speed and because I assume that the SSD will lost longer as you only use a small part of the capacity.