FreeNX and public key authentication, problems logging in

I posted this question on the opensuse@opensuse.org before, but received no reply. I thought I’d try my luck here. Apologies if you’ve read this one before!

I have run into a problem using FreeNX in combination with public key authentication. I have Googled myself silly, but I can’t find the exact answer to my problem. I am hoping that someone here will know the answer.
First, let me tell you what I am trying to accomplish:

  • I don’t want my ssh server exposed to the Internet with password logins enabled. I only want public key authentication.
  • I want to have access to NX server from anywhere in the world.
  • I want NX to use a custom keypair, not the default supplied by FreeNX.

My environment:
openSUSE 11.2 (client and server)
KDE 4.3.4 (client and server)
FreeNX-0.7.2-26.5.i586 (server)
openssh-5.2p1-9.1.i586 (client and server)
NX Client for Linux 3.4.0-5 from NoMachine (client)
qtNX Client qtnx-0.0.1SVNr281-215.1.i586 (client)

The login problem persists, whether I am connected to the same network or to another network. Since ssh logins work, I don’t think a firewall is the culprit.

What do I have working so far:

  • The ssh server on the remote host only accepts public key authentication and I can login from the client without problems. So far so good.
  • FreeNX is installed on the remote host, using a custom keypair.
    Authentication with this key works.

I can login from the client with the nx user and the public key.

$ ssh -i ~/.ssh/id_dsa.key nx@remote
HELLO NXSERVER - Version 2.1.0-72 OS (GPL, using backend: 3.2.0)
NX> 105

So far so good. This proves that public key authentication for the user nx is correct.

Because FreeNX doesn’t support logging in to the ssh daemon using public key authentication, only user/password combinations, I can’t use the default ssh login that FreeNX has configured out of the box on openSUSE. To get around that, I have set up FreeNX to use its own passdb backend. In order to do that, I edited the file /etc/nxserver/node.conf and set the
following:

ENABLE_PASSDB_AUTHENTICATION="1"
ENABLE_SSH_AUTHENTICATION="0"

I then created a user account for myself in the following way:

# /usr/bin/nxserver --adduser <username>
# /usr/bin/nxserver --passwd <username>

I add the public key in the configuration dialog of the NoMachine client and try to connect, however this does not work. Here’s what happens when I use the NoMachine client to log in:

I get a dialog that says: “The NX service is not available or the NX access was disabled on host remote.”

Clicking “Detail”, the following text is displayed (anonymized):

NX> 203 NXSSH running with pid: 26275
NX> 285 Enabling check on switch command
NX> 285 Enabling skip of SSH config files
NX> 285 Setting the preferred NX options
NX> 200 Connected to address: XXX.XXX.212.221 on port: 22
NX> 202 Authenticating user: nx
NX> 208 Using auth method: publickey
HELLO NXSERVER - Version 2.1.0-72 OS (GPL, using backend: 3.2.0)
NX> 105 hello NXCLIENT - Version 2.1.0
NX> 134 Accepted protocol: 2.1.0
NX> 105 SET SHELL_MODE SHELL
NX> 105 SET AUTH_MODE PASSWORD
NX> 105 login
NX> 101 User: <username>
NX> 102 Password: 
NX> 103 Welcome to: deepthought user: <username>
NX> 105 listsession --user="<username>" --status="suspended,running"
--geometry="1280x800x24+render" --type="unix-kde"
NX> 127 Sessions list of user '<username>' for reconnect:

Display Type             Session ID                       Options  Depth
Screen         Status      Session Name
------- ---------------- -------------------------------- -------- -----
-------------- ----------- ------------------------------


NX> 148 Server capacity: not reached for user: <username>
NX> 105 startsession  --link="adsl" --backingstore="1" --encryption="1"
--cache="16M" --images="64M" --shmem="1" --shpix="1" --strict="0"
--composite="1" --media="0" --session="Test" --type="unix-kde"
--geometry="1024x768+128+0" --client="linux" --keyboard="pc102/us"
--screeninfo="1024x768x24+render" 

Permission denied (publickey).
NX> 280 Exiting on signal: 15

I read this as that the initial connection of the user nx to the host remote is made, using public key authentication. Then, my username and password are passed to the FreeNX service and I get logged in using username/password. I thought this would complete the login and should set up a desktop. Apparently though, there is some kind of third login process which uses public key authentication and this gets refused…?

I tried the same login with the qtNX client, but this complains about a missing shared library when I try to connect:

/usr/NX/bin/nxssh: error while loading shared libraries: libXcomp.so.3:
cannot open shared object file: No such file or directory

I am unable to find out what would provide that shared library though:

$ rpm -q --whatprovides libXcomp.so.3
no package provides libXcomp.so.3

Like I said, I have Googled this error extensively, but I haven’t found this exact situation anywhere. I’m stumped now and don’t know what to try next. Does anyone here have an idea?

Don’t know about the public key error, but this from http://wiki.linuxquestions.org/wiki/FreeNX relates to a libxcomp.so.3 error:

*  starting the nxagent, you get 

/usr/NX/bin/nxagent: error while loading shared libraries: libXcomp.so.3: cannot open shared object file: No such file or directory

Download nxcomp from [NoMachine NX - Download: NX Open Source Components](http://www.nomachine.com/sources.php) and install it. Copy libXcomp.* to /usr/lib. 

I believe opennx (another free client) also provides libxcomp.so.3

Hope that helps

Afterthought - I seem to remember the centos wiki on freenx recommending to to add nx to AllowUsers in the sshd conf for key authentication, not sure if that’s relevant if you’re not using the AllowUsers setting for ssh to begin with or not but may be worth trying

Thank you Ecky, I tried your suggestions but it took me a while to get round to it (newborn baby in the house)

I downloaded nxcomp from NoMachine as suggested. It won’t build though and the errors it threw at me were so cryptic that I did not dare look into them further for fear of a migraine setting on…

opennx does provide libxcomp as you said. I found an rpm for openSUSE 11.1, but this does not behave well on 11.2. It complains about invalid symbol links if I’m not mistaken and the error I received did not turn up anything useful when plugged into Google.
I then tried building from source, which did work after pulling in a bunch of devel packages. However, it still complained about an error regarding one of the wx libs that it needs. I gave up on opennx as well. I’m not at the client at the moment, otherwise I would paste in the exact errors.

I had done that as well, actually. For both nx and my own username. But despite being in the AllowUsers line, the error persists.

On a hunch, I decided to turn my sshd to DEBUG logging, to see if that provided anything useful and it did. I found out what the third login is and also why it fails.

In the sshd logfile (I don’t log sshd messages to /var/log/messages), I find the following lines when logging in through the NoMachine NX client:

Jan  9 12:24:57 deepthought sshd[5262]: debug1: Forked child 5266.
Jan  9 12:24:57 deepthought sshd[5266]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jan  9 12:24:57 deepthought sshd[5266]: debug1: inetd sockets after dupping: 3, 3
Jan  9 12:24:57 deepthought sshd[5266]: Connection from 192.168.X.X port 44408
Jan  9 12:24:57 deepthought sshd[5266]: debug1: Client protocol version 2.0; client software version OpenSSH_4.7
Jan  9 12:24:57 deepthought sshd[5266]: debug1: match: OpenSSH_4.7 pat OpenSSH_4*
Jan  9 12:24:57 deepthought sshd[5266]: debug1: Enabling compatibility mode for protocol 2.0
Jan  9 12:24:57 deepthought sshd[5266]: debug1: Local version string SSH-2.0-OpenSSH_5.2
Jan  9 12:24:57 deepthought sshd[5266]: debug1: PAM: initializing for "nx"
Jan  9 12:24:57 deepthought sshd[5266]: debug1: PAM: setting PAM_RHOST to "zaphod.lan"
Jan  9 12:24:57 deepthought sshd[5266]: debug1: PAM: setting PAM_TTY to "ssh"
Jan  9 12:24:57 deepthought sshd[5266]: debug1: temporarily_use_uid: 110/1001 (e=0/0)
Jan  9 12:24:57 deepthought sshd[5266]: debug1: trying public key file /var/lib/nxserver/home/.ssh/authorized_keys
Jan  9 12:24:57 deepthought sshd[5266]: debug1: fd 4 clearing O_NONBLOCK
Jan  9 12:24:57 deepthought sshd[5266]: debug1: matching key found: file /var/lib/nxserver/home/.ssh/authorized_keys, line 1
Jan  9 12:24:57 deepthought sshd[5266]: Found matching DSA key: 4b:9b:38:6b:24:33:6b:48:e4:f8:c4:5b:c9:f1:fd:98
Jan  9 12:24:57 deepthought sshd[5266]: debug1: restore_uid: 0/0
Jan  9 12:24:57 deepthought sshd[5266]: debug1: ssh_dss_verify: signature correct
Jan  9 12:24:57 deepthought sshd[5266]: debug1: do_pam_account: called
Jan  9 12:24:57 deepthought sshd[5266]: Accepted publickey for nx from 192.168.X.X port 44408 ssh2
Jan  9 12:24:57 deepthought sshd[5266]: debug1: monitor_child_preauth: nx has been authenticated by privileged process
Jan  9 12:24:57 deepthought sshd[5266]: debug1: PAM: establishing credentials
Jan  9 12:24:57 deepthought sshd[5266]: User child is on pid 5273
Jan  9 12:24:57 deepthought sshd[5273]: debug1: SELinux support disabled
Jan  9 12:24:57 deepthought sshd[5273]: debug1: PAM: establishing credentials
Jan  9 12:24:57 deepthought sshd[5273]: debug1: permanently_set_uid: 110/1001
Jan  9 12:24:57 deepthought sshd[5273]: debug1: Entering interactive session for SSH2.
Jan  9 12:24:57 deepthought sshd[5273]: debug1: server_init_dispatch_20
Jan  9 12:24:57 deepthought sshd[5273]: debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
Jan  9 12:24:57 deepthought sshd[5273]: debug1: input_session_request
Jan  9 12:24:57 deepthought sshd[5273]: debug1: channel 0: new [server-session]
Jan  9 12:24:57 deepthought sshd[5273]: debug1: session_new: session 0
Jan  9 12:24:57 deepthought sshd[5273]: debug1: session_open: channel 0
Jan  9 12:24:57 deepthought sshd[5273]: debug1: session_open: session 0: link with channel 0
Jan  9 12:24:57 deepthought sshd[5273]: debug1: server_input_channel_open: confirm session
Jan  9 12:24:58 deepthought sshd[5273]: debug1: server_input_channel_req: channel 0 request x11-req reply 0
Jan  9 12:24:58 deepthought sshd[5273]: debug1: session_by_channel: session 0 channel 0
Jan  9 12:24:58 deepthought sshd[5273]: debug1: session_input_channel_req: session 0 req x11-req
Jan  9 12:24:58 deepthought sshd[5273]: debug1: channel 1: new [X11 inet listener]
Jan  9 12:24:58 deepthought sshd[5273]: debug1: channel 2: new [X11 inet listener]
Jan  9 12:24:58 deepthought sshd[5273]: debug1: server_input_channel_req: channel 0 request shell reply 0
Jan  9 12:24:58 deepthought sshd[5273]: debug1: session_by_channel: session 0 channel 0
Jan  9 12:24:58 deepthought sshd[5273]: debug1: session_input_channel_req: session 0 req shell
Jan  9 12:24:58 deepthought sshd[5273]: debug1: Forced command (key option) '/usr/bin/nxserver'
Jan  9 12:25:01 deepthought sshd[5262]: debug1: Forked child 5509.
Jan  9 12:25:01 deepthought sshd[5509]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jan  9 12:25:01 deepthought sshd[5509]: debug1: inetd sockets after dupping: 3, 3
Jan  9 12:25:01 deepthought sshd[5509]: Connection from 127.0.0.1 port 46799
Jan  9 12:25:01 deepthought sshd[5509]: debug1: Client protocol version 2.0; client software version OpenSSH_5.2
Jan  9 12:25:01 deepthought sshd[5509]: debug1: match: OpenSSH_5.2 pat OpenSSH*
Jan  9 12:25:01 deepthought sshd[5509]: debug1: Enabling compatibility mode for protocol 2.0
Jan  9 12:25:01 deepthought sshd[5509]: debug1: Local version string SSH-2.0-OpenSSH_5.2
Jan  9 12:25:01 deepthought sshd[5509]: debug1: PAM: initializing for "user"
Jan  9 12:25:01 deepthought sshd[5509]: debug1: PAM: setting PAM_RHOST to "localhost"
Jan  9 12:25:01 deepthought sshd[5509]: debug1: PAM: setting PAM_TTY to "ssh"
Jan  9 12:25:01 deepthought sshd[5509]: debug1: temporarily_use_uid: 1000/100 (e=0/0)
Jan  9 12:25:01 deepthought sshd[5509]: debug1: trying public key file /home/user/.ssh/authorized_keys
Jan  9 12:25:01 deepthought sshd[5509]: debug1: fd 4 clearing O_NONBLOCK
Jan  9 12:25:01 deepthought sshd[5509]: debug1: restore_uid: 0/0
Jan  9 12:25:01 deepthought sshd[5509]: debug1: temporarily_use_uid: 1000/100 (e=0/0)
Jan  9 12:25:01 deepthought sshd[5509]: debug1: trying public key file /home/user/.ssh/authorized_keys
Jan  9 12:25:01 deepthought sshd[5509]: debug1: fd 4 clearing O_NONBLOCK
Jan  9 12:25:01 deepthought sshd[5509]: debug1: restore_uid: 0/0
Jan  9 12:25:01 deepthought sshd[5509]: Failed publickey for user from 127.0.0.1 port 46799 ssh2
Jan  9 12:25:01 deepthought sshd[5509]: debug1: do_cleanup
Jan  9 12:25:01 deepthought sshd[5509]: debug1: PAM: cleanup
Jan  9 12:25:02 deepthought sshd[5273]: Connection closed by 192.168.X.X

Especially the last bit of the output above is interesting. If I am reading things correctly, it is trying to authenticate my user name on localhost using public key authentication, where it should not. I specifically instructed FreeNX to not authenticate users using ssh but with its own user/pass db using these parameters:

ENABLE_PASSDB_AUTHENTICATION="1"
ENABLE_SSH_AUTHENTICATION="0"

I mentioned that in my initial post.

To experiment further, I tried setting up a public key for my local user account on the server, using

ssh-keygen -t rsa

and then adding the public key to ~/.ssh/authorized_keys.
When I tried this with a passphrase, of course I got prompted for my passphrase, which during an NX client login session would not be handy. So I tried again, this time with a key without passphrase, but still I get a “Permission denied” message when logging in.

The same thing happens when I try to login to my local sshd on the command line with my local username. I’m a bit further along, but still stumped…

I was looking for the same thing today and found your problem. Perhas this page:
tgharold.com: Tech Blog

Gives a hint. It basically says you have to clone your ssh daemon for a localhost restricted variant that does allow passwords, and a public that only allows keys