sshd doesn't log remote IP addresses

English Network/Internet
#1

The problem is that all ssh connections look like they are coming from server’s public IP address instead of remote server’s IP address.

I made couple of logins from different servers and here are the results:

root@server:~> last | head
testuser pts/2 <server’s public IP> Thu Nov 5 11:11 still logged in
testuser pts/2 <server’s public IP> Thu Nov 5 11:09 - 11:10 (00:00)
testuser pts/1 <server’s public IP> Thu Nov 5 10:56 still logged in
testuser pts/0 <server’s public IP> Thu Nov 5 10:48 still logged in
testuser pts/0 <server’s public IP> Thu Nov 5 10:48 - 10:48 (00:00)

Same thing in /var/log/messages after every connection.

server sshd[3190]: Accepted keyboard-interactive/pam for testuser from <server’s public IP> port 56680 ssh2

So, what’s wrong with my server’s settings? I have no clue…

#2

It is most likely something in your network equipment. You need to describe how you are connected to Internet, how you make access for other external servers available, what devices/software you are using for this before anyone can even hazard a guess.

#3

The server is connected to Internet via Inteno FG500 router. Router has NAT enabled and port forwarding for ssh connections.
For test logins I have used different internet connection (mobile 4G) so that server and my laptop is not in the same local network.

It’s also impossible to trace & block brute force attacks because of missing remote hosts IP. Here is a sample of /var/log/messages:

sshd[13051]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<server's public IP>  user=root
sshd[13051]: Failed password for invalid user root from <server's public IP> port 55982 ssh2
sshd[13051]: Failed password for invalid user root from <server's public IP> port 55982 ssh2
sshd[13051]: Failed password for invalid user root from <server's public IP> port 55982 ssh2
sshd[13051]: Disconnected from invalid user root <server's public IP> port 55982 [preauth]

I have another server for testing purposes also running Leap 15.2. That server doesn’t have this kind of problem (router and Internet provider is not the same).

#4

Are you doing SSH port forwarding?

TSU

#5

I don’t think so…
Just ‘ssh testuser@<server’s IP> -p <port number>’ to connect server.

#6

So your “server public IP” is likely not your server public IP, but your router public IP. You really need to ask support community of your router. It sounds like it performs SNAT when forwarding packets.

If you provide network packet capture during ssh connection it is possible to say exactly whether the issue is in your server or external to it.

#7

OK. Thanks! I’ll check router settings…

Anyway, this is a little bit confusing because server’s firewall gets remote IP addresses correctly:

IN_block_REJECT: IN=eth0 OUT= MAC=64:00:6a:79:ef:4f:00:22:07:2e:3e:2e:08:00 SRC=89.231.xxx.xxx DST=192.168.1.100
IN_block_REJECT: IN=eth0 OUT= MAC=64:00:6a:79:ef:4f:00:22:07:2e:3e:2e:08:00 SRC=72.251.xxx.xxx DST=192.168.1.100

Here is tcpdump during ssh connection:

17:14:27.752274 IP (tos 0x0, ttl 63, id 33466, offset 0, flags [DF], proto TCP (6), length 60)    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [S], cksum 0x44b5 (correct), seq 1431237085, win 64240, options [mss 1460,sackOK,TS val 2491706642 ecr 0,nop,wscale 7], length 0
17:14:27.752307 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [S.], cksum 0x8a26 (incorrect -> 0xad00), seq 1144116460, ack 1431237086, win 65160, options [mss 1460,sackOK,TS val 1704662354 ecr 2491706642,nop,wscale 7], length 0
17:14:27.754812 IP (tos 0x0, ttl 63, id 33467, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xd85b (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 2491706646 ecr 1704662354], length 0
17:14:27.758164 IP (tos 0x0, ttl 63, id 33468, offset 0, flags [DF], proto TCP (6), length 73)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x1696 (correct), seq 1:22, ack 1, win 502, options [nop,nop,TS val 2491706646 ecr 1704662354], length 21
17:14:27.758189 IP (tos 0x0, ttl 64, id 64618, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags .], cksum 0x8a1e (incorrect -> 0xd83a), seq 1, ack 22, win 509, options [nop,nop,TS val 1704662359 ecr 2491706646], length 0
17:14:27.768777 IP (tos 0x0, ttl 64, id 64619, offset 0, flags [DF], proto TCP (6), length 73)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a33 (incorrect -> 0x166a), seq 1:22, ack 22, win 509, options [nop,nop,TS val 1704662370 ecr 2491706646], length 21
17:14:27.770639 IP (tos 0x0, ttl 63, id 33469, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xd810 (correct), seq 22, ack 22, win 502, options [nop,nop,TS val 2491706663 ecr 1704662370], length 0
17:14:27.770660 IP (tos 0x0, ttl 64, id 64620, offset 0, flags [DF], proto TCP (6), length 1132)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8e56 (incorrect -> 0x9071), seq 22:1102, ack 22, win 509, options [nop,nop,TS val 1704662372 ecr 2491706663], length 1080
17:14:27.772525 IP (tos 0x0, ttl 63, id 33470, offset 0, flags [DF], proto TCP (6), length 1444)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0xc95a (correct), seq 22:1414, ack 22, win 502, options [nop,nop,TS val 2491706663 ecr 1704662370], length 1392
17:14:27.772542 IP (tos 0x0, ttl 64, id 64621, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags .], cksum 0x8a1e (incorrect -> 0xce65), seq 1102, ack 1414, win 501, options [nop,nop,TS val 1704662374 ecr 2491706663], length 0
17:14:27.773357 IP (tos 0x0, ttl 63, id 33471, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xce6c (correct), seq 1414, ack 1102, win 494, options [nop,nop,TS val 2491706665 ecr 1704662372], length 0
17:14:27.776101 IP (tos 0x0, ttl 63, id 33472, offset 0, flags [DF], proto TCP (6), length 100)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x55c1 (correct), seq 1414:1462, ack 1102, win 501, options [nop,nop,TS val 2491706668 ecr 1704662374], length 48
17:14:27.776124 IP (tos 0x0, ttl 64, id 64622, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags .], cksum 0x8a1e (incorrect -> 0xce2d), seq 1102, ack 1462, win 501, options [nop,nop,TS val 1704662377 ecr 2491706668], length 0
17:14:27.787355 IP (tos 0x0, ttl 64, id 64623, offset 0, flags [DF], proto TCP (6), length 432)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8b9a (incorrect -> 0xf02f), seq 1102:1482, ack 1462, win 501, options [nop,nop,TS val 1704662389 ecr 2491706668], length 380
17:14:27.789179 IP (tos 0x0, ttl 63, id 33473, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xcc9a (correct), seq 1462, ack 1482, win 499, options [nop,nop,TS val 2491706681 ecr 1704662389], length 0
17:14:27.798203 IP (tos 0x0, ttl 63, id 33474, offset 0, flags [DF], proto TCP (6), length 68)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0xc256 (correct), seq 1462:1478, ack 1482, win 501, options [nop,nop,TS val 2491706690 ecr 1704662389], length 16
17:14:27.798223 IP (tos 0x0, ttl 64, id 64624, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags .], cksum 0x8a1e (incorrect -> 0xcc75), seq 1482, ack 1478, win 501, options [nop,nop,TS val 1704662399 ecr 2491706690], length 0
17:14:27.800132 IP (tos 0x0, ttl 63, id 33475, offset 0, flags [DF], proto TCP (6), length 96)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x2851 (correct), seq 1478:1522, ack 1482, win 501, options [nop,nop,TS val 2491706692 ecr 1704662399], length 44
17:14:27.800146 IP (tos 0x0, ttl 64, id 64625, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags .], cksum 0x8a1e (incorrect -> 0xcc45), seq 1482, ack 1522, win 501, options [nop,nop,TS val 1704662401 ecr 2491706692], length 0
17:14:27.800190 IP (tos 0x0, ttl 64, id 64626, offset 0, flags [DF], proto TCP (6), length 96)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a4a (incorrect -> 0x8ee8), seq 1482:1526, ack 1522, win 501, options [nop,nop,TS val 1704662401 ecr 2491706692], length 44
17:14:27.802251 IP (tos 0x0, ttl 63, id 33476, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xcc17 (correct), seq 1522, ack 1526, win 501, options [nop,nop,TS val 2491706694 ecr 1704662401], length 0
17:14:27.802400 IP (tos 0x0, ttl 63, id 33477, offset 0, flags [DF], proto TCP (6), length 120)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x75b3 (correct), seq 1522:1590, ack 1526, win 501, options [nop,nop,TS val 2491706694 ecr 1704662401], length 68
17:14:27.811680 IP (tos 0x0, ttl 64, id 64627, offset 0, flags [DF], proto TCP (6), length 128)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a6a (incorrect -> 0x3177), seq 1526:1602, ack 1590, win 501, options [nop,nop,TS val 1704662413 ecr 2491706694], length 76
17:14:27.814978 IP (tos 0x0, ttl 63, id 33478, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xcb6e (correct), seq 1590, ack 1602, win 501, options [nop,nop,TS val 2491706707 ecr 1704662413], length 0
17:14:27.816221 IP (tos 0x0, ttl 63, id 33479, offset 0, flags [DF], proto TCP (6), length 144)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x6b8c (correct), seq 1590:1682, ack 1602, win 501, options [nop,nop,TS val 2491706707 ecr 1704662413], length 92
17:14:27.817030 IP (tos 0x0, ttl 64, id 64628, offset 0, flags [DF], proto TCP (6), length 112)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a5a (incorrect -> 0x73e0), seq 1602:1662, ack 1682, win 501, options [nop,nop,TS val 1704662418 ecr 2491706707], length 60
17:14:27.860138 IP (tos 0x0, ttl 63, id 33480, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xcaa5 (correct), seq 1682, ack 1662, win 501, options [nop,nop,TS val 2491706751 ecr 1704662418], length 0
17:14:31.326255 IP (tos 0x0, ttl 63, id 33481, offset 0, flags [DF], proto TCP (6), length 136)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x2b1c (correct), seq 1682:1766, ack 1662, win 501, options [nop,nop,TS val 2491710218 ecr 1704662418], length 84
17:14:31.335376 IP (tos 0x0, ttl 64, id 64629, offset 0, flags [DF], proto TCP (6), length 96)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a4a (incorrect -> 0x83ad), seq 1662:1706, ack 1766, win 501, options [nop,nop,TS val 1704665937 ecr 2491710218], length 44
17:14:31.337029 IP (tos 0x0, ttl 63, id 33482, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xaed0 (correct), seq 1766, ack 1706, win 501, options [nop,nop,TS val 2491710229 ecr 1704665937], length 0
17:14:31.337086 IP (tos 0x0, ttl 63, id 33483, offset 0, flags [DF], proto TCP (6), length 136)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0x56fa (correct), seq 1766:1850, ack 1706, win 501, options [nop,nop,TS val 2491710229 ecr 1704665937], length 84
17:14:31.337534 IP (tos 0x0, ttl 64, id 64630, offset 0, flags [DF], proto TCP (6), length 80)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a3a (incorrect -> 0x4c62), seq 1706:1734, ack 1850, win 501, options [nop,nop,TS val 1704665939 ecr 2491710229], length 28
17:14:31.339201 IP (tos 0x0, ttl 63, id 33484, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xae5c (correct), seq 1850, ack 1734, win 501, options [nop,nop,TS val 2491710231 ecr 1704665939], length 0
17:14:31.340034 IP (tos 0x0, ttl 63, id 33485, offset 0, flags [DF], proto TCP (6), length 164)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0xe0ee (correct), seq 1850:1962, ack 1734, win 501, options [nop,nop,TS val 2491710231 ecr 1704665939], length 112
17:14:31.356383 IP (tos 0x0, ttl 64, id 64631, offset 0, flags [DF], proto TCP (6), length 552)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8c12 (incorrect -> 0x4957), seq 1734:2234, ack 1962, win 501, options [nop,nop,TS val 1704665958 ecr 2491710231], length 500
17:14:31.359067 IP (tos 0x0, ttl 63, id 33486, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xabd5 (correct), seq 1962, ack 2234, win 498, options [nop,nop,TS val 2491710250 ecr 1704665958], length 0
17:14:31.359092 IP (tos 0x0, ttl 64, id 64632, offset 0, flags [DF], proto TCP (6), length 96)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a4a (incorrect -> 0x180b), seq 2234:2278, ack 1962, win 501, options [nop,nop,TS val 1704665960 ecr 2491710250], length 44
17:14:31.360703 IP (tos 0x0, ttl 63, id 33487, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xaba1 (correct), seq 1962, ack 2278, win 501, options [nop,nop,TS val 2491710253 ecr 1704665960], length 0
17:14:31.362227 IP (tos 0x10, ttl 63, id 33488, offset 0, flags [DF], proto TCP (6), length 512)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [P.], cksum 0xee08 (correct), seq 1962:2422, ack 2278, win 501, options [nop,nop,TS val 2491710253 ecr 1704665960], length 460
17:14:31.363210 IP (tos 0x10, ttl 64, id 64633, offset 0, flags [DF], proto TCP (6), length 160)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a8a (incorrect -> 0xe92e), seq 2278:2386, ack 2422, win 501, options [nop,nop,TS val 1704665964 ecr 2491710253], length 108
17:14:31.363396 IP (tos 0x10, ttl 64, id 64634, offset 0, flags [DF], proto TCP (6), length 168)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a92 (incorrect -> 0x113f), seq 2386:2502, ack 2422, win 501, options [nop,nop,TS val 1704665965 ecr 2491710253], length 116
17:14:31.366055 IP (tos 0x10, ttl 63, id 33489, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xa8ec (correct), seq 2422, ack 2502, win 501, options [nop,nop,TS val 2491710258 ecr 1704665964], length 0
17:14:31.437584 IP (tos 0x10, ttl 64, id 64635, offset 0, flags [DF], proto TCP (6), length 112)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [P.], cksum 0x8a5a (incorrect -> 0x0290), seq 2502:2562, ack 2422, win 501, options [nop,nop,TS val 1704666039 ecr 2491710258], length 60
17:14:31.484152 IP (tos 0x10, ttl 63, id 33490, offset 0, flags [DF], proto TCP (6), length 52)
    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags .], cksum 0xa7f0 (correct), seq 2422, ack 2562, win 501, options [nop,nop,TS val 2491710375 ecr 1704666039], length 0

85.xxx.xxx.xxx is router’s public IP address…

#8

what does this show - mine shows the ip for all login attempts

grep USER_LOGIN /var/log/audit/audit.log 
type=USER_LOGIN msg=audit(1604587431.417:441447): pid=21451 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="(unknown)" exe="/usr/sbin/sshd" hostname=? addr=198.211.116.75 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604589869.829:442172): pid=32304 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=46.101.206.18 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604590126.237:442249): pid=8913 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=159.65.238.245 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604591110.826:442566): pid=18362 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=46.107.228.254 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604591113.554:442571): pid=18364 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=46.107.228.254 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604591537.807:442698): pid=27457 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="(unknown)" exe="/usr/sbin/sshd" hostname=? addr=71.210.33.33 terminal=ssh res=failed'
type=USER_LOGIN msg=audit(1604592445.148:443010): pid=4736 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=login acct="root" exe="/usr/sbin/sshd" hostname=? addr=199.126.198.116 terminal=ssh res=failed'
#9

Connected to the Internet or, connected to an ISP?

  • If an ISP, are you certain that, the ISP’s routers are forwarding the incoming packets without performing any address translation?

[HR][/HR]Given that, the Inteno FG500 is a residential gateway for Fiber to the Home, I suspect that, the Internet connection is via an ISP with a residential contract …

#10

In my log file only visible ips are same as my public ip:

type=USER_AUTH msg=audit(1604595606.702:1126): pid=12406 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=PAM:authentication grantors=? acct="root" exe="/usr/sbin/sshd" hostname=85.xxx.xxx.xxx addr=85.xxx.xxx.xxx terminal=ssh res=failed'
type=USER_AUTH msg=audit(1604595608.890:1127): pid=12404 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='op=challenge-response acct="(unknown)" exe="/usr/sbin/sshd" hostname=? addr=85.xxx.xxx.xxx terminal=ssh res=failed'
#11

Connected to ISP with a residential contract… I’m not 100% sure what router does to incoming packets.
Anyway… I can filter certain incoming IP:s with firewall, I can see remote IP:s in firewall logs but for some reason with sshd all the connections look like they are coming from my public ip address (that one router gets from ISP).

#12

And these servers are trying to connect to port 22 on your server? We only see IP addresses, nothing more. We have no information to decide whether this is “correct” or not.

Here is tcpdump during ssh connection:

17:14:27.752274 IP (tos 0x0, ttl 63, id 33466, offset 0, flags [DF], proto TCP (6), length 60)    85.xxx.xxx.xxx.57040 > 192.168.1.100.22: Flags [S], cksum 0x44b5 (correct), seq 1431237085, win 64240, options [mss 1460,sackOK,TS val 2491706642 ecr 0,nop,wscale 7], length 0
17:14:27.752307 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.1.100.22 > 85.xxx.xxx.xxx.57040: Flags [S.], cksum 0x8a26 (incorrect -> 0xad00), seq 1144116460, ack 1431237086, win 65160, options [mss 1460,sackOK,TS val 1704662354 ecr 2491706642,nop,wscale 7], length 0

85.xxx.xxx.xxx is router’s public IP address…

So it confirms that whatever reason for this problem is, it is external to your server. Your server already gets “wrong” address.