I'm currently stumped by a strange problem… I have a dual stack host to which I want to SSH. If I connect via IPv6 everything works like expected
datenwolf@foo ~/ > ssh -6 bar.example.com
Password:
datenwolf@bar ~/ >
However when doing the same via IPv4 it fails
datenwolf@foo ~/ > ssh -4 bar.example.com
Password:
Permission denied (publickey,keyboard-interactive).
datenwolf@foo ~/ >
Excerpt from /var/log/sshd for the failing login
Apr 24 16:34:03 [sshd] SSH: Server;Ltype: Version;Remote: www.xxx.yyy.zzz-38427;Protocol: 2.0;Client: OpenSSH_5.9p1 Debian-5ubuntu1
Apr 24 16:34:03 [sshd] SSH: Server;Ltype: Kex;Remote: www.xxx.yyy.zzz-38427;Enc: aes128-ctr;MAC: hmac-md5;Comp: none [preauth]
Apr 24 16:34:04 [sshd] SSH: Server;Ltype: Authname;Remote: www.xxx.yyy.zzz-38427;Name: wolfgangd [preauth]
Apr 24 16:34:07 [sshd] pam_access(sshd:account): access denied for user `datenwolf' from `foo.example.com'
Apr 24 16:34:07 [sshd] error: PAM: User account has expired for datenwolf from foo.example.com
Apr 24 16:34:07 [sshd] Connection closed by www.xxx.yyy.zzz [preauth]
Of course the account did not expire and I can perfectly log in via IPv6. Using Google I found various reports on the log messages but none of them matched my problem (in the sense that applying the proposed solutions didn't work for my case).
I'm pretty much out of ideas here.
Update
/var/log/sshd for successfull IPv6 login on the very same target machine:
Apr 24 16:56:42 [sshd] SSH: Server;Ltype: Version;Remote: 2001:x:x:x:x:x:x:x-46025;Protocol: 2.0;Client: OpenSSH_5.9p1 Debian-5ubuntu1
Apr 24 16:56:42 [sshd] SSH: Server;Ltype: Kex;Remote: 2001:x:x:x:x:x:x:x-46025;Enc: aes128-ctr;MAC: hmac-md5;Comp: none [preauth]
Apr 24 16:56:43 [sshd] SSH: Server;Ltype: Authname;Remote: 2001:x:x:x:x:x:x:x-46025;Name: datenwolf [preauth]
Apr 24 16:56:47 [sshd] Accepted keyboard-interactive/pam for datenwolf from 2001:x:x:x:x:x:x:x port 46025 ssh2
Apr 24 16:56:47 [sshd] pam_unix(sshd:session): session opened for user datenwolf by (uid=0)
I tried logging in from various machines all the same result: IPv6 works, IPv4 doesn't.
Update 2
For reference this are the used IP tables. Note that these are battle tested, i.e. they are in use for several years now and were not changed recently. Remote login via IPv4 did work with them.
IPv4 iptables:
Chain INPUT (policy ACCEPT 2144 packets, 336K bytes)
pkts bytes target prot opt in out source destination
132 20762 fail2ban-SSH tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
12M 14G ACCEPT all -- ppp0 * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
3111 95984 ACCEPT icmp -- ppp0 * 0.0.0.0/0 0.0.0.0/0
18692 1123K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
2 112 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1194
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:1194
4633 288K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:6880:6899
2826 154K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:6880:6899
4 160 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:123
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:123
44165 3069K REJECT all -- ppp0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT 48032 packets, 44M bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:631 reject-with icmp-port-unreachable
0 0 REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:515 reject-with icmp-port-unreachable
0 0 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:631 reject-with icmp-port-unreachable
0 0 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:515 reject-with icmp-port-unreachable
0 0 REJECT all -- ppp0 ppp0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
133K 8347K TCPMSS tcp -- * ppp0 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
Chain OUTPUT (policy ACCEPT 14378 packets, 2172K bytes)
pkts bytes target prot opt in out source destination
Chain fail2ban-SSH (1 references)
pkts bytes target prot opt in out source destination
132 20762 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
IPv6 ip6tables
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all * * ::/0 ::/0 rt type:0 segsleft:0
484K 86M ACCEPT icmpv6 * * ::/0 ::/0
105K 7943K ACCEPT tcp * * ::/0 ::/0 tcp dpt:22
0 0 ACCEPT udp * * ::/0 ::/0 udp dpt:1194
0 0 ACCEPT tcp * * ::/0 ::/0 tcp dpt:1194
0 0 ACCEPT udp * * ::/0 ::/0 udp dpts:6880:6899
0 0 ACCEPT tcp * * ::/0 ::/0 tcp dpts:6880:6899
0 0 ACCEPT tcp * * ::/0 ::/0 tcp dpt:123
0 0 ACCEPT udp * * ::/0 ::/0 udp dpt:123
0 0 ACCEPT all ppp0,sixxs * ::/0 ::/0 ctstate RELATED,ESTABLISHED
4164K 466M ACCEPT all !ppp0,sixxs * ::/0 ::/0
0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-port-unreachable
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all * * ::/0 ::/0 rt type:0 segsleft:0
2864 311K ACCEPT icmpv6 * * ::/0 ::/0
0 0 REJECT tcp * * ::/0 ::/0 multiport ports 631 reject-with icmp6-port-unreachable
0 0 REJECT udp * * ::/0 ::/0 multiport ports 631 reject-with icmp6-port-unreachable
0 0 REJECT tcp * * ::/0 ::/0 multiport ports 515 reject-with icmp6-port-unreachable
0 0 REJECT udp * * ::/0 ::/0 multiport ports 515 reject-with icmp6-port-unreachable
0 0 REJECT all ppp0,sixxs ppp0,sixxs ::/0 ::/0 reject-with icmp6-port-unreachable
0 0 accept_with_pmtu_clamp tcp ppp0,sixxs * !2001:x:x::/48 2001:x:x::/48 tcp dpt:22
18M 14G accept_with_pmtu_clamp all * * ::/0 ::/0 ctstate RELATED,ESTABLISHED
65503 5289K accept_with_pmtu_clamp all !ppp0,sixxs * ::/0 ::/0
0 0 REJECT all * * ::/0 ::/0 reject-with icmp6-port-unreachable
Chain OUTPUT (policy ACCEPT 8099K packets, 11G bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all * * ::/0 ::/0 rt type:0 segsleft:0
Chain accept_with_pmtu_clamp (3 references)
pkts bytes target prot opt in out source destination
0 0 TCPMSS tcp * ppp0,sixxs ::/0 ::/0 tcp flags:0x06/0x02 TCPMSS clamp to PMTU
18M 14G ACCEPT all * * ::/0 ::/0
Update 3
This is /etc/sshd/sshd_config of the system I try connect to, stripped of all comments:
Port 22
ListenAddress 0.0.0.0
ListenAddress ::
PubkeyAuthentication yes
PasswordAuthentication no
UsePAM yes
AllowAgentForwarding yes
AllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes
PrintMotd no
PrintLastLog no
UseDNS yes
Subsystem sftp /usr/lib64/misc/sftp-server
Best Answer
After things getting stranger and stranger (see the thread of comments in my question) I finally figured it out. First things first: The authentication process did fail in pam_access.so however not due to some misconfiguration in
/etc/security/access.confas it was suggested.To understand why, we must look at the setup of this box in particular: It acts as a router toward IPv4 which goes natively over the PPP link and IPv6 which is over a 6in4 tunnel. Also this box acts as a DNS recursive resolver, and here it is getting interesting. I did configure the resolver in a way that IPv4 reverse lookups are resolved recursively starting with the IPv4 root servers and IPv6 reverse lookups start with the IPv6 root servers. This setup did work when I first installed it.
Now my ISP enters the pictures and people who don't understand, how DNS amplification attacks work. To make a long story short: I know for sure that my ISP messes with incoming DNS packets at random, i.e. some things must be resolved through their own resolvers for some time now, while resolving other DNS addresses recursively on your own works – the official reason is to mitigate DNS amplification attacks, but they're doing it wrong then^1.
Since I didn't want to largely change my setup I just threw my ISP's DNS resolvers at the end of my local DNS resolver as nonrecursive forward, so if the recursive resolving attempt times out it tries my ISP's resolvers. This works so far. But when I did configure this I made a small mistake: I entered the ISP's DNS resolvers to work only from hosts within my local scope, i.e. 192.168.0.0/16 but forgot about localhost, aka my router, which is the host I tried to SSH into, for which the resolver would not take the ISP's resolvers into account.
pam_access.so attempts a reverse lookup on the peers address; and this closes the circle: Because for IPv6 reverse lookup the DNS IPv6 root servers would be accessed the packets would go though the 6in4 tunnel without my ISP messing with them, getting a response. But IPv4 reverse lookup would not be done over the ISP's resolvers by my own, which would receive no response and would ultimately report a NXHOST (or it would run in a timeout). Anyway pam_access.so won't see something it likes and just says "you shall not pass".
After I fixed that resolver configuration everything now works like a charm again. But I really have to step onto my ISP's toes now…
As to how I did resolve it? Well, by yanking up logging verbosity intensely studying
/var/log/everythingto see in which order things unfolded. When I saw my resolver logging the reverse lookup attempts I knew what was going on.1: from a DNS amplification mitigation point of view this is complete nonsense, because I did test and outgoing DNS packets get through just fine – however those are the packets they should filter. In fact every end customer ISP should drop all UDP packets which sender address doesn't match those of their customers