Ubuntu – UFW Allows 22 for IPv4 and IPv6 but SSH Disconnects When Enabling

18.04firewallkernelsshufw

sudo ufw disable followed by sudo ufw enable kicks me out of SSH

DMESG reports

[UFW BLOCK] IN=eth0 OUT= MAC=30:........ SRC=192.168.1.me DST=192.168.1.server LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=15776 DF PROTO=TCP SPT=55640 DPT=22 WINDOW=253 RES=0x00 ACK URGP=0

I can log back in without having to change rules via the console (UFW still enabled).

This started after upgrading Xenial (16.04) from kernel 4.4 to 4.15 (HWE). Upgrading to 18.04.1 did not solve the issue.

Versions:

  • iptables v1.6.1
  • ufw 0.35
  • 4.15.0-29-generic #31-Ubuntu
  • Ubuntu 18.04.1 LTS

UFW status verbose is (some rules were omitted, but they are all ALLOW)

Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22                         ALLOW IN    Anywhere
22 (v6)                    ALLOW IN    Anywhere (v6)

Why is this happening, or at least, how to revert to the expected behavior?

I looked at this answer, and I am not sure it applies, but here's /etc/ufw/before.rules

#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines


# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT

# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP

# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type source-quench -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT

# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type source-quench -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT

# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT

#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local

# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN

# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN

# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN

# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP

# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT

# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT

# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT

PS: I didn't expect this to "fix" the issue but just for reference I changed the port SSHD listens on (and the corresponding rule) and the problem persists.

Best Answer

Background, and bounds for the issue:

  • The issue only occurs when UFW, or iptables, with these ssh allow rules, is enabled and an ssh session is started. i.e. Any SSH session that was started with no iptables at all works fine, but may be subject to random drop outs once the rule set are put in place.
  • recall that ufw is merely a front end for iptables.
  • The issue is present even with kernel 4.18-rc8.

What is going on?

The sudo ufw allow in port 22 results in the following iptables rules segment:

Chain ufw-before-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination
      16     1553 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
     386   300622 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
      15     1068 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
      15     1068 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID

Upon sudo ufw disable followed by sudo ufw enable, and even though the ssh connection itself remains fine, the resulting iptables rule set seems to have forgotten the association with that particular connection and therefore classifies any incoming packets as invalid. Somehow the connection tracking table has become confused and the packet is not even considered NEW, but with incorrect flags, nor is it considered part of the existing connection.

Consider a very basic iptables equivalent of what ufw is doing. Two scripts, one for clearing the rule set and one for creating it:

#!/bin/sh
FWVER=0.01
#
# clear_firewall_min 2018.08.10 Ver:0.01
#       clear iptables minimum.
#       Currently for this question:
#       https://askubuntu.com/questions/1059781/ufw-allows-22-for-ipv4-and-ipv6-but-ssh-disconnects-when-enabling
#
echo "Loading clear_firewall_min version $FWVER..\n"

# The location of the iptables program
#
IPTABLES=/sbin/iptables

#Set some stuff
#
EXTIF="ens5"
UNIVERSE="0.0.0.0/0"

#Clearing any previous configuration
#
echo "  Clearing any existing rules and setting default policies.."
$IPTABLES -P INPUT ACCEPT
$IPTABLES -F INPUT

# Reset all IPTABLES counters
$IPTABLES -Z

#sleep 10

echo clear_firewall_min $FWVER done.

And:

#!/bin/sh
#
# test_firewall 2018.08.13 Ver:0.01
#       Minimum version of most basic iptables firewall.
#
# test_firewall 2018.08.09 Ver:0.01
#       Most basic iptables firewall.
#       Currently for this question:
#       https://askubuntu.com/questions/1059781/ufw-allows-22-for-ipv4-and-ipv6-but-ssh-disconnects-when-enabling
#

#sleep 50

# The location of the iptables program
#
IPTABLES=/sbin/iptables

#Set some stuff
#
EXTIF="ens5"
UNIVERSE="0.0.0.0/0"

#Clearing any previous configuration
#
#echo "  Clearing any existing rules and setting default policies.."
$IPTABLES -P INPUT DROP
$IPTABLES -F INPUT

# loopback interfaces are valid.
#
$IPTABLES -A INPUT -i lo -s $UNIVERSE -d $UNIVERSE -j ACCEPT

$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate INVALID -j LOG --log-prefix "IINVALID:" --log-level info
$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate INVALID -j DROP
$IPTABLES -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j LOG --log-prefix "NEW TCP no SYN:" --log-level info
$IPTABLES -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j DROP
$IPTABLES -A INPUT -i $EXTIF -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $EXTIF -p tcp -m conntrack --ctstate NEW --dport 22 -j ACCEPT

echo "test_firewall_min $FWVER done..." >> /dev/kmsg

sleep 3

Resulting in these packets counts after a clear/load cycle with an ssh session that was started after a load cycle:

doug@s17:~$ sudo iptables -v -x -n -L
Chain INPUT (policy DROP 3 packets, 220 bytes)
    pkts      bytes target     prot opt in     out     source               destination
       0        0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
      35     6388 LOG        tcp  --  ens5   *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID LOG flags 0 level 6 prefix "IINVALID:"
      35     6388 DROP       tcp  --  ens5   *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
       0        0 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:!0x17/0x02 ctstate NEW LOG flags 0 level 6 prefix "NEW TCP no SYN:"
       0        0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:!0x17/0x02 ctstate NEW
       9      680 ACCEPT     all  --  ens5   *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
       0        0 ACCEPT     tcp  --  ens5   *       0.0.0.0/0            0.0.0.0/0            ctstate NEW tcp dpt:22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 12 packets, 1408 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Notice the 35 invalid packets as I typed on the crippled ssh session terminal, and before PuTTY terminated.

Why did this stop working, it used to work?

Because this is 100% repeatable, a kernel bisection was relatively easy, just time consuming. The results were:

4d3a57f23dec59f0a2362e63540b2d01b37afe0a is the first bad commit
commit 4d3a57f23dec59f0a2362e63540b2d01b37afe0a
Author: Florian Westphal <fw@strlen.de>
Date:   Fri Jul 28 11:22:04 2017 +0200

    netfilter: conntrack: do not enable connection tracking unless needed

    Discussion during NFWS 2017 in Faro has shown that the current
    conntrack behaviour is unreasonable.

    Even if conntrack module is loaded on behalf of a single net namespace,
    its turned on for all namespaces, which is expensive.  Commit
    481fa373476 ("netfilter: conntrack: add nf_conntrack_default_on sysctl")
    attempted to provide an alternative to the 'default on' behaviour by
    adding a sysctl to change it.

    However, as Eric points out, the sysctl only becomes available
    once the module is loaded, and then its too late.

    So we either have to move the sysctl to the core, or, alternatively,
    change conntrack to become active only once the rule set requires this.

    This does the latter, conntrack is only enabled when a rule needs it.

    Reported-by: Eric Dumazet <edumazet@google.com>
    Signed-off-by: Florian Westphal <fw@strlen.de>
    Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

Link to the entire commit.

How to revert to the expected behavior?

After disabling ufw or clearing the iptables rules set, create a new SSH session. It will survive a subsequent ufw enable, but might be subject to a random drop out at some point.

This issue will be taken upstream at some point, via the related e-mail list.

EDIT: upstream e-mail thread (contains a work around). Workaround copied here:

echo 1 | sudo tee /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal

EDIT 2: upstream proposed patch , which I have tested and reported back.

EDIT 3: 2018.11.06: This has stalled upstream, and I haven't had time to pester them. I'll try to get back to it soon.

EDIT 4: 2019.03.17: I can not reliably reproduce this issue with kernel 5.0.

Related Question