How to add marks together in iptables (targets MARK and CONNMARK)

iptablesopenwrtrouting

When you want to mark a packet in iptables, you would generally add the following line to your firewall script:

iptables -t mangle -A POSTROUTING -p tcp -m multiport --dports 80,443 -j MARK --set-mark 2

I know this subject is a little bit complicated, but let's focus just on the single rule. When you add something like the one above, in the /proc/net/nf_conntrack file some entries will have the mark set:

ipv4     2 tcp      6 3706 ESTABLISHED ... mark=2

The problem with the rule is that when you add another rule that matches, for instance source or destination address and set another mark, the previous mark will be rewritten to something you set. But there are some ways to "add" the marks. So if one rule set mark=2 and another rule set mark=5, then the resulting mark will be mark=7, or something like that.

I have one working example based on mwan3 , but I don't really get it. I know what the mangle table looks like after starting the tool, and what rules were added:

enter image description here

So to understand the mechanism, I have to know what actually happens to the arriving packet. But in this example, the marking rules are different.

We have two different WAN interfaces. Based on the marks, the packets will go to different routing tables. So what actually happens to a packet that is destined to port 443 and, for example, to port 1000? Could anyone help me analyze the rules?

Best Answer

The target "-j MARK --set-mark 2" will set the mark 2 on the packet, whatever the previous value was. If you want to avoid your mark to be erased, you can simply end the packet path in the chain with -j ACCEPT. For example : 

iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j MARK --set-mark 10
iptables -t mangle -A POSTROUTING -p tcp --dport 80 -j ACCEPT
iptables -t mangle -A POSTROUTING -d 8.8.8.8 -j MARK --set-mark 20
iptables -t mangle -A POSTROUTING -d 8.8.8.8 --dport 80 -j ACCEPT
# If you open a connection to 8.8.8.8:80, the mark will be 10

Although, you have to take care if you are in the main chain (eg. POSTROUTING) or in a custom chain: ACCEPT will end the main chain, while RETURN will end the current chain. It depends on your needs.

About the mwan3 example, it is quite hard to be sure without the filter, nat, and raw table, and without tc configuration.

However, it looks like this:

  • The "mwan3_rules" chain sends packets from a new connection with dport 80 or 443 to the "mwan3_policy_balanced" chain
  • The "mwan3_policy_balanced" chain set the mark 200 on 40% of the packets, and the mark 100 on the other ones
  • The "mwan3_rules" chain sends packets from a new connection not with dport 80 or 443 to the "mwan3_policy_wan_only" chain
  • The "mwan3_policy_wan_only" chain set the mark 100
  • I guess packet with mark 100 will go through cable, while packets with mark 200 will go through lte

So 40% of http(s) traffic will go through lte, 60% through cable, and everything else through cable.

If your goal is to loadbalance connections on your 2 ISP, you should probably write your own iptables rules from scratch, since mwan3 ones are hard to read.

You could start with this question.

Good luck !

Edition:

The documentation states:

--set-xmark value[/mask]
    Zero out the bits given by mask and XOR value into the ctmark.

If you have a 0x100 mark, and try to set xmark 0x200/0xff00:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR 0000 0010 0000 0000 --> 0000 0010 0000 0000 --> 0x200 --> 512

If you have a 0x100 mark, and try to set xmark 0x200/0xf000:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 0000 0000 0000 --> 0000 0001 0000 0000
  • XOR value into the ctmark: 0000 0001 0000 0000 XOR 0000 0010 0000 0000 --> 0000 0011 0000 0000 --> 0x300 --> 768

If you have a 0x100 mark, and try to set xmark 0x100/0xf000:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 0000 0000 0000 --> 0000 0001 0000 0000
  • XOR value into the ctmark: 0000 0001 0000 0000 XOR 0000 0001 0000 0000 --> 0000 0000 0000 0000 --> 0x000 --> 0

If you have a 0x100 mark, and try to set xmark 0x100/0xff00:

  • In the ctmark, clear the bits given by the mask: 0000 0001 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR 0000 0001 0000 0000 --> 0000 0001 0000 0000 --> 0x100 --> 256

In the mwan3 file the case is always this one:

  • You have a 0x0 mark, and try to set xmark 0x??00/0xff00
  • In the ctmark, clear the bits given by the mask: 0000 0000 0000 0000 AND NOT 1111 1111 0000 0000 --> 0000 0000 0000 0000
  • XOR value into the ctmark: 0000 0000 0000 0000 XOR ???? ???? 0000 0000 --> ???? ???? 0000 0000 --> 0x??00 --> ?
  • With this mask and these values, set-xmark just replace the previous value

Now, let's go through the chains:

  • Prerouting jumps to mwan3_hook.
  • mwan3_hook restore connmarks into marks
  • mwan3_hook sends the new connections (mark = 0x0) to mwan3_ifaces
  • mwan3_ifaces sends the new connections to mwan3_iface_wan
  • mwan3_iface_wan set the mark 0xff00 to the new connections from interface eth0, when the source address is in the ipset list mwan3_connected
  • mwan3_iface_wan set the mark 0x100 to the other new connections on interface eth0
  • mwan3_ifaces sends the new connections to mwan3_iface_lte
  • mwan3_iface_lte set the mark 0xff00 to the new connections from interface wwan, when the source address is in the ipset list mwan3_connected
  • mwan3_iface_lte set the mark 0x200 to the other new connections on interface wwan
  • Note : at this point, all incoming connections are marked
  • mwan3_hook sends the connections to mwan3_connected
  • mwan3_connected set the mark 0xff00 when the destination address is in the ipset list mwan3_connected
  • mwan3_hook sends the connections to mwan3_track
  • mwan3_track set the mark 0xff00 to the connection if the destination ip is in the ipset list mwan3_track_wan, the packet is an icmp echo of 32 bytes length
  • mwan3_track set the mark 0xff00 to the connection if the destination ip is in the ipset list mwan3_track_lte, the packet is an icmp echo of 32 bytes length
  • mwan3_hook sends the connections to mwan3_rules
  • mwan3_rules sends the new tcp/80 or tcp/443 connections from the inside to mwan3_policy_balanced
  • mwan3_policy_balanced set the mark 0x200 to 40% of the new connection
  • mwan3_policy_balanced set the mark 0x100 to the other new connection
  • mwan3_hook restore marks into connmarks
  • mwan3_hook set the mark 0xff00 to the connections still not marked (from the inside, not icmp type 8, tcp/80, or tcp/443)
  • Prerouting jumps to fwmark, but the chain is not on the screen
  • Forward fix the mss
  • Output jumps to mwan3_hook, and every steps are done again.

In the end, we have 3 states:

  • 0x100 mark (256) : connections from wan, and 60% of http(s) connections to the internet
  • 0x200 mark (512) : connections from lte, and 40% of http(s) connections to the internet
  • 0xff00 mark (65280) : other traffic

Since we haven't the ip rule, we can only guess:

  • 0x100 mark will go through wan routing table
  • 0x200 mark will go through lte routing table
  • 0xff00 will go through another routing table
Related Question