Log the dropped packets by IPTables only if its DROPPED by a specific rule

The NFLOG target can be used for this purpose. Here is a very basic example:

# Drop traffic by default
iptables -P INPUT DROP

# add your whitelists here
# iptables -A INPUT ...

# Pass the packets to NFLOG (just like LOG, but instead of syslog,
# it uses netlink). You can add extra filters such as '-p tcp' as usual
iptables -A INPUT -j NFLOG
# packets that get here will now be dropped per INPUT policy

# Finally you can use tcpdump to capture from this interface (there
# can only be one active user of nflog AFAIK)
tcpdump -i nflog ...

Refer to the iptables-extensions manual page for a description of the NFLOG target.

The creation of the TCP RST packet is from your rule

-A INPUT -p tcp -j REJECT --reject-with tcp-reset

The default policy (ACCEPT in your case) only applies to packets that do not match any of the rules in your chain. If a packet matches the rule above with the REJECT target, it will not be subject to the default policy and will be REJECTed (and generate a TCP RST) rather than ACCEPTed.

This TCP RST will not match your rule:

-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

because it is not RELATED to another established connection and it is not part of an ESTABLISHED connection. It will continue through your rules and match

-A OUTPUT -m limit -j LOG --log-prefix "UNKNOWN_OUTGOING: " --log-level 5

and end up in your log. If you do not want to log these RST packets, either adjust this rule to not match them or insert an earlier rule to match the RST packets and so something with them before they get here.

Something else I'm noticing is that the first packet you are logging is a SYN/ACK packet from a remote webserver, which looks like a response packet from the remote webserver to a SYN packet you would have earlier sent to begin the connection to the remote host on port 80. If you didn't send an initial SYN, I don't think the connection would match 'ESTABLISHED', but if you did send a SYN then I think the connection should mach 'ESTABLISHED'. This could be messing with which rule your RST ends up matching.