TCPDump – Lost Packets

Here is further guesswork. Hope this is helpful, but it may as well be embarassingly wrong.

tap0 has two ends, the kernel network stack end and the program interface. It seems to me if you supply 'nicserver' with tap0, it won't attach to it the way it is intended with tap devices, using the program interface. Instead, nicserver will simply write to it from the network stack end, and with no application reading from the program interface end, you will end up overflowing the device queue. This explains the dropped packets. Also, no packets will be delivered, which might explain the iptables result.

I guess if you let tcpdump capture on tap0 it actually attaches to the program interface end of tap0 and, voila, you see the packets. I searched the internet but found no source to confirm such behaviour. To falsify this theory, capture tap0 and see how it affects packet drops and the iptables log.

Finally, an advice that addresses your original problem: what about using the loopback device, instead? Like so:

nicserver -p 7801 -a lo

According to the Packet flow in Netfilter and General Networking schematic, tcpdump captures (AF_PACKET) after egress (qdisc). So it's normal you don't see the delay in tcpdump: the delay was already present at initial capture.

You'd have to capture it one step earlier, so involve a 3rd system:

S1: system1, runs tcpdump on outgoing interface
R: router (or bridge, at your convenience, this changes nothing), runs the qdisc netem
S2: system2, runs tcpdump on incoming interface

 __________________     ________________     __________________
|                  |   |                |   |                  |
| (S1) -- tcpdump -+---+- (R) -- netem -+---+- tcpdump -- (S2) |
|__________________|   |________________|   |__________________|

That means 3 network stacks involved, be they real, vm, network namespace (including ip netns, LXC, ...)

Optionally, It's also possible to cheat and move every special settings on the router (or bridge) by using an IFB interface with mirred traffic: allows by a trick (dedicated for this case) to insert netem sort-of-after ingress rather than on egress:

 _______     ______________________________________________     _______
|       |   |                                              |   |       |         
| (S1) -+---+- tcpdump -- ifb0 -- netem -- (R) -- tcpdump -+---+- (S2) |
|_______|   |______________________________________________|   |_______|

There's a basic IFB usage example in tc mirred manpage:

Using an ifb interface, it is possible to send ingress traffic through an instance of sfq:

# modprobe ifb
# ip link set ifb0 up
# tc qdisc add dev ifb0 root sfq
# tc qdisc add dev eth0 handle ffff: ingress
# tc filter add dev eth0 parent ffff: u32 \
  match u32 0 0 \
  action mirred egress redirect dev ifb0

Just use netem on ifb0 instead of sfq (and in non-initial network namespace, ip link add name ifbX type ifb works fine, without modprobe).

This still requires 3 network stacks for proper working.

using NFLOG

After a suggestion from JenyaKh, it turns out it's possible to capture a packet with tcpdump, before egress (thus before qdisc) and then on egress (after qdisc): by using iptables (or nftables) to log full packets to the netlink log infrastructure, and still reading them with tcpdump, then again using tcpdump on the egress interface. This requires only settings on S1 (and doesn't need a router/bridge anymore).

So with iptables on S1, something like:

iptables -A OUTPUT -o eth0 -j NFLOG --nflog-group 1

Specific filters should probably be added to match the test done, because tcpdump filter is limited on nflog interface (wireshark should handle it better).

If the answer capture is needed (here done in a different group, thus requiring an additional tcpdump):

iptables -A INPUT -i eth0 -j NFLOG --nflog-group 2

Depending on needs it's also possible to move them to raw/OUTPUT and raw/PREROUTING instead.

With tcpdump:

# tcpdump -i nflog:1 -n -tt ...

If a different group (= 2) was used for input:

# tcpdump -i nflog:2 -n -tt ...

Then at the same time, as usual:

# tcpdump -i eth0 -n -tt ...