Linux Networking – Simulating Packet Loss on Bridged Interface Using Netem

bridgelinuxnetworking

I am trying to simulate a network loss on a bridged interface.
I have 4 network interfaces on my server eth1, eth2 and eth3 are bridged in br0 and eth0 which is currently disconnected. As a matter of fact eth2 is also disconnected (down). I have connected to eth1 a video transmitting device and eth3 is connected directly to the network switch.
A receiving device is connected to the same switch but on another port. I am using netem to simulate network impairments on the path. This is an example of my netem command:
sudo tc qdisc add dev eth1 root netem delay 100ms 50ms loss 20%.
If I ping the transmitter I am getting exactly the time delay, jitter and packet loss configured with this command but this is somehow not affecting the network transport between the transmitter and decoder. If I run the same command on eth3 the link between the transmitter and receiver starts to drop, but the question is why the transport is running fine if if I define eth1 as a network interface?

Best Answer

The reason for this behaviour is described in the tc-netem(8) (bold mine):

delay

adds the chosen delay to the packets outgoing to chosen network interface.

loss random

adds an independent loss probability to the packets outgoing from the chosen network interface.

So tc ... netem works only on outgoing packets and has no effect on incoming packets from eth1. Applying the rule to eth3 allows netem to have the intented effect since it's now an outgoing interface for the video traffic.

If eth3 shouldn't have such rules applied, because it should behave normally with traffic not related to eth1, and only incoming traffic from eth1 should suffer these rules, then an intermediate interface should be put between eth1 and the bridge in order to have netem applied on an outgoing side.

An easy to understand example, would be to add an other bridge enslaving eth1 and also linked with a veth pair to the first bridge: then the tc rule can be applied on the veth interface of this additional bridge: since incoming packets from eth1 would then be outgoing via this veth interface, it would work as intended.

But actually the Intermediate Functional Block device (ifb0) is designed to be used for this kind of problem by inserting an artificial internal egress interface in the network flow, right after the incoming/ingress interface. ifb0 will still be located before most other network layers and stay mostly invisible. Since it's outgoing/egress (but only internally), netem will work on it.

So here's the solution to have netem work for incoming instead of outgoing traffic on eth1 interface, using ifb0's trick adapted from an example from tc-mirred(8).:

ip link add ifb0 type ifb || :
ip link set ifb0 up
tc qdisc add dev ifb0 root netem delay 100ms 50ms loss 20%
tc qdisc add dev eth1 handle ffff: ingress
tc filter add dev eth1 parent ffff: u32 match u32 0 0 action mirred egress redirect dev ifb0

_{^{u32 match u32 0 0 is the simplest possible ANY filter to use for the filter command to be accepted.}}

Of course it also works when eth1 is in a bridge.

Related Solutions

Linux – 802.3ad on FreeBSD and Linux using crossover cables

You can do that, but it won't do what you want. Due to an esoteric requirement of the 802.11ad standard that requires packets sent to the same host over the trunk to arrive in the same order they were sent in, the kernel can not load balance packets to the same host over multiple links -- it can only send packets to some hosts over one link, and packets to some other hosts over the other. Since you only have one host ( on each end ), all of the packets will end up going over one link anyhow.

Linux – how to get network QoS statistics in linux kernel

It is hard to give full details, especially not knowing what distribution you are using and how detailed you want the results to be - so I will just list some very basic examples.

If you are looking to get individual stats for each interface, we can break each item down separately (the following assumes root access on a distro like CentOS):

For bandwidth usage - I really like iftop. When you have iftop installed you could run the following command:

iftop -i eth0 -B

The -B option is for Bytes. Once inside iftop I like to press "T" to get a cummulative total.

For error rate you can easily see this in the output of ifconfig in your console. To get a more streamline approach you could run a very crude command like this (assuming something like Centos):

# ifconfig | grep -E "^\w|errors.* " | sed 's/pack.*errors:/Errors:/g' | sed 's/ drop.*//g' | sed 's/HW.*//g'
eth0      Link encap:Ethernet
          RX Errors:0
          TX Errors:0
eth1      Link encap:Ethernet
          RX Errors:3
          TX Errors:1
lo        Link encap:Local Loopback
          RX Errors:0
          TX Errors:0

This will give you the error count for each interface.

For collisions you can still use ifconfig. To get just the collision count, another basic command would be as follows:

# ifconfig | grep -E "^\w|collisions.* " | sed 's/pack.*collisions:/Collisions:/g' | sed 's/ txq.*//g' | sed 's/HW.*//g'
eth0      Link encap:Ethernet
          collisions:0
eth1      Link encap:Ethernet
          collisions:0
lo        Link encap:Local Loopback
          collisions:0

For dropped packets, still using ifconfig, you can run another simple command:

# ifconfig | grep -E "^\w|dropped.* " | sed 's/pack.*dropped:/Dropped:/g' | sed 's/ over.*//g' | sed 's/HW.*//g'
eth0      Link encap:Ethernet
          RX Dropped:0
          TX Dropped:0
eth1      Link encap:Ethernet
          RX Dropped:1
          TX Dropped:0
lo        Link encap:Local Loopback
          RX Dropped:0
          TX Dropped:0

If you are having a lot of bad packets (errors, collisions, or dropped), you could put the previous 3 commands into a bash script and use the watch command to monitor them:

FILE /tmp/netErrors:

#!/bin/sh

ifconfig | grep -E "^\w|errors.* " | sed 's/pack.*errors:/Errors:/g' | sed 's/ drop.*//g' | sed 's/HW.*//g'
echo
ifconfig | grep -E "^\w|collisions.* " | sed 's/pack.*collisions:/Collisions:/g' | sed 's/ txq.*//g' | sed 's/HW.*//g'
echo
ifconfig | grep -E "^\w|dropped.* " | sed 's/pack.*dropped:/Dropped:/g' | sed 's/ over.*//g' | sed 's/HW.*//g'
echo

COMMAND:

watch /tmp/netErrors

For QoS, it really depends on how you are setting it. For Linux, traffic control is common for setting QoS. To see the current QoS with traffic control you can run the following command:

# tc qdisc ls
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev tun0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1

tc - traffic control command qdisc - queuing disciplines ls - list

To get a good break down of traffic control you should check out the following link on traffic control.

Best Answer

Related Solutions

Linux – 802.3ad on FreeBSD and Linux using crossover cables

Linux – how to get network QoS statistics in linux kernel

Related Question