Monitor network traffic of a process and its entire subprocesses tree

monitoringnetworkingprocess

I want to know the amount of the network traffic (inbound and outbound) in a time period, generated a specific process and all subprocesses that it spawns.

I have developed a software that contains a "job manager" that runs forever and generates no network traffic on its own. It instead spawns child "workers" that does the main work, including the majority of network traffic. The tricky point is, several "workers" may work simultaneously, and a single worker process is expected to exit after a short period (a few hours). Furthermore, these workers also spawns more subprocesses that generates traffic like git fetch that needs to be monitored as well.

There will be only one instance of "job manager" and it can be started or killed on-demand on my development and testing server, which runs Ubuntu Server 18.04, architecture amd64.

I want to monitor the network traffic of all the workers and the processes that workers spawn, for a prolonged period (one week or more). Is there a solution?

Best Answer

Probably the easiest way is to put the job manager in a network namespace. All child processes will also be in that namespace. Connect up the namespace via veth or macvlan, measure traffic on that interface.

Related Solutions

Networking – Only Allow Certain Outbound Traffic on Specific Interfaces

I would setup a separate routing table and a policy to route marked packets using that table and have iptables/netfilter mark certain packets.

Create a table: echo 1 known >> /etc/iproute2/rt_tables

Create a routing rule (the ip command is from iproute2): ip rule add from all fwmark 1 table known

We created a table called "known" and created a routing rule that says any packet with a mark equal to 1 gets routed according to the "known" table. I only called it known because it's for the list of known protocols - you can name it whatever you want. Now we setup the known table to route the proper way.

ip route add default dev eth0 table known

Create iptables rules:

iptables -t mangle -I PREROUTING -p tcp --dport 111 -j MARK --set-mark 1
iptables -t mangle -I PREROUTING -p tcp --dport 2049 -j MARK --set-mark 1

The example marks packets on the NFS ports (111, 2049) with a 1. We are adding this rule to the 'mangle' iptable. This is different from the routing tables and is not changeable; the mangle table is specifically for altering packets in any way other than NAT.

Now, to route everything else through the other interface, we add a route to the standard routing table.

ip route add default dev eth1

To really understand this, read sections 4 and 11 of the LARTC howto.

Show network connections of a process

You're looking for strace! I found this answer on askubuntu, but it's valid for Unix:

To start and monitor an new process:
strace -f -e trace=network -s 10000 PROCESS ARGUMENTS
To monitor an existing process with a known PID:
strace -p $PID -f -e trace=network -s 10000

Otherwise, but that's specific to Linux, you can run the process in an isolated network namespace and use wireshark to monitor the traffic. This will probably be more convenient than reading the strace log:

create a test network namespace:
ip netns add test
create a pair of virtual network interfaces (veth-a and veth-b):
ip link add veth-a type veth peer name veth-b
change the active namespace of the veth-a interface:
ip link set veth-a netns test
configure the IP addresses of the virtual interfaces:
ip netns exec test ifconfig veth-a up 192.168.163.1 netmask 255.255.255.0
ifconfig veth-b up 192.168.163.254 netmask 255.255.255.0
configure the routing in the test namespace:
ip netns exec test route add default gw 192.168.163.254 dev veth-a
activate ip_forward and establish a NAT rule to forward the traffic coming in from the namespace you created (you have to adjust the network interface and SNAT ip address):
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 192.168.163.0/24 -o YOURNETWORKINTERFACE -j SNAT --to-source YOURIPADDRESS
(You can also use the MASQUERADE rule if you prefer)
finally, you can run the process you want to analyze in the new namespace, and wireshark too:
ip netns exec test thebinarytotest
ip netns exec test wireshark
You'll have to monitor the veth-a interface.

Best Answer

Related Solutions

Networking – Only Allow Certain Outbound Traffic on Specific Interfaces

Show network connections of a process

Related Question