How to utilize TUN/TAP tunnel from user program

Here is further guesswork. Hope this is helpful, but it may as well be embarassingly wrong.

tap0 has two ends, the kernel network stack end and the program interface. It seems to me if you supply 'nicserver' with tap0, it won't attach to it the way it is intended with tap devices, using the program interface. Instead, nicserver will simply write to it from the network stack end, and with no application reading from the program interface end, you will end up overflowing the device queue. This explains the dropped packets. Also, no packets will be delivered, which might explain the iptables result.

I guess if you let tcpdump capture on tap0 it actually attaches to the program interface end of tap0 and, voila, you see the packets. I searched the internet but found no source to confirm such behaviour. To falsify this theory, capture tap0 and see how it affects packet drops and the iptables log.

Finally, an advice that addresses your original problem: what about using the loopback device, instead? Like so:

nicserver -p 7801 -a lo

The traditional answer is to run the program as another user and use iptables -m owner. That way, the network configuration is shared. However, with the advent of namespaces, there is an easier way.

With namespaces, you unshare the network, then create a virtual network link if you need limited network access.

To share unix domain sockets, all you need is to have a recent enough kernel, after this 2010 patch, so 2.6.36 or above (which is the case on all current distributions at the time of writing except RHEL/CentOS).

Run the program in its own IP namespace. Before starting the program, establish a virtual ethernet interface. There doesn't seem to be much documentation; I found the right incantation in a few blogs:

• Linux Namespaces at Arista
• Some notes on veth interfaces by waldner
• Introducing Linux Network Namespaces by Scott Lowe

In the snippet below, I use the ns_exec is the this wrapper around setns listed in the man page to bring up the host side of the network link from the process running in the restricted namespace. You don't actually need this: you can set up the host side of the link from outside the restricted namespace. Doing it from inside merely facilitate the flow control, otherwise you need some synchronization to set up the link after it's been established but before starting your program.

unshare -n -- sh -c '
  # Create a virtual ethernet interface called "confined",
  # with the other end called "global" in the namespace of PID 1
  ip link add confined type veth peer name global netns 1

  # Bring up the confined end of the network link
  ip addr add 172.16.0.2/30 dev confined
  ip link set confined up

  # Bring up the global end of the network link
  ns_exec /proc/1/ns/net ifconfig global 172.16.0.1 netmask 255.255.255.252 up

  # Execute the test program
  exec sudo -E -u "$TARGET_USER" "$0" "$@"
' /path/to/program --argument

You need to do all of this as root. The introduction of user namespaces in kernel 3.8 may make this doable with no special permissions, except setting up the global end of the network link.

I don't know how to share localhost between the two namespaces. Virtual ethernet interfaces create a point-to-point bridge. You can use iptables forwarding rules to redirect traffic from/to lo if desired.