UDP to Broadcast – How to Transform a UDP Unicast Packet

I think it depends on you to make the kernel knows which is blackhole address type.

From xt_addrtype.h file in iptables source code, you can see:

/* rtn_type enum values from rtnetlink.h, but shifted */                        
enum {                                                                          
    XT_ADDRTYPE_UNSPEC = 1 << 0,                                                
    XT_ADDRTYPE_UNICAST = 1 << 1,   /* 1 << RTN_UNICAST */                      
    XT_ADDRTYPE_LOCAL  = 1 << 2,    /* 1 << RTN_LOCAL, etc */                   
    XT_ADDRTYPE_BROADCAST = 1 << 3,                                             
    XT_ADDRTYPE_ANYCAST = 1 << 4,                                               
    XT_ADDRTYPE_MULTICAST = 1 << 5,                                             
    XT_ADDRTYPE_BLACKHOLE = 1 << 6,                                             
    XT_ADDRTYPE_UNREACHABLE = 1 << 7,                                           
    XT_ADDRTYPE_PROHIBIT = 1 << 8,                                              
    XT_ADDRTYPE_THROW = 1 << 9,                                                 
    XT_ADDRTYPE_NAT = 1 << 10,                                                  
    XT_ADDRTYPE_XRESOLVE = 1 << 11,                                             
};

And in rtnetlink.h, you will see the same definition:

enum {                                                                          
    RTN_UNSPEC,                                                                 
    RTN_UNICAST,        /* Gateway or direct route  */                          
    RTN_LOCAL,      /* Accept locally       */                                  
    RTN_BROADCAST,      /* Accept locally as broadcast,                         
                   send as broadcast */                                         
    RTN_ANYCAST,        /* Accept locally as broadcast,                         
                   but send as unicast */                                       
    RTN_MULTICAST,      /* Multicast route      */                              
    RTN_BLACKHOLE,      /* Drop             */                                  
    RTN_UNREACHABLE,    /* Destination is unreachable   */                      
    RTN_PROHIBIT,       /* Administratively prohibited  */                      
    RTN_THROW,      /* Not in this table        */                              
    RTN_NAT,        /* Translate this address   */                              
    RTN_XRESOLVE,       /* Use external resolver    */                          
    __RTN_MAX                                                                   
};

You can see iptables use the same definition of address type with kernel tcp networking stack.

Then from man ip:

Route types:

      unicast - the route entry describes real paths to the destinations covered by the route prefix.

      unreachable  - these destinations are unreachable.  Packets are discarded and the ICMP message host unreachable is generated.
               The local senders get an EHOSTUNREACH error.

      blackhole - these destinations are unreachable.  Packets are discarded silently.  The local senders get an EINVAL error.

      prohibit - these destinations are unreachable.  Packets are discarded and the  ICMP  message  communication  administratively
               prohibited is generated.  The local senders get an EACCES error.

      local - the destinations are assigned to this host.  The packets are looped back and delivered locally.

      broadcast - the destinations are broadcast addresses.  The packets are sent as link broadcasts.

      throw  - a special control route used together with policy rules. If such a route is selected, lookup in this table is termi‐
               nated pretending that no route was found.  Without policy routing it is equivalent to the absence of the route in the routing
               table.   The  packets  are  dropped  and the ICMP message net unreachable is generated.  The local senders get an ENETUNREACH
               error.

      nat - a special NAT route.  Destinations covered by the prefix are considered to  be  dummy  (or  external)  addresses  which
               require  translation  to  real  (or  internal)  ones  before forwarding.  The addresses to translate to are selected with the
               attribute Warning: Route NAT is no longer supported in Linux 2.6.

               via.

      anycast - not implemented the destinations are anycast addresses assigned to this host.  They are mainly equivalent to  local
               with one difference: such addresses are invalid when used as the source address of any packet.

      multicast - a special type used for multicast routing.  It is not present in normal routing tables.

So when you define a route to a network by ip command and mark it as a blackhole route, the kernel now make this network address blackhole type:

ip route add blackhole X.X.X.X/24

I just tested smcroute with two network namespaces and two veth pairs. Setup:

ns1         <-- main namespace -->    ns2
10.0.0.1 -- 10.0.0.254  10.0.1.254 -- 10.0.1.1
veth0b      veth0a          veth1a    veth1b

The Debian smcroute package is version 2.0.0, and doesn't seem to support virtual eth, so I installed version 2.3.1 from the smcroute homepage. The multicast route howto of smcroute is also very helpful.

I used the ssmping package to test multicasts. I ran ssmpingd in ns2, while pinging with ssmping -4 -I veth0b 10.0.1.1 from ns1. These are source-specific multicasts (SSM) using group 232.43.211.234, you can also test any-source multicasts (ASM) with asmping. I don't know what LAN messenger uses.

I enabled forwarding in the main namespace to allow the unicast ping requests to get through, then did

smcroutectl add veth1a 10.0.1.1 232.43.211.234 veth0a

and everything worked fine. I would expect it also to work, adjusted to your setup, though you also may have to smcroutectl join to tell your switches they should forward multicasts properly. Multiple tcpdump terminal windows on all relevant interfaces greatly help with debugging.

I found the following pieces of information interesting:

To be able to setup multicast routes a program must connect to the multicast routing socket in the kernel, when that socket is closed, which is done automatically when a UNIX program ends, the kernel cleans up all routes.

This means if you intend to use the multicast routing feature of the kernel, you must use a demon, not a commandline tool.

For static vs. dynamic routing, it says:

The intended purpose of smcroute is to aid in situations where dynamic multicast routing does not work properly. However, a dynamic multicast routing protocol is in nearly all cases the preferred solution. The reason for this is their ability to translate Layer-3 signalling to Layer-2 and vice versa (IGMP or MLD).

Finally, pay close attention to the TTL that is produced by your LAN messenger, see multicast FAQ at the end.