Debian – Communication fails on DSA ethernet ports of Banana Pi R1 (Lamobo R1)

bananapidebianvlan

I don't get TCP/IP communication running on my Banana PI R1 on the switch ports (i.e., not the separate LAN port which shows up as eth0, which is working). It runs "Debian GNU/Linux 9.8", upgraded from an Armbian image with Debian 8 [1].

This is my /etc/network/interfaces file:

allow-hotplug eth0
iface eth0 inet dhcp

# Local loopback
auto lo
iface lo inet loopback

# Router ports
auto lan1
iface lan1 inet static
    address 192.168.12.253
        netmask 255.255.255.0
        network 192.168.12.0
        broadcast 192.168.12.255
        gateway 192.168.12.253
        dns-search greatsoft.local

auto lan2
iface lan2 inet static
        address 192.168.13.253
        netmask 255.255.255.0
        network 192.168.13.0
        broadcast 192.168.13.255
        gateway 192.168.13.253
        dns-search greatsoft.local

auto lan3
iface lan3 inet static
        address 192.168.14.253
        netmask 255.255.255.0
        network 192.168.14.0
        broadcast 192.168.14.255
        gateway 192.168.14.253
        dns-search greatsoft.local

auto lan4
iface lan4 inet static
        address 192.168.15.253
        netmask 255.255.255.0
        network 192.168.15.0
        broadcast 192.168.15.255
        gateway 192.168.15.253
        dns-search greatsoft.local

After boot, ifconfig reports these lines (lo0/wlan0 omitted):

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.113  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::47:aff:fec1:f535  prefixlen 64  scopeid 0x20<link>
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 4581  bytes 816911 (797.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2392  bytes 294521 (287.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 49  

lan1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.12.253  netmask 255.255.255.0  broadcast 192.168.12.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.13.253  netmask 255.255.255.0  broadcast 192.168.13.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan3: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.14.253  netmask 255.255.255.0  broadcast 192.168.14.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.15.253  netmask 255.255.255.0  broadcast 192.168.15.255
        inet6 fe80::47:aff:fec1:f535  prefixlen 64  scopeid 0x20<link>
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 1070 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The output of "ip -d link show" is (lo0/wlan0 omitted):

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
3: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
4: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
5: lan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
6: wan@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
7: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 

For example, if I connect a cable on the lan 4 port and give the connected system an IP address of 192.168.15.250 with netmask 255.255.255.0, ping from Banana PI says:

From 192.168.15.253 icmp_seq=1 Destination Host Unreachable

arp -n shows

Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.15.250                   (incomplete)                              lan4

It is unclear to me why communication fails. I guess there must be anything missing. Does anybody have a hint for me?

[1] https://www.armbian.com/lamobo-r1/

Best Answer

Per default in ArmBian and also in the former Bananian, the 4 ports besides the WAN port are bridged; they are only presented as a single IP interface, being the second interface the WAN port. That is why your IP OS level configuration was not working. Beware in the R1, the ports/chipset at boot time boots as a 5 port hub because the board lacks one resistor connected to the BCM53125 for that not to happen. Worse yet, if Linux kernel does not boot, it stays as a 5 port hub.

The BCM53125 switch chipset must be programmed to turn each port into an individual VLAN, not bridged with any other port.

Beware of the numbering of ports. At least in the old kernel config switching, have not tested the new DSA one, they were:

|2|1|0|4| |3|

The following configuration should be useful for setting up the switch VLAN, with each of the 5 ports as an independent port/VLAN.

When the boot is finished:

eth.10 (VLAN 10) will be your former eth0, 192.168.1.x interface
eth.12 (VLAN 12) will be your former lan1, 192.168.12.x interface
eth.13 (VLAN 13) will be your former lan2, 192.168.13.x interface
eth.14 (VLAN 14) will be your former lan3, 192.168.14.x interface
eth.15 (VLAN 15) will be your former lan4, 192.168.15.x interface

Beware those are internal VLANs to the chipset. The Broadcom BCM53125 does not support 802.1Q. So thus, the use of the untagged directive, in the following script.

Create the file at: /etc/network/if-pre-up.d/dsa

#!/bin/bash

# to avoid execute more than once
[ "$IFACE" == "eth0" ] || exit 0

ip link set eth0 up

# ** create the native VLAN **    
ip link add link eth0 name eth0.10 type vlan id 10
ip link add link eth0 name eth0.12 type vlan id 12
ip link add link eth0 name eth0.13 type vlan id 13
ip link add link eth0 name eth0.14 type vlan id 14
ip link add link eth0 name eth0.15 type vlan id 15

# ** ALLOCATION VLAN
bridge vlan add vid 10 dev wan pvid untagged
bridge vlan add vid 12 dev lan1 pvid untagged
bridge vlan add vid 13 dev lan2 pvid untagged
bridge vlan add vid 14 dev lan3 pvid untagged
bridge vlan add vid 15 dev lan4 pvid untagged




# ** Delete VLAN default 1 **
bridge vlan del vid 1 dev wan

bridge vlan del vid 1 dev lan1
bridge vlan del vid 1 dev lan2
bridge vlan del vid 1 dev lan3
bridge vlan del vid 1 dev lan4
bridge vlan del vid 1 dev eth0.10
bridge vlan del vid 1 dev eth0.12
bridge vlan del vid 1 dev eth0.13
bridge vlan del vid 1 dev eth0.14
bridge vlan del vid 1 dev eth0.15

# ** ACTIVATE /Start **
ip link set up wan

ip link set eth0.10 up
ip link set eth0.12 up
ip link set eth0.13 up
ip link set eth0.14 up
ip link set eth0.15 up

ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
ip link set lan4 up

Make the file executable: chmod 0777 /etc/network/if-pre-up.d/dsa

Checking it all after boot ip -d link show eth0.10 ip -d link show eth0.11 ....

bridge vlan show bridge link show

Remove all rights from other VLAN files, so it is read-only: chmod 0444 /etc/network/if-post-down.d/vlan chmod 0444 /etc/network/if-pre-up.d/vlan

Script untested and adapted from the guide: BPi-R1 - Distributed Switch Architecture 2017

In the linked guide, there is also a script for turning off the switch.

Disclaimer: I won't claim the script works at first try, but should be a good start. I will be testing it in mine R1 in 6-7 weeks time too, have not the time before that.

PS: has your R1 random crashes every other day? Mine had mysterious read-only filesystem situations bi-weekly.

I tracked down the cause, and solved it cutting out the realtek wifi chipset.

Someone later on, doing electrical measurements, managed to track down that occasionally the realtek chipset overloads completely the R1, causing instability, even when it is not being used, confirming my empirical findings.

PPS: When I bought my R1, I tried to make it an AP. A lost cause. See related: Wi-Fi problems using ASUS USB-N13 adapter (realtek)

See also, related https://electronics.stackexchange.com/questions/236530/dht21-am2301-sensor-not-measuring-humidity/236751

Related Question