Linux – Getting big endian linux build to boot on ARM with u-boot

I’m trying to make a big endian build of a linux distribution for ARM. Since I’m on Gentoo, cross-compiling couldn’t be easier. I’ve already built it all, but then got stuck with getting it/the kernel to boot.

I’m targetting Cubieboard with AllWinner A10 CPU. As a bootloader I use u-boot. Since u-boot doesn’t support big endian ARM, I patched it exactly before passing control to the kernel:

diff -Naur u-boot-2016.01-1/arch/arm/lib/bootm.c u-boot-2016.01-2/arch/arm/lib/bootm.c
--- u-boot-2016.01-1/arch/arm/lib/bootm.c       2016-01-12 15:06:54.000000000 +0100
+++ u-boot-2016.01-2/arch/arm/lib/bootm.c       2017-07-09 14:13:29.675865446 +0200
@@ -315,7 +315,16 @@
                                                          0, machid, r2);
                } else
 #endif
+               {
+                       {
+                               unsigned long v;
+                               __asm volatile ("mrc p15, 0, %0, c1, c0, 0\n\t"
+                                               "orr %0, %0, #(1 << 7)\n\t" /* Switch to bigendian */
+                                               "mcr p15, 0, %0, c1, c0, 0" : "=&r" (v));
+                       }
+
                        kernel_entry(0, machid, r2);
+               }
        }
 #endif
 }

I originally used a little different syntax, but in the end took a known-to-work code from APEX boot loader (its arm-kernel-shim). (Nonetheless, the register is the same as what I had and what I read in ARM documentation.)

Also, since u-boot needs to be little endian, I prepared another toolchain for it—with target arm-linux-gnueabihf. As far as I can tell, u-boot itself boots fine.

The (mainline) kernel is compiled with a toolchain for target armeb-linux-gnueabihf. From the compiled image (arch/arm/boot/Image within kernel source/build tree) I built a bootable image (using mkimage from my u-boot build):

mkimage -C none -A arm -T kernel -n Linux-4.9.9-gentoo -d /usr/armeb-linux-gnueabihf/usr/src/linux/arch/arm/boot/Image -ep 0x48000000 -a 0x48000000 /usr/armeb-linux-gnueabihf/boot/uimage

I took the DTB file also from kernel:

cp /usr/armeb-linux-gnueabihf/usr/src/linux/arch/arm/boot/dts/sun4i-a10-cubieboard.dtb /usr/armeb-linux-gnueabihf/boot/

When I load it all up to µSD card and try to boot, I get the following output on serial console:

U-Boot SPL 2016.01 (Jul 16 2017 - 13:52:00)
DRAM: 1024 MiB
CPU: 1008000000Hz, AXI/AHB/APB: 3/2/2
Trying to boot from MMC


U-Boot 2016.01 (Jul 16 2017 - 13:52:00 +0200) Allwinner Technology

CPU:   Allwinner A10 (SUN4I)
I2C:   ready
DRAM:  1 GiB
MMC:   SUNXI SD/MMC: 0
In:    serial
Out:   serial
Err:   serial
SCSI:  SUNXI SCSI INIT
SATA link 0 timeout.
AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
flags: ncq stag pm led clo only pmp pio slum part ccc apst
Net:   eth0: ethernet@01c0b000
starting USB...
USB0:   USB EHCI 1.00
USB1:   USB OHCI 1.0
USB2:   USB EHCI 1.00
USB3:   USB OHCI 1.0
scanning bus 0 for devices... 1 USB Device(s) found
scanning bus 2 for devices... 1 USB Device(s) found
Hit any key to stop autoboot:  0
=> setenv bootargs console=tty0 console=ttyS0,115200 earlyprintk hdmi.audio=EDID:0 disp.screen0_output_mode=EDID:1280x800p60 root=PARTUUID=AC9D6C6F-01 rootwait panic=10
=> ext2load mmc 0 0x48000000 boot/uimage
5025856 bytes read in 592 ms (8.1 MiB/s)
=> ext2load mmc 0 0x51000000 boot/sun4i-a10-cubieboard.dtb
28542 bytes read in 237 ms (117.2 KiB/s)
=> bootm 0x48000000 - 0x51000000
## Booting kernel from Legacy Image at 48000000 ...
   Image Name:   Linux-4.9.9-gentoo
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    5025792 Bytes = 4.8 MiB
   Load Address: 48000000
   Entry Point:  48000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 51000000
   Booting using the fdt blob at 0x51000000
   Loading Kernel Image ... OK
   Loading Device Tree to 49ff6000, end 49ffff7d ... OK

Starting kernel ...

And it just hangs there like this – no progress, no output, nothing. My question is, how to proceed from here/how to find out what is actually happening? Am I missing something, did I do anything wrong (or didn’t do something)?

Some more things I tried, but without success:

• word-swapping the kernel image (as APEX does) (resulted in undefined instruction on boot),

• using compressed kernel image,

• using legacy FEX file instead of FDT.

Update 2017/07/21: I’ve been partly successful in solving my issue. I took the hint from Tom Rini’s comment and tried packing zImage into the uImage which made the kernel boot. With a custom init program (just a simple Hello World compiled either BE or LE) I confirmed the other Tom Rini’s suspicion: my original kernel was not big endian, but little. To fix this, I added the following two lines to the top of kernel’s .config:

CONFIG_ARCH_SUPPORTS_BIG_ENDIAN=y
CONFIG_CPU_BIG_ENDIAN=y

I sourced my inspiration from a how-to for Nvidia Jetson TK1. I also added the following line to arch/arm/mach-sunxi/Kconfig at the end of A10 section:

select ARCH_SUPPORTS_BIG_ENDIAN

It turned out that this is enough to build and boot a big endian kernel as the kernel itself switches the CPU to big endian mode. However, it does so with setend instruction—its scope is only for the kernel itself, not for userspace (as link in Murray Jensen’s answer explains).

I’m going to try:

• jumping to kernel’s decompressor entry point via exception, or (if I fail)

• patching the kernel so that it spawns processes into big endian settings (although I sort of smell a minefield there…).