Setting up a diskless Debian system for a cluster of ARM's Juno dev boards (ARM64)

Board setup for remote access
The board provides a mean of operating it remotely, somehow as one would expect of a server with a board management card (IPMI, Dell idrac, aso).
The board is indeed equiped with 2 serial lines:
The first serial line gets the preboot command line, then the bootloader command line, and finally the Linux command line.
As a result, once a Linux is booted, and if it happens to hang, we cannot use that serial line to operate a reboot. Fortunately, there is a way to configure the second serial line to take over the preboot command line, which allows to (hardware) reset or reboot the board at anytime.


For this setup, one needs to change the preboot configuration file:config.txt
.
We can access and modify the preboot files using 3 different means:
- connecting the usb-B female of the board to out computer, and mounting the mass storage device.
- connecting the RJ45 configuration port (ethernet) and FTP to the board (needs FTP activation from the preboot command line)
- opening the board case and extracting the microSD card from it's slot, then mounting it on our workstation.
Open config.txt and set MBLOG: UART1
.
While we are here, we can also change the HOSTNAME
(cosmetic).
Debian nfsroot setup
We will setup a Debian nfsroot setup, so first of all, we need to configure a nfs server.
As usual, we Install the nfs-kernel-server Debian package and setup the /etc/exports
file as follows:
/srv/diskless/nfsroot 192.168.1.0/24(rw,no_subtree_check,secure,no_root_squash,async)
The juno board requires a recent Linux kernel (4.2.0 is ok), so we will use Debian unstable or testing. ARM64 is (most likely) a foreign architecture for our workstation (x86_64), so we use debootstrap in 2 stages:
workstatiom$ debootstrap --arch=ARM64 --foreign sid /src/diskless/nfsroot/debian-sid-arm64 http://ftp.fr.debian.org/debian
Then we boot the Linaro Linux provided with the Juno board (or use qemu-arm64) and mount the NFS directory from there. We can then run the second stage:
juno$ mount -t nfs 192.168.1.1:/srv/diskless/nfsroot/debian-sid-arm64 /mnt
juno$ chroot /mnt
juno(chroot)$ debootstrap --second-stage
We can now continue setting up the system within the chroot:
- install the linux kernel, etc
- configure network, etc
In order to enable the nfsroot boot, initramfs must be configured. We edit /etc/initramfs/initramfs.conf
, and set:
MODULES=netboot
NFSROOT=auto
Then run update-initramfs -u
.
I did not succeed getting the juno to boot with systemd, but sysvinit boots just fine.
Fortunately, switching back to sysvinit is quite straight forward with Debian (Thanks Debian !):
juno(chroot)$ apt-get install sysv-rc sysvinit sysvinit-utils
Once finished, our Debian nfsroot system is ready.
Network boot setup
The board ships with UEFI as its boot loader (edk2). While it's possible to boot linux from UEFI, or to chain grub to get in friendlier hands, edk2 PXE seems to be badly broken (tftp issues) (actually I had a lot of issues with the board's UEFI...).
As a result, I found a nice workaround, switching to u-boot at the initial boot loader.
Linaro indeed provides the 2 options,
thanks to them!
We use the juno-latest-busybox-uboot.zip
package. (U-boot support seems actually to be newer than UEFI)
U-boot on juno supports dhcp and tftp boot, all we need. However, we have to change some bits in order to have the network boot triggered automatically upon poweron:
Since the default u-boot boot script is hard-coded in u-boot binary, we need to change it there.
U-boot binary is stored as the SOFTWARE/fip.bin file in the preboot files (See above to access/modify them).
Use hexedit or the like to locate the bootcmd = ...
string, and change it to bootcmd = dhcp ; source 0x90000000 ;
.
End of the hard-core bits.
We now setup isc dhcpd for serving IP and boot filename to the board:
host juno {
next-server 192.168.1.1;
filename "boot.scr";
hardware ethernet 00:02:F7:00:65:22;
fixed-address 192.168.1.10;
option host-name "juno-1";
}
And setup a tftpd server, and setup the remote boot script. Prefer atftpd to tftpd-hpa because the board's u-boot tftp client requires to disable the 'timeout' option (RFC2349): We use the --no-timeout
option of atftpd.
Tftpd will serve kernel, initrd and boot script. Copy kernel and initrd from the nfsroot directory:
workstation$ cp /srv/diskless/debian-sid-arm64/boot/vmlinuz /srv/tftpboot/
workstation$ cp /srv/diskless/debian-sid-arm64/boot/initrd.img /srv/tftpboot/
Convert initrd.img to the u-boot format (not needed for vmlinuz...):
workstation$ mkimage -A arm64 -O linux -T ramdisk -C gzip -a 0x84000000 -e 0x84000000 -d initrd.img -n "initrd.img" u-initrd.img
Finally we create the boot script:
workstation$ cat <<EOF > /srv/tftpboot/boot.script
setenv kernel_name vmlinuz
setenv initrd_name u-initrd.img
setenv fdt_name juno/juno-r1.dtb
tftpboot ${kernel_addr} ${kernel_name}
tftpboot ${initrd_addr} ${initrd_name}
tftpboot ${fdt_addr} ${fdt_name}
setenv bootargs console=ttyAMA0,115200n8 root=/dev/nfs nfsroot=192.168.1.1:/srv/diskless/nfsroot/debian-sid-arm64 ro rootwait
booti ${kernel_addr} ${initrd_addr} ${fdt_addr}
EOF
And convert it to the u-boot format (boot.src file):
workstation$ mkimage -A arm64 -O linux -T script -C gzip -a 0x90000000 -e 0x90000000 -d boot.script -n "arm64-nfsroot" boot.scr
We are done, our juno should now be able to boot our Debian nfsroot system.
Setting up a cluster of diskless Juno
In order to boot a cluster of juno boards with a diskless OS, we need to setup a copy-on-write overlay so that every board has a dedicated root filesystem (/var). (One may consider having a different nfs export for every board, but that's not very elegant.)
For that purpose, we use Linux overlay fs (aufs is not supported with Debian 4.2.0 kernel) and squashfs (as of writing this, overlay fs support for nfs lower layer is broken. See Debian related bug which was closed and reopened in Aug 2015).
We'll describe that setup in a next post.
Enjoy.
References
- http://www.arm.com/products/tools/d...
- http://snapshots.linaro.org/member-...
- http://snapshots.linaro.org/member-...
- http://infocenter.arm.com/help/inde...
- https://www.linaro.org/blog/core-du...
- https://wiki.linaro.org/ARM/Juno/Fi...
- http://dflund.se/~triad/krad/junobo...
- https://www.debian.org/releases/sta...
- https://wiki.debian.org/Arm64Port
- https://wiki.tizen.org/wiki/Tizen_A...