A couple of weeks ago I bought a Cubietruck to mess with ARMv7 and the ARM virtualization extension. One of my first goals was to get KVM working and run a guest with bridged network support. Because it took a lot of research, troubleshooting and kernel builds I decided to share this on my blog.
Because I'm a Fedora user this howto is based on Fedora 20 (which is the current stable at the moment).
Things you need:
- Cross compile build host with (in my case) Fedora 20 on it
- Cubietruck
- SD-card with minimal 8GB capacity
- UART TTL USB to COM Cable (usefull to control/follow the boot process and managing the console)
Prepare the build host
First create the build environment on your Fedora build host by installing these packages:
sudo yum install -y gcc-arm-linux-gnu ncurses-devel \
uboot-tools libusb libusb-devel git make wget
Create the disk image
Create a 2GB disk image with 2 partitions (later on we are gonna resize it to the full capacity of the SD card). The first partition needs to be 100MB minimal and the second contains the rest:
fallocate -l 2G fedora20-ct.img
sudo kpartx -a fedora20-ct.img
sudo fdisk /dev/loop0
n
p
1
2048
+100M
n
p
2
<ENTER>
<ENTER>
w
sudo kpartx -u fedora20-ct.img
Create the filesystems (the first partition must be ext2 which is supported by u-boot to boot from):
sudo mkfs.ext2 /dev/mapper/loop0p1
sudo mkfs.ext4 /dev/mapper/loop0p2
Mount the filesystems:
sudo mkdir /mnt/ct-boot
sudo mkdir /mnt/ct-root
sudo mount /dev/mapper/loop0p1 /mnt/ct-boot
sudo mount /dev/mapper/loop0p2 /mnt/ct-root
Note: you can also use a SATA disk for the Fedora rootfs, just mount it on /mnt/ct-root instead and configure it in uEnv.txt which is covered below in this post.
Kernel compilations
Get the latest sunxi-devel branch kernel:
git clone git://github.com/linux-sunxi/linux-sunxi.git -b \
sunxi-devel
Cross-compile the kernel with the Cubietruck host configuration:
cd <kernel sourcedir>
cp <path to>/config.txt-ct-host .config
make ARCH=arm oldconfig
If you want to add extra kernel options or drivers:
make ARCH=arm menuconfig
Build the kernel:
LOADADDR=0x40008000 make ARCH=arm \
CROSS_COMPILE=/usr/bin/arm-linux-gnu- -j4 uImage dtbs
Copy the kernel and dtb file to the boot mount:
sudo cp arch/arm/boot/uImage \
arch/arm/boot/dts/sun7i-a20-cubietruck.dtb /mnt/ct-boot
Note: you can also download the pre-compiled host kernel and dtb file from here:
Cross-compile the kernel with the QEMU guest configuration (this is handy because compiling on the cubietruck isn't very fast):
cd <kernel sourcedir>
cp <path to>/config.txt-ct-guest .config
make ARCH=arm clean
make ARCH=arm oldconfig
If you want to add extra kernel options or drivers:
make ARCH=arm menuconfig
Build the kernel:
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnu- \
-j4 zImage dtbs
Copy the kernel and dtb file to the boot mount:
sudo cp arch/arm/boot/zImage \
arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb /mnt/ct-boot
Notes:
- You can also download the pre-compiled guest kernel and dtb file from here:
- This kernel doesn't boot with SELinux support otherwise the guest will fail to boot (maybe I will fix this later).
- Above kernels are compiled with loadable module support, but they don't contain any modules in the configurations.
Build and install the u-boot bootloader
Get Hans de Goede's patched u-boot which supports ARM HYP mode:
git clone https://github.com/jwrdegoede/u-boot-sunxi.git -b \
sunxi-test
Cross-compile u-boot:
cd <u-boot source>
make cubietruck_config
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnu- -j4
Install the u-boot bootloader:
sudo dd if=u-boot-sunxi-with-spl.bin of=/dev/loop0 bs=1024 seek=8
Create the file "/mnt/ct-boot/uEnv.txt" with the following contents:
loglevel=9
# uncomment this if you want to use SD storage:
root=/dev/mmcblk0p2
# uncomment this if you want to use SATA storage:
#root=/dev/sda2
console=ttyS0,115200
extraargs=rw rootwait panic=10 rootfstype=ext4 rootflags=discard
Create the file "/mnt/ct-boot/boot.cmd" with the following contents:
setenv bootargs console=${console} root=${root} loglevel=${loglevel} ${panicarg} ${extraargs}
ext2load mmc 0 0x46000000 uImage
ext2load mmc 0 0x4b000000 sun7i-a20-cubietruck.dtb
env set fdt_high ffffffff
bootm 0x46000000 - 0x4b000000
Build the boot.scr image (this contains above boot parameters which are used by u-boot):
sudo mkimage -C none -A arm -T script -d \
/mnt/ct-boot/boot.cmd /mnt/ct-boot/boot.scr
Copy the Fedora 20 files
Get the Fedora 20 ARM image:
Mount the third Fedora 20 ARM image partition:
sudo kpartx -a Fedora-Minimal-armhfp-20-1-sda.raw
sudo mkdir /mnt/f20-arm
sudo mount /dev/mapper/loop1p3 /mnt/f20-arm
Set the default root password to "fedora" in the image by changing the root line in
"/mnt/f20-arm/etc/shadow" with:
root:$6$TdrjRNbI$8ra4nFKjqj0s0cFBtEcKxugGy5VD6RheEJHEDC8CJakdB2UT2pT3kjH2uHwy9RzjyKHafb92veemXSKRxH.pf/:16198:0:99999:7:::
Note: this is handy when the Fedora initial setup fails due memory errors or if your system boots in emergency mode.
Create the file "/tmp/rsync-excludes" which is used by rsync:
/dev
/proc
/sys
/media
/mnt
/run
/tmp
Rsync the files:
cd /mnt/f20-arm
sudo rsync -aH --exclude-from=/tmp/rsync-excludes * /mnt/ct-root
rm -f /tmp/rsync-excludes
Clear the fstab and mount the first partition on /boot2:
sudo mkdir /mnt/ct-root/boot2
sudo sh -c "echo '/dev/mmcblk0p1 /boot2 ext2 defaults 1 1' >\ /mnt/ct-root/etc/fstab"
Make sure to relabel the SELinux file contexts on the filesystems during boot. Otherwise you can't login:
sudo touch /mnt/ct-root/.autorelabel
Unmount all partitions and remove the loop devices:
cd <path to images>
sudo umount /mnt/f20-arm
sudo kpartx -d Fedora-Minimal-armhfp-20-1-sda.raw
sudo umount /mnt/ct-root
sudo umount /mnt/ct-boot
sudo kpartx -d fedora20-ct.img
Write the image file to your SD card using dd:
dd if=<path to images>/fedora20-ct.img of=/dev/mmcblk0
Boot your Cubietruck
Put the SD-card in your Cubietruck and power it on.
The kernel only shows output on the serial interface so use the UART TTL USB to COM cable to access the console.
You can use screen for example to connect to the serial console:
sudo screen /dev/ttyUSB0 11520
If HYP mode is activated you should see in the kernel messages:
Brought up 2 CPUs
SMP: Total of 2 processors activated.
CPU: All CPU(s) started in HYP mode.
CPU: Virtualization extensions available.
Also /dev/kvm should exist. If this is not the case, see the FAQ below.
If you get the console login prompt with a python memory error above the initial setup program failed. Follow these steps to setup you system manually:
Login as root with the password "fedora" (which you set in one of the previous steps above) and create a user, sudo rights etc.
Disable the initial setup:
sudo systemctl disable initial-setup-text
Set your timezone:
sudo rm -f /etc/localtime
sudo ln -s /usr/share/zoneinfo/<area directory>/<location file> /etc/localtime
Setup the Fedora host to your needs
Set the hostname:
sudo sh -c "echo 'cubietruck.example.org' > /etc/hostname"
Configure the network and a bridge for the guest
Stop and disable NetworkManager (this is because NetworkManager doesn't support bridge interfaces at the moment):
sudo systemctl stop NetworkManager
sudo systemctl disable NetworkManager
Enable network:
sudo systemctl enable network
Configure the ethernet interface by adding the following content to "/etc/sysconfig/network-scripts/ifcfg-eth0":
DEVICE=eth0
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
Configure the bridge interface by adding the following content to "/etc/sysconfig/network-scripts/ifcfg-br0":
DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
NM_CONTROLLED=no
Note: this configures the use of dhcp, you can also configure a static ip address if you like.
Start the network:
sudo systemctl start network
Disable netfilter on the bridge by adding the following content to "/etc/sysctl.d/bridge.conf":
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
Activate the sysctl parameters:
sudo sysctl --system
Resize the second partition to the full size of the SD card:
sudo fdisk /dev/mmcblk0
d
2
n
p
2
<ENTER>
<ENTER>
w
Reboot the system to make the new partition visible.
Resize the rootfs to the full size of the SD card:
sudo resize2fs /dev/mmcblk0p2
Build the latest QEMU
Install the required rpm's to build and run the latest QEMU:
sudo yum install -y gcc cpp gcc-c++ pixman-devel libfdt-devel \
git zlib-devel glib2 glib2-devel uml_utilities
Build and install QEMU (I installed QEMU in /kvm because I don't want to mix it with the Fedora RPM version):
git clone git://git.qemu-project.org/qemu.git
cd qemu
./configure --prefix=/kvm --target-list=arm-softmmu \
--disable-xen --enable-kvm
make
sudo make install
Note: The Fedora 20 version of QEMU doesn't support the Cortex A7 CPU. That's why we have to build the latest version.
Create a KVM guest
Create the /kvm/images directory:
sudo mkdir -p /kvm/images
Copy the Fedora-Minimal-armhfp-20-1-sda.raw image you downloaded and extracted earlier to the Cubietruck and place it in /kvm/images.
Place the following script in the same directory as the Fedora image file and make it executable (for example: "/kvm/bin/start-guest.sh"):
#!/bin/bash
export QEMU_AUDIO_DRV=none
bridge=br0
tap=tap0
# create tap0 interface
tunctl > /dev/null 2>&1
sleep 1
# bridge tap0 with br0
brctl addif $bridge $tap > /dev/null 2>&1
# bring the tap0 link up
ip link set $tap up > /dev/null 2>&1
echo "Guest is starting in the background.."
echo "Use "telnet localhost 1234" to connect to the console."
# start the guest with the console redirected to telnet
/kvm/bin/qemu-system-arm -enable-kvm -M vexpress-a15 -smp 2 \
-cpu host -m 1024 -kernel /boot2/zImage \
-dtb /boot2/vexpress-v2p-ca15-tc1.dtb -drive if=none,\
file=/kvm/images/Fedora-Minimal-armhfp-20-1-sda.raw,\
cache=writeback,id=foo -device virtio-blk-device,drive=foo \
-serial telnet:localhost:1234,server,nowait \
-append "rw root=/dev/vda3 rootwait physmap.enabled=0 \
console=ttyAMA0,115200n8" -net nic,vlan=0 \
-net tap,vlan=0,ifname=$tap,script=no,downscript=no \
> /dev/null 2>&1
echo "Guest is powered off. Cleaning up the tap interface."
# bring the tap0 link down
ip link set $tap down > /dev/null 2>&1
# remove tap0 from the bridge
brctl delif $bridge $tap > /dev/null 2>&1
# remove the tap0 interface
tunctl -d $tap > /dev/null 2>&1
Make it executable:
sudo chmod +x /kvm/bin/start-guest.sh
Start the KVM guest in the background:
sudo /kvm/bin/start-guest.sh &
Connect to the guest console:
telnet localhost 1234
If you get the console login prompt with a python memory error above the initial setup program failed. Follow these steps to setup you system manually:
Login as root with the password "fedora" (which you set in one of the previous steps above) and create a user, sudo rights etc.
Disable the initial setup:
sudo systemctl disable initial-setup-text
Set your timezone:
sudo rm -f /etc/localtime
sudo ln -s /usr/share/zoneinfo/<area directory>/<location file> \ /etc/localtime
Setup the Fedora guest to your needs
Set the hostname:
sudo sh -c "echo 'cubietruck-guest.example.org' > /etc/hostname"
Configure the network
Show all connections and get the uuid:
sudo nmcli con
Delete the "Wired" connection:
sudo nmcli con del <uuid>
Add the eth0 connection:
dhcp:
sudo nmcli con add con-name eth0 ifname eth0 type ethernet
static example:
sudo nmcli con add con-name eth0 ifname eth0 type ethernet \
ip4 192.168.0.2/24 gw4 192.168.0.1
sudo nmcli con mod eth0 ipv4.dns "192.168.0.1"
sudo nmcli con down id eth0
sudo nmcli con up id eth0
That's it!
FAQ
If you see that only one CPU is activated in the kernel messages you used the wrong version of the u-boot bootloader:
CPU1: failed to boot: -38
Brought up 1 CPUs
Make sure you downloaded the sunxi-test branch of Hans de Goede's git repo:
git clone https://github.com/jwrdegoede/u-boot-sunxi.git -b \
sunxi-test
Then rebuild the u-boot bootloader and install it on the boot device (/dev/mmcblk0).
If you get the messages following messages during boot:
Timed out waiting for device ...
...
Dependency failed for /boot ...
You probably forgot to clear the fstab.
My guest kernel doesn't boot, the screen stays black.
If this happens your guest kernel configuration isn't properly configured to boot from QEMU. I built serveral kernels based on the host kernel configuration with many added options. They all failed to boot. Eventually I took the vexpress_defconfig from the kernel source and extend it with all the needed options to run a proper Fedora system.
It's also possible that you used the wrong device tree blob for QEMU to setup the hardware. Always use the vexpress-v2p-ca15-tc1.dtb, the other ones don't work with the vexpress-a15 machine option!
Dont buy an expensive SSD drive for speed!
The internal SATA bus isn't very fast. I bought a Kingston ssdnow v300 drive for the cubietruck which should theoretically achieve a read and write speed of 450MB/s. I did some tests which resulted in a 100MB/s read speed and a 42MB/s write speed. This was also confirmed by others.
