My Adventure on cross-compilation to arm musl static binary

My adventure starts simply with a request from one of my friends. They were running some Python scripts on a smart network switch to monitor the traffic. However, it was a very limiting environment as only the packages curated by the company were available for use. Although they were given full control of the device, unable to being run any code or software did harm them a lot. For instance, we wanted to run a monitoring code for Phidgets22. Unfortunately, the libraries for this hadn't been vetted by the company so they were unable to use it. After hearing about their situation, I thought of cross-compilation.

I have used C/C++ for some time in the past and was well aware of the toolchains involved. Also, I am aware of cross-compilation which I use quite often in languages like Golang and dotnet. However, I had never got to cross-compiling on the grandfather language yet. So I simply suggested to my friend that he try C and leave me with the cross-compilation step.

The first step of having a C code that interfaces with Phidget22 to get sensor data was actually the easiest part and it took us barely an hour to get working on our regular laptops. After this came the monumental task of cross-compiling.

We started with figuring out the architecture of the Network Switch. I was aware that the switch used some sort of ARM CPU. For me as a starting step and a method to test the architecture of the CPU, I thought of starting by compiling the basic helloworld.c program.

#include <stdio.h>
int main() {
    printf("Hello, World!\n");
}

The first step to cross-compiling was setting up the toolchain in the system. I could have searched for the toolchain on the package manager for my distro but I was also aware that building binary for C is not just compiling. We also have to deal with linking the binary with the correct tool and the correct library compiled for that architecture. Based on all of this and the fact that I am quite familiar with docker and container for such isolation, I searched for a docker image with my requirements and found dockcross. Given that the project has quite recent activities, and my adventure this time was just to get the cross-compilation I decided to use their provided images. Perhaps configuring my own image would be a possible future adventure. After consulting the docs and understanding my requirements I choose to compile using armhf musl. The reason for choosing armhf was simple as I heard we previously did try to compile using a Raspberry Pi which was running aarch64 and that didn't seem to work. As for why musl, from my previous experience, I knew gnu and its networking libraries made use of dynamic dopen calls in its code which meant that using it for static linking was not the best idea. Even most modern languages like Golang seem to prefer musl when creating a static binary. So, I was off to work with it.

$ docker run --rm dockcross/linux-armv7l-musl > ./dockcross-linux-armv7l-musl
$ chmod u+x ./dockcross-linux-armv7l-musl
$ ./dockcross-linux-armv7l-musl bash

The first step was to compile the hello world program, I tried creating both the dynamic and statically linked binary as,

$ armv7l-linux-musleabihf-gcc helloworld.c -o helloworld
$ armv7l-linux-musleabihf-gc -static helloworld.c -o helloworld-static

Now running file on generated binaries, we see the following output.

$ file helloworld
helloworld: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, not stripped
$ file helloworld-static
helloworld-static: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), static-pie linked, with debug_info, not stripped

The moment of truth. We transfer the files and run them on the switch.

Starting with the dynamically linked binary.

$ ./helloworld
error: no such file or directory

No luck on this, then the statically linked binary, and .....

$ ./helloworld-static
Hello, World!

Whew, at least this worked. So off to compiling the actual Phidget22 code.

Since I had already compiled the Phidget22 code previously for running on our machines, I knew that it had a dependency on libphidget22 library. While we were able to find an armhf build of the library, it wasn't built for musl. So I had to start by building libphidget22 for musl.

The build tool used by libphidget22 was autoconf. So on obtaining the source code, I got started with,

$ autoreconf --force --install

This ensured that the autoconf was properly setup with the build environment. Then after a quick look into the options available using ./configure --help I was off to compilation with,

$ export TOOL=armv7l-linux-musleabihf
$ export PREFIX=/usr/xcc/armv7l-linux-musleabihf/armv7l-linux-musleabihf/
$ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX

Here the options, --enable-static --disable-shared allow me to generate libraries to be used for static linking. Similarly --host specifies the host system where the final binary will run, which is in contrast to the build system where the binary is compiled. This is always specified by a triplet. The triplet here specifies that we are compiling for armv7 architecture running linux system, with musl abi. There is also hf tag that is arm specific. The triplet for the build system is not specified here but would have been, x86_64-linux-gnu . As my system is running a 64-bit x86 processor with GNULinux.

Unfortunately, not everything was sunshine and rainbow as the compilation failed with an error message,

configure: error: Missing libusb!

Turns out Phidget22 is also capable of interfacing using USB, and while that is a feature that isn't used by us it wasn't possible to disable it during the compilation process using autoconf's defines. So, we had to compile libusb too. In hindsight, I could have visited Alpine Linux's package repository and got the library from there. But at the time, I was at that time too invested in cross-compiling by myself that I didn't think ahead.

After getting the source code for libusb I was happy to find that they also used autoconf as a build system and I could employ similar techniques. So once again I restarted the process with,

$ autoreconf --force --install 
$ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX

..... and a failure once again.

configure: error: udev support requested byt libudev header not installed

Ok, so we are missing libudev now? Another headhunt later I found out that libudev has already been merged with systemd and that if I wanted to compile the missing libraries I might have to build systemd. Now, this is a task too big just for running Phidgets on a network switch. I spent hours trying to figure out ways of only compiling udev or finding the libraries to no avail.

The next day, I went through the options for libusb again. Lo and behold, there, under optional features, what do I find?

Optional Features:
    ....
    --enable-udev     use udev for device enumeration and hotplug support (recommnded) [default=yes]
    ....

well.... turns out I didn't have to use udev after all. All the hours I spent searching for answers could have been better resolved by looking better once again. Sigh

So now armed with this knowledge I built and installed libusb ,

$ ./configure --enable-static --disable-shared --disable-udev --host=$TOOL --prefix=$PREFIX
$ make -j8
$ sudo make install

And ... done! Now I can finally build libphidget22 as,

$ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX
$ make -j8
$ sudo make install

Now, with all the dependencies ready, it was just a matter of running,

$ armv7l-linux-musleabihf-gcc -ggdb3 -Wall -s -static -o phidget_motion_static main.c -lphidget22 -lusb-1.0

and?

$ file phidget_motion_static
phidget_motion_static: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), static-pie linked, stripped

Finally, just what we needed. This entire binary is also only 1.6MB.

Running it on the Switch also gave us the required sensor values. Perfect. With this my adventure is complete.