We’ve just finished building a Win32 toolchain for building Raspberry PI applications (BTW, if you have not checked Raspberry PI yet, do it, as it’s a pretty amazing gadget for it price). The build process was far away from being straight-forward and slightly different from the way barebone cross-compilers are built, so this post summarizes all problems and gives workarounds for them.
The build
The main difference between building a cross-compiler for a Linux system and a barebone system (such as arm-eabi) is the immense amount of libraries already available on the target system, each of them having include and library files under /usr/include and /usr/lib. Building separate versions of them on Windows would not only be tricky and time-consuming, but would also cause troubles in case our Windows binaries end up slightly different from the ones on the target Linux machine. So we simply import the headers and the libraries from the Linux machine before building the toolchain.
Importing the files
Identifying what to import is easy: just run this command line on your Linux box:
echo | gcc -v -E -
You will get something like that:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/4.6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Debian 4.6.3-8+rpi1' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 4.6.3 (Debian 4.6.3-8+rpi1)
COLLECT_GCC_OPTIONS='-v' '-E' '-march=armv6' '-mfloat-abi=hard' '-mfpu=vfp'
/usr/lib/gcc/arm-linux-gnueabihf/4.6/cc1 -E -quiet -v -imultilib . -imultiarch arm-linux-gnueabihf - -march=armv6 -mfloat-abi=hard -mfpu=vfp
ignoring nonexistent directory "/usr/local/include/arm-linux-gnueabihf"
ignoring nonexistent directory "/usr/lib/gcc/arm-linux-gnueabihf/4.6/../../../../arm-linux-gnueabihf/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include
/usr/local/include
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include-fixed
/usr/include/arm-linux-gnueabihf
/usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "<stdin>"
COMPILER_PATH=/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/
LIBRARY_PATH=/usr/lib/gcc/arm-linux-gnueabihf/4.6/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/../../../arm-linux-gnueabihf/:/usr/lib/gcc/arm-linux-gnueabihf/4.6/../../../:/lib/arm-linux-gnueabihf/:/lib/:/usr/lib/arm-linux-gnueabihf/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-E' '-march=armv6' '-mfloat-abi=hard' '-mfpu=vfp'
The keywords here are #include and LIBRARY_PATH. Most of the directories are empty or do not exist, so on Raspberry PI you end up with 3 key directories to import:
- /usr/include
- /usr/lib
- /lib
Transfer them to your Windows machine and save somewhere preserving the relative paths.
Another important point is to look at the “Configured with” message that the target gcc shows, as we will need to replicate those arguments when building our cross-compiler.
Sysroot vs. Prefix
The usual way of building a barebone cross-compiler is to provide the target and prefix parameters to the configure script and let it decide where to put the includes and libraries:
../gcc-X.Y/configure --prefix=/c/my-gcc-folder/ --target=arm-eabi
This won’t be work with cross-compilers for Linux due to different locations of headers and libraries, so the configuration process is slightly different:
- Go to the usual target directory for your toolchain (e.g. c:\folder\arm-linux-gnueabihf)
- Create a subdirectory there (e.g. called sysroot, the name is arbitrary)
- Copy the imported Linux files to that directory, preserving the paths (e.g. /usr/include becomes c:\folder\arm-linux-gnueabihf\sysroot\usr\include).
- When configuring binutils, GCC and GDB provide an additional argument:
--with-sysroot=/c/folder/arm-linux-gnueabihf/sysroot
That’s not all though. If you are impatient enough to try building your GCC now, you will encounter multiple missing header/library errors when it comes up to building libgcc and other target libraries. In fact, the compiler you’ll get won’t ever find most of the headers. This can be diagnosed by running xgcc.exe the same way we did for Linux gcc:
echo | xgcc -v -E -
You might find some really strange paths there: e.g. c:/folder/sysrootc:/mingw/msys/1.0/include, or some of the Linux library paths (e.g. /usr/lib/arm-linux-gnueabihf) will be missing.
There are two totally different reasons behind this problem:
- Trying to maximize compatibility with Windows, MinGW/MSys silently replaces /usr/include with the MSys include directory. While this is useful when building Windows software, it has one inadvertent side effect: it replaces the NATIVE_SYSTEM_HEADER_DIR definition passed to GCC via command line with the MSys include path! Thus gcc ends up trying to append c:/mingw/msys/1.0/include to your sysroot, as it believes it’s the standard include file location.
- The Debian Linux distro (that Raspberry PI is based on) uses a slightly different include/lib path structure (e.g. /usr/include/<target>) and building GCC without the Debian patches would result in a crippled GCC.
The second one can be easily fixed by using the sources from Debian repositories (run apt-get source gcc-<version> on your Linux machine or apply Debian patches manually). The first one can be resolved by adding the following code to cppdefault.c:
#if defined(__MINGW32__) && defined(TARGET_SYSTEM_ROOT)
#define NATIVE_SYSTEM_HEADER_DIR "/usr/include"
#endif |
#if defined(__MINGW32__) && defined(TARGET_SYSTEM_ROOT)
#define NATIVE_SYSTEM_HEADER_DIR "/usr/include"
#endif
Building target libs
Another surprise will be waiting for you when your GCC build process tries to link libgcc. The immense amount of object files stuffed into just one link command line will exceed the Windows maximum command line length and will cause a strange CreateProcess() error to appear. The following command causes the overflow:
$(subst @multilib_flags@,$(CFLAGS) -B./,$(subst \
@multilib_dir@,$(MULTIDIR),$(subst \
@shlib_objs@,$(objects),$(subst \
@shlib_base_name@,libgcc_s,$(subst \
@shlib_map_file@,$(mapfile),$(subst \
@shlib_slibdir_qual@,$(MULTIOSSUBDIR),$(subst \
@shlib_slibdir@,$(shlib_slibdir),$(SHLIB_LINK))))))))
The following simple hack changes it to use a response file instead:
echo $(objects) > __objects.txt
$(subst @multilib_flags@,$(CFLAGS) -B./,$(subst \
@multilib_dir@,$(MULTIDIR),$(subst \
@shlib_objs@,@__objects.txt,$(subst \
@shlib_base_name@,libgcc_s,$(subst \
@shlib_map_file@,$(mapfile),$(subst \
@shlib_slibdir_qual@,$(MULTIOSSUBDIR),$(subst \
@shlib_slibdir@,$(shlib_slibdir),$(SHLIB_LINK))))))))
Another problem would be related to wrong definition of caddr_t caused by conflicting Windows and Linux definitions and is resolved by adding -DUSED_FOR_TARGET to CFLAGS used for building libgcc.
Debugging with GDB
The sysroot trick is vital if you are planning to debug your apps with gdb running on Windows via gdbserver. Unless your sysroot structure exactly replicates the Linux directory structure and unless gdb is also configured with the –with-sysroot argument, you’ll get the following error message when trying to debug your programs:
warning: Unable to find dynamic linker breakpoint function.
This would mean a failure trying to load symbols for ld-linux-armhf.so.3 rendering most of module-related functionality inoperable. Needless to say, your debugging won’t be very usable afterwards…
The summary
Summarizing all previously said, here’s a checklist for building your Linux cross-compiler on Windows:
- If you are targeting Debian, apply Debian patches to GCC.
- Fix NATIVE_SYSTEM_HEADER_DIR inside cppdefault.c
- Use –with-sysroot when configuring binutils, gcc and gdb
- Import Linux includes and libs before building gcc
- Fix $(objects) in libgcc/Makefile
- Add -DUSED_FOR_TARGET to libgcc CFLAGS
- Test your toolchain with some C++ code (including STL)
- Test debugging with gdbserver to ensure all symbols are usable
If you like it simple…
If you don’t feel like reliving the adventurous enterprise of building your own Windows toolchain for Raspberry PI, you can download our pre-built version.
