본문 바로가기

취미

crosstool-howto

crosstool-howto

Crosstool is a set of scripts to build and test several versions of gcc and glibc for most architectures supported by glibc. It will even download and patch the original tarballs for you. The resulting script and associated patches, and the latest version of this doc, are available at kegel.com/crosstool.

Crosstool was originally developed for embedded system developers, but is also useful for mainstream developers who simply want their compiles to go fast or who need to build programs that run on older versions of Linux (e.g. Red Hat 6.2), but don't want to develop on those ancient systems.

It includes minimal patches for gcc and glibc needed to build a few combinations of (alpha, arm, i686, ia64, mips, powerpc, powerpc64, sh4, sparc, sparc64, s390, x86_64) x (gcc-2.95.3 ... gcc-4.0.0) x (glibc-2.1.3 ... glibc-2.3.5).

It also supports building toolchains that target Cygwin; see demo-cygwin.sh.

Crosstool is a portable shell script. You can use it to build linux-targeted compilers that run on Linux, Mac OS X, Solaris, and Cygwin. It includes support for creating hetrogenous build clusters; it lets you use virtually every computer in the building, regardless of operating system or CPU type, to speed up your Linux compiles.

Contents

Quick Start

Download and unpack. For instance:
wget http://kegel.com/crosstool/crosstool-0.38.tar.gztar -xzvf crosstool-0.38.tar.gzcd crosstool-0.38
Then look at the demo scripts; there's one for each supported CPU type. For instance, demo-i686.sh is an example of how to build a toolchain that targets the i686 processor. It sets three important variables:
TARBALLS_DIR=$HOME/downloads        # where it will save source tarballsRESULT_TOP=/opt/crosstool           # where it will install the toolsGCC_LANGUAGES="c,c++,java,f77"      # which languages it will make compilers for
It then builds gcc-3.4.0 and glibc-2.3.2 for i686 with the line
 eval `cat i686.dat gcc-3.4.0-glibc-2.3.2.dat`  sh all.sh --notest 
Edit the script if you want to change any of these settings or versions. Then (as root) create the directory /opt/crosstool and make it writable by you, and finally (as yourself) run the demo script, e.g.
sudo mkdir /opt/crosstoolsudo chown $USER /opt/crosstoolsh demo-i686.sh
When it finishes, you can run the new compiler as /opt/crosstool/gcc-3.4.0-glibc-2.3.2/i686-unknown-linux-gnu/bin/i686-unknown-linux-gnu-gcc. (You might want to put /opt/crosstool/gcc-3.4.0-glibc-2.3.2/i686-unknown-linux-gnu/bin on your PATH; then you can run the compiler as i686-unknown-linux-gnu-gcc.)

Static Builds

If for some reason you want the resulting toolchain binaries to be statically linked, set the following environment variables before running crosstool.sh (or all.sh):
BINUTILS_EXTRA_CONFIG="LDFLAGS=-all-static"GCC_EXTRA_CONFIG="LDFLAGS=-static"

Building RPMs

Building RPMs of the compiler used to be done by the same shell script, all.sh, with the --buildrpm option, but that was not enough to satisfy real Linux distributions, which require true .src.rpm's. So now the procedure to build an RPM is to first build the .src.rpm, then build the .rpm from that, using rpmbuild in the traditional way.

A script demonstrating how to build .src.rpm's is buildsrpms.sh. A script demonstrating how to build both .src.rpm's and .rpm's is buildrpms.sh. Both of these are only examples; I use them myself to build a specific set of toolchains.

Each .src.rpm generated by buildsrpms.sh builds toolchains for all supported CPUs, where 'supported' means 'the CPU is listed in the buildlogs directory as having successfully built a working toolchain'. This greatly cuts down on the number of .src.rpm's needed. To build for just e.g. i686, run rpmbuild with options "--without all --with i686".

Faster builds with distcc

You can use distcc to distribute compilation across multiple computers for faster compiles. The distcc that comes with your version of Linux will work fine for most cases. However, it may be more convenient to instead use the script 'mkdistcc.sh' included with crosstool to install a crosstool-specific distcc (partly because that's what the mkdistcclinks.sh script assumes, and partly because it includes a patch that improves support for large hetrogenous clusters; see below).

To install distcc/distccd from source, run

RESULT_TOP=/opt/crosstool \TARBALLS_DIR=$HOME/downloads \sh mkdistcc.shcd /opt/crosstoolsh common/bin/mkdistcclinks.sh
To set up distccd as a service, run
sudo sh /opt/crosstool/common/bin/install-distccd.sh

Regardless of how you installed crosstool-distcc, you then need to edit /opt/crosstool/common/etc/hosts and append the hostnames of all the computers running your distccd.

You can then run the distributed compiler as /opt/crosstool/gcc-3.3.3-glibc-2.3.2/i686-unknown-linux-gnu/distributed/bin/i686-unknown-linux-gnu-gcc. (You might want to put /opt/crosstool/gcc-3.3.3-glibc-2.3.2/i686-unknown-linux-gnu/distributed/bin on your PATH; then you can run the distributed compiler as i686-unknown-linux-gnu-gcc.)

To get any speed benefit, you'll need to run several compiles in parallel. See e.g. make's -j option. Also note that only simple compiling with the -c option, not linking, is sped up, and that only C and C++ compiles are distributed (fortran and java compiles are not distributed, sorry).

Getting the best performance out of distcc is a fine art. See e.g. Benjamin Meyer's page "Distcc optimizations and how to compile kdelibs from scratch in six minutes".

You can monitor your distcc jobs by running

/opt/crosstool/common/bin/distccmon-test 5
This will display a description of your active remote jobs once every five seconds.

Hetrogenous distcc clusters

Here's a scenario sometimes useful at large universities or companies, where one often has a central file server which all clients access. Crosstool is built for multiple versions of gcc and glibc, once for kind of workstation in the cluster, and installed in e.g. /shared/crosstool/`config.guess`. Compilers are invoked via absolute paths so toolchains with different versions of glibc can be distinguished. The absolute path is built using config.guess so, regardless of which kind of workstation the developer is on, he or she can invoke a compiler that can run on their workstation.

For example, when building a C program for modern x86 linux, developers set

CC=/shared/crosstool/`config.guess`/gcc-3.3.3-glibc-2.3.2/i686-unknown-linux-gnu/distributed/bin/i686-unknown-linux-gnu-gcc
And when building a C program for old Red Hat Linux 6.2 x86, developers might set
CC=/shared/crosstool/`config.guess`/gcc-3.3.3-glibc-2.1.3/i686-unknown-linux-gnu/distributed/bin/i686-unknown-linux-gnu-gcc

A tricky part of this scenario is that the distcc server needs to be able to handle absolute paths for *other* architectures, possibly installed at a different location. The patch patches/distcc-2.14/distcc-stringmap.patch, applied by mkdistcc.sh, adds a feature to the distccd server to read a $prefix/etc/distcc/apps file containing absolute paths to all known compilers, and to ignore all but the last N path components when locating the compiler to satisfy received compile requests. The distccd startup scripts created by crosstool's install-distccd.sh turn on that feature.

gcc-3.4.0's precompiled headers and profile-driven optimization features require lockstep synchronization, so they probably work only if the client and the server are the same CPU type and operating system. (And using pch with distcc may require apple's -fpch-preprocess patch; see http://gcc.gnu.org/ml/gcc/2003-03/msg00369.html)

Canadian Cross Builds

Those long, strange names of the form "i686-unknown-linux-gnu" are called GNU configuration names. The GNU build and test scripts use these extensively. When you run the demo-$CPU.sh script, it sources the $CPU.dat file, which sets a variable TARGET containing the GNU configuration name for the target CPU. For instance, i686.dat contains the line
TARGET=i686-unknown-linux-gnu
In the general case, there can be three machine types: the build machine which builds the compilers, the host machine where the compilers will run, and the target machine for which the compilers will generate code.

Building compilers that will run on some other linux system is called a Canadian Cross. It's useful if, say, you're putting together a hetrogenous build cluster consisting of 32 bit and 64 bit workstations, and you want to run natively compiled compilers on each.

To do a Canadian Cross build with crosstool, you have to run it three times:

  1. once to build a toolchain that runs on the build system and generates code for the host system
  2. once to build a toolchain that runs on the build system and generates code for the target system
  3. once to build a toolchain that runs on the host system and generates code for the target system
The last two runs are identical except that on the last run, you have to set a few environment variables:
  • GCC_HOST should be the GNU configuration name (e.g. powerpc-750-linux-gnu) of the host machine
  • PATH must be set to include the bin directory of the host and target compilers that can run on the build machine
  • CC and AR have to be set to a compiler that generates code that runs on the host, and a version of ar that can handle archives of code for the host.
demo-canadian.sh (when I finish it) will demonstrate how to get this working.

If you want the use resulting toolchain as a native toolchain, i.e. if you want it to search /lib and /usr/lib, you'll probably need to edit its specs file to set the cross_compiler parameter to 0; see this thread in the crossgcc mailing list.

Scripts

  • crosstool.sh: Compiles gcc and glibc. This is the most important file, and it can be used by itself, without any other files from the tarball, if you are so inclined.
  • getandpatch.sh: Download, unpack, and patch the binutils, linux, gcc, and glibc source tarballs.
  • crosstest.sh: Run the gcc and glibc regression tests remotely.
  • ptx.sh: Build userland apps (e.g. /bin/sh) using ptxdist.
  • testhello.sh: Verifies trivial programs can be built with the new compiler
  • all.sh: Invoke all the above scripts. Supposedly more convenient that running them individually. (See below.)
  • mkdistcc.sh: A script to download, build, and patch distcc.
  • mkdistcclinks.sh: A script to create masquerade directories for each installed compiler to make using distcc easier.
  • demo-CPU.sh: One demo script for each CPU type that sets environment variables for that CPU, then runs all.sh and mkdistcc.sh.
  • demo.sh: Big demo script that runs all the little demo-CPU.sh scripts.
  • clean.sh: remove junk files; used by maintainer before creating tarballs
  • mkjail.sh: create the files needed for a chroot jail (useful when doing regression testing of glibc)
  • testjail.sh: test a remote chroot jail

Data Files

  • CPU.dat: One file for each CPU type; sets GNU target name
  • gcc-VERSION-glibc-VERSION.dat: One file for each supported combination of gcc and glibc; sets binutils, gcc, and glibc versions and options
  • patches/PROGRAM/*.patch: the patches I needed for each version of each program. The patches for each tool are stored in a subdirectory of patches/ named after the tool (e.g. patches/gcc-3.3). Each patch starts with comments about what it's for, and has links to any associated discussion. This is a small but hopefully high quality and maintainable patch repository; newer versions of ptxdist use a mirror of this repository.
  • summaries/*: example outputs from old crosstest.sh runs

Notes

If you want to build gcc-3.3 or later, you'll need a recent gcc (3.2 or later) on your workstation to build it with.

The scripts are fairly generic. You may need to tweak the parameters of the script to match your exact CPU type, or add a few patches needed to the patches/* directories, and run the build script again, if your tests indicate that programs built with the new compiler have problems.

In particular, if your CPU lacks an FPU, you might need to tell glibc that by setting before running all.sh. For example, see powerpc-405.dat, which sets

GLIBC_EXTRA_CONFIG="--without-fp"

Once you trust the toolchain can build and run simple statically linked 'hello, world' programs (see e.g. testhello.sh), test it with real applications.

If you use these scripts to build a toolchain, please send a note to the crossgcc mailing list indicating which platform you used it on, and how well it worked for you. Please be sure to mention which release of the crosstool scripts you used.

If you add support for a new CPU type, please send your changes to the crossgcc mailing list so they can be incorporated in a future release.

Testing

If you're targeting i686-linux, and are using a released version of gcc and glibc, you probably don't need to worry about testing the toolchain.

But if you're using a new or uncommon CPU type, or an unreleased version of gcc or glibc, and want some assurance that you have built a working compiler and C library, you should run the gcc and glibc test suites. See crosstest-howto.html.

all.sh

all.sh invokes the four scripts that download, build, and test the toolchain. It takes four options:
  • --nounpack, which means 'don't run getandbuild.sh'. This is useful for quick reruns or when just testing.
  • --nobuild, which means 'don't run crosstool.sh'. This is useful for when you just want to run regression tests.
  • --builduserland, which means 'run ptx.sh'. This is useful for when you need to build busybox for some reason, e.g. if you want to run the regression tests, but the target's normal shell can't run against the new shared C libraries.
  • --notest, which means 'don't run crosstest.sh'. This is useful for when you don't have a target to test on, or don't want to spend the time to test. See crosstest-howto.html for information about running the test suite.

Build Platform Notes

Linux

Crosstool was developed on Linux, so most of the rough edges have been polished off. However, if running crosstool.sh on Linux fails with an error like
"soinit.c:25: internal compiler error: in named_section_flags, at varasm.c:..."
then you may be running into gcc bug 9552. One workaround is to delete the file gcc-pr-9552-workaround.patch from crosstool/patches/glibc-2.3.2, and rerun. Another is to switch to a different version of binutils (2.14 seems to be the dividing line).

NetBSD

You'll need to install GNU Diffutils or GNU Patch, since BSD's patch utility doesn't accept the --fuzz parameter.

Mac OS X

You need to install gawk, gnu sed, and wget before you can build crosstool on Mac OS X. The way I installed them was using fink, but DarwinPorts would probably do as well.

On some (older?) versions of Mac OS X, you'll need to raise the stack size with the command

ulimit -s 8192
else make may segfault.

If wget or any other fink program segfaults, try 'sudo /sw/var/lib/fink/prebound/update-package-prebinding.pl -f' once to get around a hiccup in fink's incremental prelinking of apps. See the fink FAQ.

When using 2.6 kernel headers on systems (like Mac OS X) where gcc doesn't support the -shared flag, you may see the error

gcc: unrecognized option `-shared'ld: Undefined symbols:_mainmake[1]: *** [scripts/kconfig/libkconfig.so] Error 1make: *** [oldconfig] Error 2 
This is a well-known issue (see e.g. Peter Samuelson's post of 7 Nov 2002 Bertrand Marquis a patch that might help on 29 June 2004. It would be nice if someone figured out a patch that could go into the mainline kernel sources to deal with this issue.

Another problem building Linux on Mac OS X is described, together with a Mac OS X specific workaround, by Martin Schaffner's post of 22 May 2004.

Cygwin

Crosstool, and probably gcc and glibc's configure scripts, assume that directory names do not contain any spaces. This is often violated on Windows. Please take care to not use directory names with spaces in them when running crosstool. It might work, but if it doesn't, you've been warned. (Same goes for Mac OS X.)

This isn't really a crosstool problem, but configuring linux-2.6 on cygwin may fail with the error

$ make menuconfig HOSTCC  scripts/basic/fixdep HOSTCC  scripts/basic/split-include HOSTCC  scripts/basic/docproc HOSTLD  scripts/kconfig/mconfcollect2: ld terminated with signal 11 [Segmentation fault], core dumped
A possible fix described e.g. by Bertrand Marquis' post of 29 June 2004 is to patch linux-2.6/scripts/kconfig/Makefile to just use libkconfig.o rather than first making a .so.

Cygwin-1.5.9-1 had a bug that prevents it from building glibc. You can work around this by updating to Cygwin-1.5.10-2.

Solaris Build Considerations

Crosstool ought to build on Solaris as long as it has been updated with the appropriate set of GNU tools, but this has not been tested.

binutils-2.15 may require this patch to work on Solaris, else it is said to segfault when building the Linux kernel.

glibc documents which GNU tools it requires in its INSTALL file. The list is roughly: make 3.79, GCC 3.2, binutils 2.13, texinfo 3.12f, awk 3.0, sed 3.02, or newer. gcc documents a few other requirements in gcc.gnu.org/install/specific.html, which says that gcc won't build at all with the default solaris shell, and you're supposed to work around it like this:

% CONFIG_SHELL=/bin/ksh% export CONFIG_SHELL
before running crosstool.

That page also advises that you may need to install some Solaris patches, and install gcc-3.2.3 or newer before trying to build newer versions of gcc.

Troubleshooting

Download Problems

all.sh calls getandpatch.sh, which looks in the directory specified by TARBALLS_DIR for source tarballs, and downloads them to there if not found.

If the download hangs, and you need to use a proxy, try telling wget about your proxy before running all.sh by doing

$ export http_proxy=<proxy_host>:<port>
If the download still hangs, download the tarball that's causing the hang manually to the directory specified by TARBALLS_DIR.

Current Issues

  • all.sh's --builduserland seems to be broken at the moment.
  • NPTL is not yet supported.
  • These scripts, unlike Bill Gatliff's original crossgcc scripts, don't support bare metal newlib targets. They should, but I needed to focus on targeting Linux first. See contrib/newlib for a user-contributed fix.
  • These scripts don't support uClibc yet, but see below.
  • The hppa target is not yet supported.
  • glibc-2.3.2 doesn't build for cris; looks like the maintainer needs to create a sysdep-cancel.h for cris. I no longer try cris, since it seems to be unmaintained.
See the ChangeLog for more issues.

Build Status

The script regtest-run.sh tries to build a number of combinations of gcc, glibc, and CPU, and saves the last hundred or so lines of the build log in the file buildlogs$cpu-$gcc-$glibc.log.txt, along with a summary of the build in the file buildlogs$cpu-$gcc-$glibc.dat.txt. The script regtest-report.sh creates the file buildlogs/$version/index.html summarizing the results. Note: this reflects whether the toolchains could be built, not whether they work! Also note: running regtest-run.sh takes a LOT of cpu power, so it uses ssh to spawn build jobs on multiple machines. Read the script carefully before running it.

Contributed Patches

A few users of the crosstool scripts have submitted patches. I'm saving these in the 'contrib' directory until I have time to test them.
  • testsetup - my scripts which are handy when running test suite remotely
  • newlib, newlib2 - scripts to build newlib toolchains
  • crosstool-uclibc-0.28-rc5.patch to build uclibc toolchains (see below)
  • contrib/xtool-ro.patch lets you run with sources in a read-only directory.

uClibc

The script contrib/demo-uclibc.sh demonstrates how to patch an older version of crosstool to support uclibc. This script, with editing, can be used to build ARM7, PPC405, and maybe MIPS uclibc toolchains. A future version of crosstool will merge the patch.

Links

Patch repositories (handy if you are looking for a fix to a specific problem): There are many good general references for people building crosscompilers. Here is a somewhat outdated set: Here are a few posts about which combinations of versions work: Related Mailing lists (corrections welcome):


Portions copyright 2003, Ixia Communications
Portions copyright 2003, 2004, 2005 Google
Released under the GPL.
Last revision 21 July 2005 by dank@kegel.com