icecc setup how to

I wrote this article back last February just to document the way I did setup icecc in my home and decided to put it here for easy access. Just ignore the time of compilation for both Qt and webkit (since then, both have changed considerably and I also added another node into the cluster, an Intel i7). For reference, the trick of  combining ccache and icecc is not very well know and this is why I think it is still worthwhile to document it here.

Maybe those are techniques that you are already familiar with, so my advanced apologies. If you don’t have used icecc/ccache before or just want to known a little more about it, let’s proceed.

Ok, so this is the scenario: you are working with a big code base or maybe you need to recompile a specific version of Qt/WebKit. Or it can be another reason, maybe you just want to self compile your own kernel/desktop like real programmers do, right?

Even with today’s multicore/gigabyte based systems, it may take a little longer than you wish to compile it all: http://xkcd.com/303/

So, what to do in this cases? There are some projects that can help you out, let’s examine the case of ccache first.

Ccache (http://ccache.samba.org/) is a tool created by samba team, that project that offers CIFS/SMB protocol implementation over *nix kernels. It works with a simple principle of generating a cache of compiled files and reusing those object files if no changes in the source code were detected. Instalation is quite easy in Linux (sudo apt-get install ccache), and you only need to point your PATH environment variable to ccache.

An alternative is to have a file in your home (e.g. ~/.samba.bashrc) with the following function (and source it in .bashrc):
function sambacache {
export PATH=/usr/lib/ccache:$PATH
}

So that you can enable samba by doing:
adenilson@macmini:~$ which gcc
/usr/bin/gcc
adenilson@macmini:~$ sambacache
adenilson@macmini:~$ which gcc
/usr/lib/ccache/gcc

The first compilation will take the same time as you are used to, but the following ones (after the cache is populated i.e. check ~/.ccache directory) will give huge speed ups. So, as an example, a favorite of mine (http://cellardoor.googlecode.com) takes 4s to compile in a Intel i5@2.3Ghz the fist time but next it will take only 0.48s to recompile after a make clean. Pretty neat, hum?

Ccache adds almost no overhead in the compilation and is a perfect helper if you have just 1 computer to do your compilation jobs.

There is an issue with the way that ccache works. It is easy to see that if your project has many self-generated files by each new compilation, the cache will not be reused. So, if for recompiling just Qt there are not big benefits (it self generates lots of header files at beginning of compilation), for WebKit it brings the time compilation down to just 2 minutes.

The next project is icecc (http://en.opensuse.org/Icecream) and it allows to use several computers in the same network to distribute the compilation jobs. It is the ‘evolution’ of distcc, offers some extra features and has a great monitoring tool called ‘icemon’. Instalation is easy in Linux (sudo apt-get install icecc icecc-monitor), configuration not that easy and the documentation could be better.

The speed ups are almost linear, being the network speed the most probable bottleneck. I’ve used it in my previous job with up to 15 computers connected and it allowed me to do a cold recompile of Qt in less than 8 minutes (bear in mind that those 15 computers were also compiling other projects at same time…).

To make it easier to follow, I will explain my home setup. I have 2 machines:
a) macmini: Intel i5@2.3Ghz running Linux 32 bits (dual boot with OSX Lion).

b) blackbloat: Acer notebook (cheapest one I could find, so portability is not its strongest feature…) Intel i5@2.67Ghz running linux 64 bits (dual boot with Windows 7). IP is 192.168.1.135

What is neat about icecc is the fact that you can have varied nodes in your cluster and still perform compilation jobs distribution. It does that by allowing you to create a ‘rootstrap’ of libraries and compilers to be used, being those sent to the other nodes in your first compilation. To do it, you should run:

adenilson@blackbloat:~$ icecc –build-native
adding file /usr/bin/gcc
adding file /lib/x86_64-linux-gnu/libc.so.6
adding file /lib64/ld-linux-x86-64.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.53-system.20110810.so
adding file /usr/lib/libbfd-2.21.53-system.20110810.so
adding file /lib/x86_64-linux-gnu/libz.so.1
adding file /lib/x86_64-linux-gnu/libdl.so.2
adding file /usr/bin/cc1=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/cc1
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/lib/libgmp.so.10
adding file /usr/bin/cc1plus=/usr/lib/gcc/x86_64-linux-gnu/4.6.1/cc1plus
adding file /usr/lib/gcc/x86_64-linux-gnu/4.6.1/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confpM2h21
creating de07a31507267d47693646853b78125e.tar.gz

adenilson@macmini:~$ icecc –build-native
adding file /usr/bin/gcc
adding file /lib/i386-linux-gnu/libc.so.6
adding file /lib/ld-linux.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.53-system.20110810.so
adding file /usr/lib/libbfd-2.21.53-system.20110810.so
adding file /lib/i386-linux-gnu/libz.so.1
adding file /lib/i386-linux-gnu/libdl.so.2
adding file /usr/bin/cc1=/usr/lib/gcc/i686-linux-gnu/4.6.1/cc1
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/lib/libgmp.so.10
adding file /usr/bin/cc1plus=/usr/lib/gcc/i686-linux-gnu/4.6.1/cc1plus
adding file /usr/lib/gcc/i686-linux-gnu/4.6.1/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confWHgW78
creating 73c127da41faa91bfdbae1faedbb2113.tar.gz

And later point the environment variable ICECC_VERSION to where is the rootstrap tarball. In my case, I added a file in my home (i.e. .icecc.bashrc) and renamed the tarball with the following:

function coolice {
export ICECC_VERSION=/home/adenilson/Desktop/ice_rootstrap.tar.gz
export PATH=/usr/lib/icecc/bin/:$PATH
}

Similar to ccache, icecc requires to add it in the PATH before the standard compiler.
adenilson@macmini:~$ which gcc
/usr/bin/gcc
adenilson@macmini:~$ coolice
adenilson@macmini:~$ which gcc
/usr/lib/icecc/bin//gcc

The final step is to choose which machine will run the scheduler, responsible for distributing the compiler jobs between the nodes. In my case, I picked blackbloat, for making the changes permanent, added in /etc/default/icecc
START_ICECC=”true”
START_ICECC_SCHEDULER=”true”

And all the nodes (including blackbloat, since I doubled it as both a scheduler and a compiler node) should know which network to connect to (I decided to call my network ‘tucks’) thus editing the file /etc/icecc/icecc.conf:

ICECC_NETNAME=”tucks”
ICECC_ALLOW_REMOTE=”yes”
ICECC_SCHEDULER_HOST=”192.168.1.135″

There are other parameters you can tweak. For example, by default icecc will assign a compiler job for each CPU/Core/virtual cpu in your system, so it would run at maximum 4 jobs in each node in the case of an intel i5 (2 cores + hyper thread). You can change that by assigning a value in ICECC_MAX_JOBS filed in icecc.conf. Another tweak is the nice level of each running job process, default is 5 (which is good for not disturbing the normal workflow in a computer, but you can get a slightly better performance by changing it to a lesser value).

To monitor the machines in the cluster, just use: icemon -n tucks (see attached image).

It is important to known if *really* the jobs are being distributed. You can check:
a) If in your machine there are several ‘g++’ named processes (using basically no CPU) and a few ‘cc1plus’ named processes using all the available cpu;
b) If the other node(s) has several ‘cc1plus’ processes;
c) If there is considerable I/O going through the network interface (in my case spikes of up to 2.8MB/s)
d) If icemon shows the jobs moving through the nodes.

I once thought that I’ve configured it all fine and then started ‘make -j30’ with all the jobs running in my machine… needless to say, the machine locked.
🙂

Alright, enough talk, let’s see some numbers… To compile Qt 4.8 from the git repository it takes 34 minutes in macmini, running with ‘make -j6’. By using icecc, and running ‘make -j12’ it compiles in 21 minutes (almost half the time). It is important to remember that there are steps that can not be made parallel (e.g. qmake creating the .pro files, moc running, linking and so on).

For webkit plus Qt5, the numbers are even better: 1h 10m X 19minutes (it is almost 400% faster!). Those 50 minutes *by compilation* can add a lot at end of day.

Finally, it is also possible to combine ccache *with* icecc. To do it, just define your path as: export PATH=/usr/lib/ccache:/usr/lib/icecc/bin/:$PATH. In my desktop, I added the following file in my home:

adenilson@macmini:~/apps/webkit/Webkit$ more ~/.bamf.bashrc
function bamf {
export ICECC_VERSION=/home/adenilson/Desktop/ice_rootstrap.tar.gz
export PATH=/usr/lib/ccache:/usr/lib/icecc/bin/:$PATH
}

Attached you can check how icemon looks like while recompiling Qt 4.8 with 2 different views (starview and gantt view).