Icecream Scratchbox Howto - Revision history

lizardo: /* Prerequisites */

lizardo — Wed, 10 Feb 2010 12:27:59 GMT

Prerequisites

← Older revision		Revision as of 12:27, 10 February 2010
Line 34:		Line 34:
	export ICECC_VERSION=$icecc_tarball		export ICECC_VERSION=$icecc_tarball
	export PATH=/usr/lib/icecc/bin:$PATH		export PATH=/usr/lib/icecc/bin:$PATH
-	export ~~MAKEFLAGS~~=~~-j12~~	+	export DEB_BUILD_OPTIONS=$DEB_BUILD_OPTIONS,parallel=12
	fi		fi
	unset icecc_tarball procfamily DISTRIB_ID DISTRIB_RELEASE DISTRIB_CODENAME DISTRIB_DESCRIPTION		unset icecc_tarball procfamily DISTRIB_ID DISTRIB_RELEASE DISTRIB_CODENAME DISTRIB_DESCRIPTION
Line 40:		Line 40:

	Note that on other distributions than Ubuntu, you might have to modify this, depending on the contents on /etc/lsb-release.		Note that on other distributions than Ubuntu, you might have to modify this, depending on the contents on /etc/lsb-release.
		+
		+	Also note the "parallel=N" parameter being added to DEB_BUILD_OPTIONS. This enables parallel builds (limited to N parallel jobs) for those packages that support it. Alternatively, if you are doing your own builds by calling "make", you should instead pass the "-jN" parameter to enable parallel builds, e.g. "make -j12".

	Then, create directory $HOME/icecc and run <pre>icecc --build-native</pre> there. Symlink the created tarball to a descriptive name, e.g. icecc-i686-karmic.tar.gz (this name has to match the $icecc_tarball variable in the .bashrc snippet above).		Then, create directory $HOME/icecc and run <pre>icecc --build-native</pre> there. Symlink the created tarball to a descriptive name, e.g. icecc-i686-karmic.tar.gz (this name has to match the $icecc_tarball variable in the .bashrc snippet above).

mairas: /* Using icecc */

mairas — Wed, 10 Feb 2010 07:36:20 GMT

Using icecc

← Older revision		Revision as of 07:36, 10 February 2010
Line 122:		Line 122:
	sbox_compiler="$(sb-conf show --compiler)"		sbox_compiler="$(sb-conf show --compiler)"
	# use the tarball name with the MD5, avoiding conflicting names in peer caches		# use the tarball name with the MD5, avoiding conflicting names in peer caches
-	icecc_tarball="$(readlink -f $HOME/icecc_sbox-$sbox_compiler.tar.gz)"	+	icecc_tarball="$(readlink -f $HOME/icecc/icecc_sbox-$sbox_compiler.tar.gz)"
	if [ -f "$icecc_tarball" ]; then		if [ -f "$icecc_tarball" ]; then
	export ICECC_VERSION="i486:$icecc_tarball"		export ICECC_VERSION="i486:$icecc_tarball"

mairas: added a warning about failing arm compilations

mairas — Tue, 09 Feb 2010 11:50:07 GMT

added a warning about failing arm compilations

← Older revision		Revision as of 11:50, 9 February 2010
Line 2:		Line 2:

	= Icecream + Scratchbox HOWTO =		= Icecream + Scratchbox HOWTO =
		+
		+	'''''NB:''' With some Scratchbox versions, distributed arm target compilations occasionally fail due to gcc/g++ targeting wrong ARM variant. Once a workaround is developed, this warning will disappear.''

	From Icecream web page <ref>http://en.opensuse.org/Icecream</ref>:		From Icecream web page <ref>http://en.opensuse.org/Icecream</ref>:

mairas: IcecreamScratchboxHowto moved to Icecream Scratchbox Howto: Avoid CamelCase

mairas — Tue, 09 Feb 2010 11:38:56 GMT

IcecreamScratchboxHowto moved to Icecream Scratchbox Howto: Avoid CamelCase

← Older revision

Revision as of 11:38, 9 February 2010

mairas: /* Using icecc */

mairas — Tue, 09 Feb 2010 11:34:18 GMT

Using icecc

← Older revision		Revision as of 11:34, 9 February 2010
Line 124:		Line 124:
	export ICECC_VERSION="i486:$icecc_tarball"		export ICECC_VERSION="i486:$icecc_tarball"
	# comment the following line if you don't want to build in parallel by default		# comment the following line if you don't want to build in parallel by default
-	MAKEFLAGS="-j12"	+	export MAKEFLAGS="-j12"
	fi		fi
	unset icecc_tarball sbox_compiler		unset icecc_tarball sbox_compiler

mairas at 09:23, 9 February 2010

mairas — Tue, 09 Feb 2010 09:23:27 GMT

← Older revision		Revision as of 09:23, 9 February 2010
Line 1:		Line 1:
-	''Original instructions written by Anderson Lizardo~~. This version copied verbatim from the INdT wiki~~.''	+	''Original instructions written by Anderson Lizardo.''

	= Icecream + Scratchbox HOWTO =		= Icecream + Scratchbox HOWTO =

192.100.116.142: Added a prerequisites section and some notes in the actual instructions.

192.100.116.142 — Tue, 09 Feb 2010 09:21:57 GMT

Added a prerequisites section and some notes in the actual instructions.

← Older revision		Revision as of 09:21, 9 February 2010
Line 11:		Line 11:
	This document describes how to use icrecream for parallel compilation inside		This document describes how to use icrecream for parallel compilation inside
	Scratchbox.		Scratchbox.
		+
		+	== Prerequisites ==
		+
		+	''Skip this section if you have a working Icecream setup on your network.''
		+
		+	Before attempting to use icecc with Scratchbox, be absolutely certain it runs without any issues on the host itself (and that the processes migrate properly in the network). Basic Icecream installation includes once machine running the scheduler and every participating host running the icecc daemon.
		+
		+	On Ubuntu, a basic Icecream setup can be achieved by installing the package icecc on every participating host. On the scheduler, modify /etc/default/icecc to enable the scheduler daemon. If some host can't find the scheduler, modify /etc/icecc/icecc.conf to explicitly set the scheduler hostname.
		+
		+	Make sure the host sees the scheduler by running icemon (package icecc-monitor on Ubuntu). You can leave icemon running to observe the task migration during compilations.
		+
		+	To enable automatic distributed compilations on the host OS in Ubuntu, add the following snippet to $HOME/.bashrc:
		+
		+	<pre>
		+	# set the ICECC version and enable it by default
		+	. /etc/lsb-release
		+	procfamily=$(uname -m)
		+	icecc_tarball="$(readlink -f $HOME/icecc/icecc-$procfamily-$DISTRIB_CODENAME.tar.gz)"
		+	if [ -f "$icecc_tarball" ]; then
		+	export ICECC_VERSION=$icecc_tarball
		+	export PATH=/usr/lib/icecc/bin:$PATH
		+	export MAKEFLAGS=-j12
		+	fi
		+	unset icecc_tarball procfamily DISTRIB_ID DISTRIB_RELEASE DISTRIB_CODENAME DISTRIB_DESCRIPTION
		+	</pre>
		+
		+	Note that on other distributions than Ubuntu, you might have to modify this, depending on the contents on /etc/lsb-release.
		+
		+	Then, create directory $HOME/icecc and run <pre>icecc --build-native</pre> there. Symlink the created tarball to a descriptive name, e.g. icecc-i686-karmic.tar.gz (this name has to match the $icecc_tarball variable in the .bashrc snippet above).
		+
		+	Now, open a new terminal and ensure that $ICECC_VERSION variable is set. Then run a test compilation to see that the compilation tasks really migrate to other hosts.
		+
		+	Do not proceed beyond this point before you have made a test compilation and seen that your Icecream setup works as intended.

	== Compiling icecc ==		== Compiling icecc ==
Line 54:		Line 87:
	<li>Run the script for each target you wish to use icecc for. E.g.:</li>		<li>Run the script for each target you wish to use icecc for. E.g.:</li>
	<pre>		<pre>
-	host$ ./icecc_create_env_sbox.sh ~~maemo5-armv7~~	+	host$ ./icecc_create_env_sbox.sh FREMANTLE_ARMEL
-	host$ ./icecc_create_env_sbox.sh ~~maemo5-i486~~	+	host$ ./icecc_create_env_sbox.sh FREMANTLE_X86
	</pre>		</pre>
	</ol>		</ol>
Line 63:		Line 96:
	* a symbolic link to this tarball, named "icecc_box-<compiler_name>.tar.gz"		* a symbolic link to this tarball, named "icecc_box-<compiler_name>.tar.gz"

-	These tarballs will be used later when running icecc. It is important that they uniquely identify the toolchain, because all peers in the build network will use the same tarball, and will cache the toolchain locally using that name. Therefore we use the MD5 signature, plus a symlink to easily map it to the actual Scratchbox toolchain name.	+	Place the tarballs in $HOME/icecc inside Scratchbox. These tarballs will be used later when running icecc. It is important that they uniquely identify the toolchain, because all peers in the build network will use the same tarball, and will cache the toolchain locally using that name. Therefore we use the MD5 signature, plus a symlink to easily map it to the actual Scratchbox toolchain name.

	== Using icecc ==		== Using icecc ==
Line 89:		Line 122:
	icecc_tarball="$(readlink -f $HOME/icecc_sbox-$sbox_compiler.tar.gz)"		icecc_tarball="$(readlink -f $HOME/icecc_sbox-$sbox_compiler.tar.gz)"
	if [ -f "$icecc_tarball" ]; then		if [ -f "$icecc_tarball" ]; then
-	export ICECC_VERSION="i486:$icecc_tarball"	+	export ICECC_VERSION="i486:$icecc_tarball"
		+	# comment the following line if you don't want to build in parallel by default
		+	MAKEFLAGS="-j12"
	fi		fi
	unset icecc_tarball sbox_compiler		unset icecc_tarball sbox_compiler
Line 121:		Line 156:
	</pre>		</pre>
	</ol>		</ol>
		+
		+	The snippet above can be placed in .bashrc as well.

	You can now build applications normally and they will automatically use icecc when calling gcc or g++.		You can now build applications normally and they will automatically use icecc when calling gcc or g++.

mairas at 08:05, 8 February 2010

mairas — Mon, 08 Feb 2010 08:05:48 GMT

← Older revision		Revision as of 08:05, 8 February 2010
Line 1:		Line 1:
-	~~= Icecream + Scratchbox HOWTO =~~
-
	''Original instructions written by Anderson Lizardo. This version copied verbatim from the INdT wiki.''		''Original instructions written by Anderson Lizardo. This version copied verbatim from the INdT wiki.''
		+
		+	= Icecream + Scratchbox HOWTO =

	From Icecream web page <ref>http://en.opensuse.org/Icecream</ref>:		From Icecream web page <ref>http://en.opensuse.org/Icecream</ref>:

mairas: New page: = Icecream + Scratchbox HOWTO = ''Original instructions written by Anderson Lizardo. This version copied verbatim from the INdT wiki.'' From Icecream web page http://en.opensuse.org...

mairas — Mon, 08 Feb 2010 08:05:15 GMT

New page: = Icecream + Scratchbox HOWTO = ''Original instructions written by Anderson Lizardo. This version copied verbatim from the INdT wiki.'' From Icecream web page <ref>http://en.opensuse.org...

New page

= Icecream + Scratchbox HOWTO =

''Original instructions written by Anderson Lizardo. This version copied verbatim from the INdT wiki.''

From Icecream web page <ref>http://en.opensuse.org/Icecream</ref>:

"Icecream is created by SUSE and is based on ideas and code by distcc. Like
distcc it takes compile jobs from your build and distributes it to remote
machines allowing a parallel build on several machines you've got."

This document describes how to use icrecream for parallel compilation inside
Scratchbox.

== Compiling icecc ==

To be able to use icecc both on ARM and x86 targets, we need to compile it use
the "special" host-gcc and host-g++ compilers inside a scratchbox target. The
binaries generated by these compilers can then be run from any target, because
they link to scratchbox own libraries.

We will use only the icecc client, and share the icecc daemon with the host.
This allows using icecc both inside and outside scratchbox.

To compile icecc, enter a scratchbox target (the architecture set on it is not
important) and unpack the icecc source tarball (you can download it from <ref>ftp://ftp.suse.com/pub/projects/icecream/</ref>).
Next, enter the source directory and run:

<pre>
sbox > ./configure --prefix=/scratchbox/tools CC=host-gcc CXX=host-g++
sbox > make
</pre>

== Installing icecc ==

The next commands should be done outside scratchbox, as root.

Copy the "client/icecc" binary to /scratchbox/tools/bin and a "icecc++"
symlink necessary for compiling C++ programs:

<pre>
host# cp -a client/icecc /scratchbox/tools/bin/icecc
host# ln -sfn icecc /scratchbox/tools/bin/icecc++
</pre>

== Preparing toolchain tarballs ==

To make sure that the correct compiler is used, it is necessary to create a
tarball containing all files required by the toolchain. This can be done either
manually, or by using the "icecc_create_env_sbox.sh" script.

<ol>
<li>Download [[Media:Icecc_create_env_sbox.sh|icecc_create_env_sbox.sh]] script.</li>
<li>(Optional) Edit the script and change the "sbox_dir" variable, in case your scratchbox is not installed in /scratchbox</li>
<li>Run the script for each target you wish to use icecc for. E.g.:</li>
<pre>
host$ ./icecc_create_env_sbox.sh maemo5-armv7
host$ ./icecc_create_env_sbox.sh maemo5-i486
</pre>
</ol>

For each run, two files will be created:
* a tarball named <md5>.tar.gz, where <md5> is the MD5 signature of the tarball contents.
* a symbolic link to this tarball, named "icecc_box-<compiler_name>.tar.gz"

These tarballs will be used later when running icecc. It is important that they uniquely identify the toolchain, because all peers in the build network will use the same tarball, and will cache the toolchain locally using that name. Therefore we use the MD5 signature, plus a symlink to easily map it to the actual Scratchbox toolchain name.

== Using icecc ==

In order to run icecc, you need to set the ICECC_VERSION environment variable
to point to the tarball created in previous section. Note that this variable
will be set to different values, depending on which target icecc will run (ARM
or x86).

The ICECC_VERSION variable has the following format:

<pre>
<build_architecture>:<cross_toolchain_for_target>
</pre>

Hint:

You can add the following snippet to ~/.bashrc (inside scratchbox) and have
that variable set automatically to the right value (change tarball names and
path accordingly):

<pre>
sbox_compiler="$(sb-conf show --compiler)"
# use the tarball name with the MD5, avoiding conflicting names in peer caches
icecc_tarball="$(readlink -f $HOME/icecc_sbox-$sbox_compiler.tar.gz)"
if [ -f "$icecc_tarball" ]; then
export ICECC_VERSION="i486:$icecc_tarball"
fi
unset icecc_tarball sbox_compiler
</pre>

Done! To effectively use icecc to compile code, set the compiler to "icecc".
This is usually done by setting the CC and CXX environment variables.

== Using icecc "transparently" under Scratchbox ==

Modifying build systems to use "icecc" instead of the defaults g++/gcc can be error prone. Therefore, the following instructions can be used to make it transparent:

<ol>
<li>Create a "/host_usr/bin" directory if it does not exist already:</li>

<pre>sbox > mkdir -p /host_usr/bin</pre>

(This directory is automatically added by Scratchbox to beginning of the PATH, therefore it is not necessary to add it manually.)

<li>Create "gcc" and "g++" symlinks into that directory, pointing to icecc:</li>

<pre>
sbox > ln -sfn /scratchbox/tools/bin/icecc /host_usr/bin/gcc
sbox > ln -sfn /scratchbox/tools/bin/icecc++ /host_usr/bin/g++
</pre>

<li>Disable Scratchbox redirection for the gcc and g++ binaries:</li>

<pre>
sbox > export SBOX_REDIRECT_IGNORE=/usr/bin/gcc,/usr/bin/g++
</pre>
</ol>

You can now build applications normally and they will automatically use icecc when calling gcc or g++.

== References ==

<references/>

[[Category:Development]]
[[Category:Scratchbox]]