Distcc - Revision history

amigadave: /* References */ better author link

amigadave — Fri, 20 Aug 2010 08:38:30 GMT

References: better author link

← Older revision		Revision as of 08:38, 20 August 2010
Line 196:		Line 196:

	* [http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]		* [http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]
-	* [~~http~~:~~//www.GlobalLinuxSecurity.com~~ Original author]	+	* [[User:Ewheelerinc\|Original author]]

	[[Category:N900]]		[[Category:N900]]
	[[Category:Software]]		[[Category:Software]]
	[[Category:Development]]		[[Category:Development]]

amigadave: wikify slightly, tidy

amigadave — Fri, 20 Aug 2010 08:37:28 GMT

wikify slightly, tidy

Show changes

ewheelerinc: /* Distributed Remote-Distributed-Compile from the n900 to native armel targets */

ewheelerinc — Fri, 20 Aug 2010 03:55:48 GMT

Distributed Remote-Distributed-Compile from the n900 to native armel targets

← Older revision		Revision as of 03:55, 20 August 2010
Line 1:		Line 1:
-	= ~~Distributed~~ Remote-Distributed-Compile from the n900 to native armel targets =	+	= Remote-Distributed-Compile from the n900 to native armel targets =

	For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!		For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!

ewheelerinc: /* 6. Halt the vm and launch multiple instances */

ewheelerinc — Fri, 20 Aug 2010 03:38:58 GMT

6. Halt the vm and launch multiple instances

← Older revision		Revision as of 03:38, 20 August 2010
Line 119:		Line 119:
	d. Launch the vm's:		d. Launch the vm's:

-	host-root# for i in 1 2 3; do qemu-system-arm -name armvm$i \	+	host-root# for arm in 1 2 3; do qemu-system-arm -name armvm$arm \
	-M versatilepb -kernel vmlinuz-2.6.26-1-versatile \		-M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
	-initrd initrd.img-2.6.26-1-versatile \		-initrd initrd.img-2.6.26-1-versatile \
Line 135:		Line 135:
	tap2		tap2
	tap3		tap3
-
-

	== Run a distributed build on your n900 ==		== Run a distributed build on your n900 ==

ewheelerinc: /* About the Author */

ewheelerinc — Fri, 20 Aug 2010 03:34:15 GMT

About the Author

← Older revision		Revision as of 03:34, 20 August 2010
Line 166:		Line 166:

	--[[User:ewheelerinc\|ewheelerinc]] 03:27, 20 August 2010 (UTC)		--[[User:ewheelerinc\|ewheelerinc]] 03:27, 20 August 2010 (UTC)
		+
		+	=== Forum Thread ===
		+	[http://talk.maemo.org/showthread.php?p=792606 Click here for the forum thread on this subject]

	=== About the Author ===		=== About the Author ===
	Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com]. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].		Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com]. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].

ewheelerinc: distcc+ccache distributed gcc make -jX on the n900 against armel VMs on x86 hardware

ewheelerinc — Fri, 20 Aug 2010 03:27:28 GMT

distcc+ccache distributed gcc make -jX on the n900 against armel VMs on x86 hardware

New page

= Distributed Remote-Distributed-Compile from the n900 to native armel targets =

For those who have tried to compile on the n900, you might notice it is a bit slow since an ARM CPU is great for general use, but not---for example---compiling a kernel. It, however, often useful to compile natively on the architecture you will be using. Originally this process was used to configure a distcc farm for the mini2440 ARM(el) board---and this is directly applicable to the n900 platform---and it works on my phone!

This assumes familiarity with qemu-system-arm, gcc (and possibly ccache+distcc), chroot installs, bridged networking and perhaps a few other concepts.

Where possible I distinguish the different system root-prompts as:
n900-chroot# The easydebian chroot prompt (you can ssh in too if you like)
host-root# The high-power desktop system to run the ARM VM's under qemu
guest-root# The ARM VM's themselves.

== 1. Install Easy Debian ==

Before continuing, you will need to install [http://wiki.maemo.org/Easy_Debian Easy Debian] on your phone. This is the fastest way to get a working native gcc environment on the n900. Once it is installed, install ccache, distcc, and build-essential:

n900-chroot# apt-get install ccache distcc build-essential

Then configure /usr/bin/gcc-ccache the same way as #4 below on the easydebian chroot.

== 2. Install a few QEMU VMs ==

On your host computer (not the phone), download and install the Lenny armel qcow2, vmlinuz and initrd images from here:

[http://people.debian.org/~aurel32/qemu/armel/ Lenny armel qemu images]

I used the _small version so I can be as specific as possible with packages. I assume you use the _small version from here.

Install qemu-arm on your system (usually apt-get install [http://www.qemu.org qemu] and boot the VM:

host-root# qemu-system-arm -M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
-initrd initrd.img-2.6.26-1-versatile \
-hda debian_lenny_armel_small.qcow2 \
-append "root=/dev/sda1"

== 3. Install the required packages on the VM ==

First update /etc/apt/sources.list for the country of your choice (defaults to .de). I recommend commenting the volatile repo unless you intend to use it.
Then add the packages:

guest-root# apt-get update
guest-root# apt-get upgrade
guest-root# apt-get install build-essential ccache distcc

== 4. Configure distcc and ccache ==

a. Add the hosts to /etc/distcc/hosts

b. Configure /etc/default/distcc with at least:
STARTDISTCC="true"
# configure your subnet
ALLOWEDNETS="127.0.0.1 192.168.0.0/16"

c. create a ccache/distcc wrapper

root# mv /usr/bin/gcc /usr/bin/gcc-orig
root# cat<<EOT > /usr/bin/gcc-ccache
#!/bin/sh
export CCACHE_PREFIX=distcc
# List the distcc hosts here; I had trouble with /etc/distcc/hosts working
export DISTCC_HOSTS='192.168.99.101 192.168.99.102 192.168.99.103'
exec ccache gcc-orig "$@"
EOT
root# chmod 755 /usr/bin/gcc-ccache

Be sure to do this on the n900 too!

d. link the wrapper as the new gcc:
root# ln -s /usr/bin/gcc-ccache /usr/bin/gcc

e. test the distcc-ccache-gcc wrapper:
root# gcc -version
gcc-orig: no input files
distcc[2486] ERROR: compile (null) on localhost failed

The distcc error is expected

== 5. Configure the network to DHCP ==

guest-root# echo -e "auto eth0\niface eth0 inet dhcp" >> /etc/network/interfaces
guest-root# echo -e "auto eth1\niface eth1 inet dhcp" >> /etc/network/interfaces

== 6. Halt the vm and launch multiple instances ==

a. Halt the guest and close the window:
guest-root# shutdown -h now

b. Configuring a network bridge is beyond the scope of this document, but roughly, this is my config on the *host* system. Here we assume that the n900 can route to the ARM VM farm (my network)---or that the bridge lives on the same network (more common).

root# cat<<EOT > /usr/local/bin/qemu-ifup-br0
#!/bin/sh
iface="$1"
bridgeName=`basename "$0" | cut -f3 -d-`
echo "bridgeName=$bridgeName iface=$1"
/sbin/ifconfig "$iface" 0.0.0.0 up
/usr/sbin/brctl addif "$bridgeName" "$iface"
EOT
chmod 755 /usr/local/bin/qemu-ifup-br0

host-root# brctl addbr br0
host-root# brctl addif br0 eth0
host-root# ifconfig br0 192.168.99.222
host-root# ip route replace default via 192.168.99.1

c. Configure your local dhcp server to hand out 192.168.99.101--103 for mac addresses 00:16:3e:00:00:01--03. If you are using ISC's dhcpd, the config looks something like this:

host arm-vm1 {
hardware ethernet 00:16:3e:00:00:01;
fixed-address 192.168.99.101; }
host arm-vm2 {
hardware ethernet 00:16:3e:00:00:02;
fixed-address 192.168.99.102; }
host arm-vm3 {
hardware ethernet 00:16:3e:00:00:03;
fixed-address 192.168.99.103; }

d. Launch the vm's:

host-root# for i in 1 2 3; do qemu-system-arm -name armvm$i \
-M versatilepb -kernel vmlinuz-2.6.26-1-versatile \
-initrd initrd.img-2.6.26-1-versatile \
-hda debian_lenny_armel_small.qcow2 \
-append "root=/dev/sda1" \
-net nic,macaddr=00:16:3e:00:00:0$arm \
-net tap,script=/usr/local/bin/qemu-ifup-br0 \
-daemonize \
-snapshot ; done

e. The bridge listing should look something like this:
host-root# brctl show
br0 8000.001cc0b1dadf no eth0
tap1
tap2
tap3

== Run a distributed build on your n900 ==

a. Download a recent vanilla kernel:
cd /usr/src
wget -O- http://ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.34.tar.bz2 | tar xjv
cd linux-2.6.34
[...]
make -j3 defconfig
*** Default configuration is based on 'versatile_defconfig'
#
# configuration written to .config
#
make -j4
[...]

b. Open top and watch your ARM VM's spin at 100% cpu:
USER PR NI VIRT RES SHR S %CPU %MEM COMMAND
root 20 0 556m 362m 1492 R 101 6.0 qemu-system-arm -name armel-vm1 ...
root 20 0 539m 278m 1472 R 101 4.6 qemu-system-arm -name armel-vm2 ...
root 20 0 530m 216m 1472 R 99 3.6 qemu-system-arm -name armel-vm3 ...
root 20 0 538m 208m 1472 R 101 3.5 qemu-system-arm -name armel-vm4 ...

As you can see, I have 4 VM's on my system, one for each core on the i7. Since the ARM architecture under qemu (currently) only supports a single core, you need to run a VM for each core.

You *could* configure distcc+ccache to cross-compile to armel on your native x86/64 hardware, however, that is a different tutorial (it would probably be much faster too, but perhaps less flexible). This might be extended to run on the n900's localhost to spin the native 600mhz CPU too---and I leave this as an exercise for the reader.

--[[User:ewheelerinc|ewheelerinc]] 03:27, 20 August 2010 (UTC)

=== About the Author ===
Eric Wheeler is a Linux consultant and security professional available for hire at [http://www.GlobalLinuxSecurity.com http://www.GlobalLinuxSecurity.com]. He has 14 years of experience in Linux since 1996 with a great breadth of knowledge. Eric is currently working on a PhD in Computer Science at Portland State University under [http://cs.pdx.edu/~walpole Professor Jonathan Walpole].