joerg_rw: /* repository/stage VM */ added ===maintenance===

joerg_rw — Sat, 23 Nov 2013 18:40:21 GMT

repository/stage VM: added ===maintenance===

← Older revision		Revision as of 18:40, 23 November 2013
Line 69:		Line 69:

	this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)		this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)
		+
		+	=== maintenance ===
		+	<pre>
		+	builder@garage:~$ ./buildme
		+	>>>> /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3-1.dsc
		+	[2013-11-23 18:22:39] REJECTED: "Can't stat /mnt/incoming-builder/fremantle/gconf-cleaner_0.0.3.orig.tar.gz: No such file or directory"
		+	[2013-11-23 18:22:39] Unexpected error:
		+	Error'>: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
		+	Traceback (most recent call last):
		+	File "/usr/lib/python2.7/dist-packages/buildlib/app.py", line 81, in run
		+	mainfunc(argv, options, self._logger, self.conf)
		+	File "./buildme", line 641, in main
		+	'conf' : conf
		+	File "/usr/lib/python2.7/dist-packages/buildlib/fsm.py", line 72, in run
		+	code = handler(self)
		+	File "/usr/lib/python2.7/dist-packages/buildlib/helpers.py", line 82, in wrapper
		+	rc = func(*args)
		+	File "./buildme", line 341, in setup_build
		+	reject_fname(env, exobj)
		+	File "./buildme", line 193, in reject_fname
		+	shutil.move(name, rejected_dir)
		+	File "/usr/lib/python2.7/shutil.py", line 289, in move
		+	raise Error, "Destination path '%s' already exists" % real_dst
		+	Error: Destination path '/mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc' already exists
		+	builder@garage:~$ ls -al /mnt/incoming-builder/fremantle/rejected/gconf-cleaner_0.0.3-1.dsc
		+	</pre>

sixwheeledbeast: Moved from migration page - new page

sixwheeledbeast — Wed, 05 Jun 2013 20:30:44 GMT

Moved from migration page - new page

New page

== Autobuilder and friends ==

maemo autobuilder setup

autobuilder consists of multiple VMs

=== drop VM ===
this VM has /etc/passwd synchronised with garage and ~ folders mounted via NFS from garage

account synchronisation is handled by scripts running on garage VM and then sync is triggered using ssh and scripts in /usr/local/bin

packages are uploaded to /mnt/incoming-builder via SCP

=== garage VM ===
this is the VM where stuff happens

password/account sync to gforge/postgresql is done using
*/10 * * * * root /usr/local/bin/add_groups_users_git_ssh.sh > /tmp/add_groups_users_git_ssh.log dev/null 2>&1
this also updates ~/.ssh/authorized_keys

garage also handles web extras-uploader (/var/lib/extras-assistant/) - package is uploaded and then moved to the same folder as packages uploaded to drop and then chowned using

/var/lib/extras-assistant/bin/copy_package_files_to_autobuilder.sh

A lot of jobs on garage VM is done using local root crontab (/var/spool/cron/crontabs/root)

after package is uploaded it's processed by buildME

buildME runs as builder user and it's started from cron every minute
* * * * * builder /home/builder/buildme

buildme is configured using /etc/buildme.conf

buildme takes care of couple things
* verify that .tar.gz and other files are correct (checked using checksum from .dsc file)
* select free destination (buildme can handle parallel builds on multiple hosts/users)
* scp all required files to selected destination
* start sbdmock on the destination
* copy results back and resulting .deb to repository incoming folder (result_dir = /mnt/builder/%(product)s and repo_queue = /mnt/incoming/extras-devel/%(product)s/)
* send emails to list and user uploading package

=== builder VM ===

this VM has standard installation of scratchbox with no targets configured (it's not required for sbdmock)

when sbdmock is started it cleans up old build folder, creates new target and prepares build enviroment and then runs dpkg-buildpackage

sbdmock also generates logfiles that are parsed by buildme

=== repository/stage VM ===

this is where repository management happens
*/2 * * * * repository /home/repository/queue-manage-extras-devel.sh
*/5 * * * * repository /home/repository/queue-manage-extras.sh
*/5 * * * * repository /home/repository/queue-manage-community-testing.sh
*/5 * * * * repository /home/repository/queue-manage-community.sh

those scripts (and scripts inside /home/repository/queue-manager-extras) check for new packages in repository incoming folder and then move those to /var/repository/staging, regenerate Packages

(using sums that were previously cached) and sign it if required and then if any changes happened
#touch .changed file, so we know that we need to sync to live
touch /var/repository/staging/community/.$dist.changed

this file is then checked by
1003 10634 1 0 Mar18 ? 00:00:00 /bin/sh /usr/local/bin/packages/rqp.sh
started by /etc/init.d/repository-qp

this script starts rsync when required to sync to live repository

this script also starts repository-queue-proc.php that processes repository updates coming from midgard (old package cleanup and promotions)

Maemo.org Infrastructure/Autobuilder - Revision history

joerg_rw: /* repository/stage VM */ added ===maintenance===

sixwheeledbeast: Moved from migration page - new page