Debugging Debootstrap

Contrary to expectations, last week’s AJ Market project turned out to be debootstrap, not dak. Just goes to show a single person can make a difference in today’s world: debootstrap popped into the lead from nearly the bottom thanks to a single contribution. (I wonder if it makes more sense to make contributions anonymous or not?)

For those playing along at home, debootstrap is a tool to build a Debian system from (almost) nothing – it just requires some basic POSIX shell functionality, things like sed, grep, sort, ar, tar, gzip, and wget. So it’s useful when you’re initially installing Debian (or derived distros like Ubuntu), or if you’re creating a chroot environment for dedicated build environments.

Anyway, the debootstrap changes were pretty much bugfixes only, so there’s no fancy new features (okay, there’s one: a --make-tarball option) but a few of the fixes are worth a quick look.

Read the rest ...

Scott James Remnant Considered Harmful

I refer, of course, to the appearance of Bug#313400 in the archive and making unstable bootstrapping break in subtle ways on the very day I make my first debootstrap upload in a year and a half. Bastard.

I noticed the bug when converting the sid script to have the nifty detailed progress support Colin Watson implemented for Ubuntu. The relevant implementation detail is that various fds get munged around, and then held open by daemons like inetd and cron that get started by /usr/sbin/start-stop-daemon when they should’ve been skipped by the dummy /sbin/start-stop-daemon that debootstrap cunningly puts in place, which causes the pipe to stay open, and debootstrap to hang just before it finishes.

Ignoring that problem, the other hacking seemed to go fairly well.

Having debootstrap report progress while the install’s happening made me feel like putting all the confusing dpkg output somewhere else, so there’s now a debootstrap.log file in the /debootstrap directory in the root of the target, that gets moved to /var/log/bootstrap.log when (if…) the installation completes. That’ll probably make things awkward for installer types who might want to keep the verbose output showing up live on a tty, so something might have to be tweaked around there to have that come back. We’ll see.

The override munging I did yesterday seemed to work out, with Priority: required and Priority: important matching the packages that are meant to be part of the base system fairly well - at least on i386. So the sid script now also doesn’t have any hardcoded information about what’s in base, it pulls it from the Packages file instead. By the time it appears in the archive, there’ll also be Build-Essential: yes tags to help it work out which packages to install in the buildd variant. Unfortunately these are likely pretty wrong at the moment, but should improve soon enough. And won’t require a debootstrap upload in order to improve in future either! Awesome! (Also awesome: closing bugs that have been open for over four years!)

What else? Ah yes. I wrote a version of pkgdetails in perl (well, shell and perl), so that I could make the package just a simple matter of scripting. In theory that should let you just grab the deb and run debootstrap on Red Hat or Solaris or maybe even Windows. In practice, we’ll see – but hey, it gives me an excuse to close the bugs anyway. Unfortunately the perl version is noticeably slower than the C version, and unfortunately one of the particularly slow bits that parsed the entire Packages file was getting called once for each package and being almost as painful as reading this sentence. So the invoking code in download_release also got rewritten, and now, not only is the Packages file only parsed once while all the debs are being downloaded and validated, but the code’s in roughly the right form to make it plausible to implement support for pulling from multiple sources.

Finally, there’s the Release.gpg signature checking that Colin implemented for Ubuntu. Just install gpgv, point debootstrap at a trusted keyring with the --keyring argument, and you’re done. Sweet.

That’s about it, really. Most of the other things on the TODO list are probably right to be left for a little while yet, and there’s definitely a few bugs that’ll need to be worked out in the stuff already implemented. The CodeWiki seemed somewhat useful for organising development stuff – it’s nice to be able to see some progress, and have something that looks vaguely professional, while at the same time requiring no actual effort.

So, that’s 0.3.1. Surely something’s broken this time.

Cross-strapping Revisited

Quite some time ago now, I started hacking on cross-strapping support for debootstrap – that is, you start bootstrapping your install non-natively (eg, a Hurd install on a Linux system, a powerpc install on an i386, or a Debian install on a Solaris system), then boot into your half unpacked system, and reinvoke debootstrap to finish the job.

It got mostly implemented, then left to hang while I did other things, nominally waiting for an updated makedev. I went and had a look again last year when I started playing with darcs and forward ported bits of it, but not quite enough to have it work properly.

So third time lucky – I’ve been poking around at it again over the long weekend, and it’s getting a fair bit further this time. I decided to setup a CodeWiki to keep track of some of the things I’m hacking on, something like tridge’s junkcode but with a different focus.

The cross-strapping part’s working again, and a reasonable chunk of other stuff also, such as mostly syncing with Ubuntu, automatically resolving missing dependencies within the base system, merging the buildd variant into the main sid script to reduce duplicate code, and a first pass at the code support for determining required/base packages entirely automatically.

Of course, that latter one’s had some code scattered around my filesystem for ages too – it’s been waiting on ftpmaster getting the ability to specify tags in the Packages files on a per-arch basis. That’s still waiting on a fix to Bug#225947, which has had a patch in the BTS for over a year now, and which mvo has, in theory, cleaned up recently. In any case, I decided waiting was boring, and put my Debian ftpmaster hat on tonight and munged the required/important priorities to match required/base for i386 at least; we’ll see if that worked or not tomorrow.

Anyway, in the meantime:

Uploading via ftp debootstrap_0.3.0.dsc: done.
Uploading via ftp debootstrap_0.3.0.tar.gz: done.
Uploading via ftp debootstrap-udeb_0.3.0_i386.udeb: done.
Uploading via ftp debootstrap_0.3.0_i386.deb: done.
Uploading via ftp debootstrap_0.3.0_i386.changes: done.

Let’s see what breaks.

Cross-strapping!

So, I’ve actually done something vaguely productive with arch now; namely put debootstrap into it, and hacked on cross-strapping support. Cross-strapping is basically inspired by the needs of the Hurd project, and more particularly the crosshurd package. Basically, in order to install a Debian GNU/Hurd system you have to boot up Linux, do partitioning and a partial install, then reboot into that partial install and finish it off. You can’t do the partitioning in Hurd, and you can’t get a functional Hurd system from Linux, so you have to do it in two stages.

So naturally, that’s how debootstrap works now. The first stage can thus be executed on any Unixy platform, and just needs basic things like a POSIX sh, sed, wget and access to a Debian archive; and while the second stage does have to be run on the target host, it doesn’t require anything other than what the first stage has already setup. In fact, the procedure for completing the second stage install is just:

chroot $TARGET debootstrap/debootstrap

In theory this should be useful for a bunch of things other than the Hurd; such as setting up Linux chroots on BSD systems, installing Debian onto embedded systems, and possibly doing cool emulation stuff with things like qemu.

The only real problem with it at the moment, is that it isn’t able to create the correct devices for the target system (it creates the devices for the host system instead). Hopefully that can be fixed once Bdale finishes porting the Red Hat makedev to Debian.

A snapshot tarball is available here. If you want to play, you can do a first stage only install with:

debootstrap --arch ${TARGET_ARCH} --foreign sarge ${TARGET_DIR} ${MIRROR}

The --arch flag is required for everything interesting, the --foreign flag tells debootstrap to do the first stage only.