Debugging Debootstrap

Contrary to expectations, last week’s AJ Market project turned out to be debootstrap, not dak. Just goes to show a single person can make a difference in today’s world: debootstrap popped into the lead from nearly the bottom thanks to a single contribution. (I wonder if it makes more sense to make contributions anonymous or not?)

For those playing along at home, debootstrap is a tool to build a Debian system from (almost) nothing – it just requires some basic POSIX shell functionality, things like sed, grep, sort, ar, tar, gzip, and wget. So it’s useful when you’re initially installing Debian (or derived distros like Ubuntu), or if you’re creating a chroot environment for dedicated build environments.

Anyway, the debootstrap changes were pretty much bugfixes only, so there’s no fancy new features (okay, there’s one: a --make-tarball option) but a few of the fixes are worth a quick look.

Read the rest ...

AJ Market Update

Hrm, I’m going a bit single issue; I should fix that. But not right now.

So it’s been a couple of weeks since I first posted about my little market experiment, which seems as good a time as any to take a look at how it’s working out. On the one hand it’s going fairly well; it’s pleasing to finally have tiffani done, and I’m pretty satisfied with how usercategories have turned out, and Joey Hess has started using them for the copious bugs the debian-boot team have to deal with, providing both an archfirst ordering and a categorised ordering for the by-maintainer views.

Of course, creating those views didn’t turn out as straightforward as it should have, and ended up involving fixing quite a few bugs in my initial implementation of usercategories (and, happily, coming up with a more pleasant algorithm for handling the ordering specification in request@ mails). There were a few bugs introduced for people who didn’t use usercategories that turned up as a result of their implementation which also needed fixing.

To me, though, the striking difference between taking a “volunteer” attitude to Debian and a “professional” was in how long it took usercategories and tiffani to go from conception to implementation. I don’t think they’re terribly different in complexity; there’s a mental leap in working out what you want to do (people have been thinking about making Packages files faster to download for years, and using ed diffs isn’t particularly obvious; likewise I’ve been thinking about generalising the categories the BTS shows you since I started playing with debbugs in ‘99 or so, but usercategories didn’t really come together even as an idea until debconf this year), but given that, the actual implementation for both ideas requires a little care, but doesn’t have all that many twists and turns. The result? tiffani, the amateur project, took three and a half years to do; usercategories, the amateur project treated professionally, took two months.

So on that score, this still seems completely worth doing. On the other side of the scale, there’s been people’s interest in actually contributing. Which seems as good a place as any for a fold.

Read the rest ...

Tiffani

So this week’s project was working on dak, in particular getting the tiffani implemention included. What’s tiffani, you ask? It lets you just download the changes to Sources.gz and Packages.gz files, instead of the whole damn thing – if you have main, contrib, and non-free for unstable in your sources.list, this means your apt-get update only needs to download 27.5 KiB instead of 2.85 MiB (for bz2) or 3.75 MiB (for gz). Two orders of magnitude saving’s not bad.

I’ve mentioned this previously a few times if you want to see the history, in January I noted some quick hacks which have now been obsoleted by Michael Vogt’s integration into apt (which is in a personal repository, not uploaded to unstable yet…), back in August last year I did a little promotion of the concept, referring back to my post in December 2003 that summarised the analysis done previously, which takes us back to March and April 2002.

The server-side implementation, tiffani, is thus a little python script that groks through the list of suites and components dak knows abouts, looks for Packages, Sources and Contents files, and creates a directory names something like Packages.diff which in turn contains an Index file and a number of gzipped patches, named after the time they were created. The process for updating an existing Packages file is to look for the matching SHA1-History entry in the Index file, download the corresponding patch, apply that patch to your existing Packages file, and repeat until your Packages file matches the one you were looking for (which you can determine by looking at the checksum in either the Release or Index file).

Anyway, tiffani’s been co-developed with Andreas Barth (aka Andi) after the following exchange back in November 2003 on the RM channel:

<aba> elmo: If I find some time to take a look at katies TODO-file, what would be most appreciated? Deleting udebs with melanie? “Right” handling of orig.tar.gz when jumping out of NEW / changing of the archive section? Something else?
<elmo> aba: the TODO list is a bit of a mess to be honest
<aba> elmo: So, what should I do if I want to help the ftp-masters with katie?
<elmo> aba: I wouldn’t necessarily do anything based off that unless it’s obviously correct/wanted. none of the really urgent stuff is even on there (e.g. testing-security changes)
<aba> elmo: If you tell me something urgent, I can put some time into it.
<aj> hrm, surely i have some wishlists
<aba> urgent wishlist? :)
<aj> actually, you could critique something for me if you’ve got half an hour free now?
<aba> “critique” means?
<elmo> tell him how crap his ideas are. as brutually as possible
<elmo> ;)

…

<aj> ooo!
<aj> aba: around?
<aj> aba: one useful thing you could work on that’s kinda katie related is implementing pdiffs into katie/apt-ftparchive/apt/whatever
<Kamion> aj: mvo’s been working on that over the last couple of days, see #128818 logs
<aj> apt side or katie side?
<Kamion> apt side
<aj> cool
<aj> aba: hacking up a katie-side implementation for that would probably be interesting, worthwhile and straightforward

(aj = me, aba = Andi, elmo = James Troup, Kamion = Colin Watson)

That ended up with me finding an old script I made up called update.py (from Dec 2003 apparently) that did the core patch and Index generation, Andi updating that to some more sane pythonic style and making it work on an actual archive, then both of us cleaning it up so it would actually work sensibly without handholding.

(For those playing along at home, the idea needing critiquing was trying to fix up dak’s idea of what should happen if a new version of a package is uploaded to stable or testing, when it hasn’t changed in unstable. At the time, dak would REJECT it, because it violated the requirement that you can’t downgrade packages between stable and testing or testing and unstable. That’s still somewhat broken, unfortunately)

TTBOMK, Tiffani’s named after Tiffani Amber Thiessen, apparently of Saved by the Bell and Beverly Hills 90210 fame. I’m not sure who of aba or elmo is more to blame for that one, though I guess it is at least a little more distinctive than update…

Usercategories and other miscellania

So, this week’s AJ Market project was the first couple of items on my debbugs TODO list, viz:

1. Finish off usertag support
2. Implement usercategory support

Both of these are essentially followup for the initial usertags announcement from last month. The usertags cleanup amounted to adding some basic documentation which will hopefully make it to the website soon, adding some users to various views by default (currently the @packages.debian.org address for package and source queries, and the maintainer and submitter address for maintainer and submitter queries), limiting the characters that can be set in usertags so they don’t conflict with the characters the CGIs are willing to display, and suchlike.

Some other minor niceties, such as docs for the block command, summaries of blocking issues on the pkgreport pages, and a cute mrtg graph for monitoring the BTS’s spam related delays, also made it in.

The bulk of the work though went into the usercategory feature. The idea of usercategories is to let you sort bugs into more suitable categories than the defaults, and to leave working out what those are to individual maintainers and users. The previous possibilities were to stick them into the URL as CGI parameters (possibly via a tinyurl redirect, or a link on some other page), or to use cookies locally after filling in a form. Neither worked really well, so the solution instead is to allow you to create a named usercategory via the email request/control interface, and reference it by name in the URL. The URL syntax is just to add an

;ordering=development-view

option to the pkgreport.cgi URL; and the request bot syntax is:

To: request@bugs.debian.org

user bugs.debian.org@packages.debian.org
usercategory dev-priorities [hidden]
 * Development Priorities [tag=]
  + Sorting Features (should close) [sorting-close]
  + Usertag Related Features (should close) [usertags-close]
  + CGI Related Features [cgi]
  + Changes to rewrite rules [rewrite-rules]
  + Bug Subscriptions [bug-subscriptions]
  + Bugscan Problems [bugscan]
  + Index problems [indexes]
  + Spam Control [spam]
  + Documentation [doc]
  + Re-education Camp Required [inflexible-view-of-the-world]
  + Random Features [random]
  + Can be closed? [should-close]
  + Unprioritised bugs []

usercategory development-view
 * dev-priorities
 * status

The leading spaces are optional; the asterisks and plusses aren’t. Asterisks denote a new level, and plusses denote a section within the level. You can define a prefix for the level as a syntactic convenience. You can either define each level in full, or by reference to a different category, in which case that category’s level(s) will be inserted. If you later change “dev-priorities”, “development-view” will be correspondingly changed. Usercategories are listed in the drop down box in the Options form, though only as long as they’re not marked “[hidden]”.

The above lists the tags in the order they should be looked at, so if a bug has both “random” and “should-close” as usertags, it’ll be listed under “Random Features” not “Can be closed?”. That’s the order those sections are listed in too – but that can be changed by adding an explicit ordering, such as:

  + Random Features [7:random]
  + Can be closed? [5:should-close]

A nifty trick, which won’t necessarily continue to work, is the possibility of creating a usercategory called “normal”, which will overwrite your default view of the bugs, particularly if its assigned to a user whose tags and categories are automatically available, such as the packages.debian.org, maintainer and submitter addresses.

In theory that should be enough to make usercategories actually usable. There are certainly some bugs remaining, but they don’t seem to get in the way too much.

UPDATE 2005/10/10:

Hrm, request@ changes actually updated on bugs.debian.org now. I knew committing to debbugs CVS promptly would come at a cost...

The AJ Market

Where to begin?

One of the things that’s most struck me about Ubuntu is how far it’s progressed with little more than Debian as a base, some reasonable cash to cover a professional level of work, and some dedication to promoting itself and community building.

As an experiment, in July, August and September I tried doing something similar with debbugs – ie, actually committing myself to spend some real time on it as a professional (which I guess ended up being a day or two a week on average, but was still fairly irregular unfortunately), and promoting it both by giving a talk about it at debconf, and involving more people in its development and trying to get some of the feature requests that’d been hanging around finished with so we could move on to new stuff.

I think that’s actually had pretty impressive results – there’s a lot more interest, some fairly serious improvements in both its look an functionality, and for a project that’s been essentially moribund for half a decade, it’s even gained a little momentum. If I hadn’t already been amazed by how well Ubuntu’s done with relatively little effort, I’d’ve been utterly shocked, and heck, maybe I am even so.

Of course, the problem with this is that it really does rely on some real, professional-level commitment; it’s hard to be enthusiastic and active if you’ve just had a stressful day doing paying work, and it’s hard to be responsive if your Debian time doesn’t have any set schedule, and ends up competing against other hobbies, like sleep. But on the other hand, dedicating 40% of your potential income to free software isn’t really something that’s that easy to justify on an ongoing basis, unless perhaps you’re already ridiculously wealthy, or comfortably retired. Even Richard Stallman has a couple of awards worth a few hundred thousand each, to justify his time spent.

Adding this and my relatively recent fascination with market dynamics, I’ve been pondering over the last few weeks whether it’s not worth taking my longstanding amenability to bribes a little more seriously, and trying to construct a real justification for treating Debian work as a professional venture rather than an entertaining hobby that lets me see the world, both virtually, and occassionally for real.

Hence, the AJ market.

The idea is I dedicate some real time to work on free software, and you contribute money to tell me what’s worth working on.

I think it makes sense from both a “free software” point of view, and an “economics” point of view. On the free software side, it avoids getting entangled with proprietary software, promotes development, and provides an easy way to ensures my “users” are actually my priority without giving up my judgement on what’s actually a sensible way of doing things. On the economics side of things, for the time being at least the supply side’s okay, since at worst, I’m willing to throw away some time to see how this works out, and on the demand side, there seem to be enough people who think I should be doing more work on one area or another, that some of them might think that’s worth more than just talking about it. In theory, one or two hundred folks liked what I do for Debian enough to vote for me as DPL, I guess it’ll be interesting to see if that translates to cash rewards. :)

Anyway, that’s the theory. There’re a reasonable number of links from the market page to explanatory stuff, but if that’s too complicated I guess the simple summary is something like this: work on debbugs makes fixing bugs in Debian easier; work on dak makes organising Debian easier; work on britney makes releasing Debian easier; work on debootstrap makes installing Debian easier; work on ifupdown makes networking Debian machines easier.

I’ve also added a little chart on my blog, for those of you who don’t get this via RSS. No more Google ads or paypal buttons.