Dak Extensions

One of the challenges maintaining the Debian archive kit (dak) is dealing with Debian-specific requirements: fundamentally because there are a lot of them, and they can get quite hairy – yet at the same time, you want to keep them as separate as possible both so dak can be used elsewhere, and just so you can keep your head around what’s going on. You can always add in hooks, but that tends to make the code even harder to understand, and it doesn’t really achieve much if you hadn’t already added the hook.

However, dak’s coded in python, and being an interpreted language with lots of support for introspection, that more or less means there’s already hooks in place just about everywhere. For example, if you don’t like the way some function in some other module/class works, you can always change it (other_module.function = my_better_function).

Thus, with some care and a bit of behind the scenes kludging, you can have python load a module from a file specified in dak.conf that can both override functions/variables in existing modules, and be called directly from other modules where you’ve already decided a configurable hook would be a good idea.

So, at the moment, as a pretty simple example there’s an init() hook invoked from the main dak.py script, which simply says if userext.init is not None: userext.init(cmdname).

But more nifty is the ability to replace functions, simply by writing something like:

# Replace process_unchecked.py's check_signed_by_key
@replace_dak_function("process-unchecked", "check_signed_by_key")
def check_signed_by_key(old_chk_key):
    changes = dak_module.changes
    reject = dak_module.reject
    ...
    old_chk_key()

That’s made possible mostly by the magic of python decorators – the little @-sign basically passes the new check_signed_by_key function to replace_dak_function (or, more accurately, the function replace_dak_function(...) returns), which does the dirty work replacing the function in the real module. To be just a little bit cleverer, it doesn’t replace it with the function we define, but its own function with simply invokes our function with an additional argument to whatever the caller supplied, so we can invoke the original function if we choose (the old_chk_key parameter – the original function takes no arguments, so our function only takes one).

Right now, we don’t do much interesting with it; but that should change once Ganneff’s little patch is finished, which should be RSN…

Hopefully, this might start making it easier to keep dak maintained in a way that’s useful for non-Debian installs – particularly if we can get to the point where hacking it for Debian generally just implies changing configuration and extension stuff – then we can treat updating all the real scripts as a regular software upgrade, just like it is outside Debian.

dak dsa

So the final implementation detail in the embargoing scheme is providing a tool to move stuff from the embargoed and unembargoed queues into the archive. The existing tool the security team use is called “amber” (after the inimitable Amber Benson). amber’s pretty simple: it takes a DSA number, and the .changes files you’re looking at; then asks for confirmation, accepts the packages into the archive, regenerates Packages and Release files, fills out a template advisory with details from the deb and mails that off, and uploads the files to ftp-master for inclusion in the next stable release.

There are a few problems with that. One is it doesn’t allow for rejections. Another is it doesn’t provide the security team with the opportunity to edit advisories while the packages are being prepared. Another issue is that the entire program has to run under sudo with full archive priveleges.

Enter “dak dsa” aka “newamber”. The new tool aims to do more or less the same thing, but with a little more style. First, it provides a small interactive interface, so that processing an advisory now looks like:

$ newamber DTSA-25-1 smb4k_*.changes
Create new advisory DTSA-25-1? y
Advisory: DTSA-25-1
Changes:
 smb4k_0.6.4-0.0etch1_alpha.changes
 smb4k_0.6.4-0.0etch1_hppa.changes
 smb4k_0.6.4-0.0etch1_i386.changes
 smb4k_0.6.4-0.0etch1_m68k.changes
 smb4k_0.6.4-0.0etch1_mips.changes
 smb4k_0.6.4-0.0etch1_mipsel.changes
 smb4k_0.6.4-0.0etch1_s390.changes
 smb4k_0.6.4-0.0etch1_sparc.changes
Packages:
 smb4k 0.6.4-0.0etch1 (alpha, hppa, i386, m68k, mips, mipsel, s390, source, sparc)
Approve, [E]dit advisory, Show advisory, Reject, Quit? 

Choosing edit will grab a copy of the template and run vi – the template will only be filled out when you approve the upload though, since the values might change before then. Accept will do more or less what old amber did; though instead of mailing the filled in advisory draft it’ll just leave it in the filesystem instead.

Of course, running vi (well, $EDITOR) generally means you can get a shell too, so running the command with full archive priveleges is a bit much – at least if you’re trying to have any sort of granularity to your security regime, which was, after all, the whole point of this exercise. So instead of running the entire command as the katie user, “dak dsa” instead has to escalate its own priveleges, in this case using sudo and specific options, such as sudo dak dsa -A -- foo.changes to approve foo.changes. Fortunately sudo and the apt argument parser are cooperative enough to allow “dak dsa” users to invoke “dak dsa -A – *” as katie, and thus have only the very limited access we’re trying for.

Obviously the above is taken from the testing-security team – it’s the same source and i386 packages, recompiled on other architectures by the security.debian.org testing autobuilders. It’s shown up a few flaws in the autobuilding for etch: (a) the amd64 autobuilder isn’t active; (b) the arm buildd can’t seem to find its chroot in between running apt-get install and apt-get remove; (c) the s390 buildd only works if the source is on ftp-master; (d) of the five m68k buildds that will take packages for security.debian.org updates to testing, only two will succeed (a400t and poseidon). There’s also the notable problem that the chroots for the functional buildds have gotten out of date and that builds break somewhat obscurely as a consequenece. One of the test updates is also failing on hppa due to space restrictions. And of course, the above list is after a chunk of other problems have already been fixed.

It’s worth noting that even if the above isn’t fixed for testing now; we’ll still need etch chroots for security.debian.org when we release, so those problems have to be dealt with at some point. And that the brokenness is the result of six months’ divergence from sarge; after a year and a half when etch releases – or the three years between woody and sarge’s release – it’s probably fair to expect worse breakage.

Anyway, that’s just about it from me on this topic. Micah Anderson from the testing-security group is currently checking out the unembargoed facility, and has redone a couple of DTSAs on security.d.o. So presumably those guys will start working out whether security.d.o is something they want to make use of, and if so, working out what changes/tweaks are necessary for that. Though, unsurprisingly, I also still have to do some committing to CVS…

Changing The Security Infrastructure

One of the most exciting things about working on Debian is that since it’s developed in the open, when you want to make changes everyone sees them, so depending on what you’re working on, the risk of breaking stuff can provide a real adrenaline rush, while you’re just sitting in your chair. So what better adventure sport could there be than hacking up Debian’s security support?

One idea that comes to mind: doing it with only a minimal idea how it works at the moment.

Read the rest ...

Security Infrastructure Changes

Delays suck.

I’m actually skipping ahead here, I’d meant to blog about updating the dak codebase into shape for whatever changes were coming up first, but it turns out I’m not in the mood for that. Besides which, most of this entry is preprepared. In the last few entries we went over the background for updating our security infrastructure to make it possible to have security team members that don’t worry about vendor-sec issues (process, overview, NIv2 aka NINJA, queue-build issues), all the while skipping fairly lightly over the question of what has to actually happen. That happens to be the way I prefer to work, lots of background tinkering, then spend as little time on actual implementation as possible; it has it’s drawbacks, but hey, I like it.

Anyway, the second last step is working out in detail what’s going to happen. The delay in question was trying to get the security team to sign off on it, but unfortunately they’re busy enough that that looks like it’ll being an indefinite delay, so I’m going with the “well, ftpmaster maintains the dak/katie infrastructure for the security team, so in the end it’s our call on what’s best anyway” line of thought, and just plunging in. Fortunately the current and final plan should make pretty minimal changes to how the existing security team operate anyway, even assuming a massive testing security operation happening in parallel with them – so the downside risk is already pretty minimal. (And it was, afterall, developed in consultation with Joey, and then shopped around a few folks who’ve been doing security work for testing and kernels)

What is that plan, you may well ask. It’s something like this:

First, let’s restate the idea:

Concept

Allow two teams to exist, the current security team as the “vendor-sec subteam”, and the full team which can expand to also include the testing-security people and others interested and able to help out with security work but who do not have access to vendor-sec.

Then, we want to think about the processes that are involved at a global level. First for uploads that can’t be made public until some later date, and can only be worked on by the “vendor-sec” enabled security team.

• Source upload to queue/unchecked (as now)
  • Upload is authenticated etc
  • Upload is moved to queue/embargoed
• Upload is autobuilt
  • Logs are sent to a procmail address
  • Logs are forwarded (by procmail) onto the vendor-sec subteam
  • Logs are signed and sent back
  • Binaries are uploaded to queue/unchecked
  • Binaries are moved to queue/embargoed
• Uploads (source + binaries) are approved from queue/embargoed by
• running amber
  • Uploads are moved into the archive
  • The archive is mirrored
• The approver issues an advisory

And likewise for issues that don’t need so much secrecy:

• Source upload to queue/unchecked-disembargo (new directory)
  • Upload is authenticated etc
  • Source+version added to unembargo table
  • Upload is moved to queue/unembargoed
• Upload is autobuilt
  • Logs are sent to a procmail address
  • Logs are forwarded (by procmail, after checking unembargo table) onto the full team
  • Logs are signed and sent back
  • Binaries are uploaded to queue/unchecked
  • Noting that the source is in the unembargo table, Binaries are moved to queue/unembargoed
• Uploads (source + binaries) are approved from queue/unembargoed by running amber
  • Uploads are moved into the archive
  • The archive is mirrored
• The approver issues an advisory

Processes are great, but they have to be supported by proper information handling, so it’s worth looking at what the above actually means for the on-disk structures, too. This is particularly important, since filesystem level security is mostly how we wall off vendor-sec stuff from other users of the system (whether they’re also working on security issues, or doing, eg, web page maintenance or something equally random). Voila:

queue/unchecked, queue/unchecked-dismebargo
    world writable
    accessible only by katie
    contents automatically cleared every 15m

queue/embargoed
    writable only by katie
    accessible by vendor-sec subteam
    amber allows packages to be ACCEPTed
    [...] allows packages to be REJECTed

queue/unembargoed
    writable only by katie
    accessible by full team
    amber allows packages to be ACCEPTed
    [...] allows packages to be REJECTed

queue/accepted
    packages only sit in accepted briefly while amber is running

And finally, to sum things up, this is what a member of the security team should expect to actually do to release an advisory:

• Upload package to queue/unchecked (or queue/unchecked-disembargo)
• Wait for build logs to be sent to your address
• Check and sign those build logs
• Check the final packages in queue/embargoed (or queue/unembargoed)
• Run amber over the packages to accept them
• Fill in the template advisory generated and send it to the -announce list

The differences between that process and the current one used by the security team is pretty minor, ttbomk: currently uploads sit in queue/accepted not queue/embargoed, and currently the security team ignore the template advisories amber generates preferring to make their own – aiui, this is mostly due to delays in actually receiving the advisory template since amber mails it instead of leaving it somewhere convenient on disk.

Anyway, that’s what the plan looks like. We should see how well it survives contact with the enemy, or at least reality, remarkably soon.

(Of course, I guess “political issues” could always foul it up completely anyway, particularly going by the fact that it’s been more or less the only problem raised by the folks I’ve passed this on to… Oh well, reconstruct that bridge when we get to it, I guess)

Queue Building

The biggest difference between the hypothetical security queues and the existing byhand and new queues is that they need to be autobuilt; it’s not really feasible to try saying whether a security update is acceptable if there’s still some possibility it’s not going to build on half our architectures. Additionally, it means that there are two queues that need to be built, rather than just one.

The way accepted autobuilding works isn’t too complicated, fortunately. There’s a directory where symlinks are made to uploaded packages and sources, and a Packages files (for all architectures) is generated, as is a Sources file, all of which are made available to buildds. They’re also added to the regular unstable Packages and Sources files and passed to quinn-diff and wanna-build. In order to avoid unnecessary breakage at transitional points (such as when a package moves from accepted to unstable proper), packages are kept in the accepted autobuild directory for a day or so afterwards. There’s also the issue of including the .orig.tar.gz from the archive along with the .dsc and .diff.gz from accepted for uploads that aren’t new upstream versions.

Security accepted autobuilding adds two extra wrinkles. The first is that packages being autobuilt can’t be generally visible to other users on the security host, which includes the web server that’s supposed to get the packages to the security buildds. So instead of symlinking, copies have to be made, and care has to be taken to ensure that doesn’t accidently leak information in the process. The second trick is that security needs autobuilding of multiple suites (oldstable and stable, and hopefully testing as well), whereas for the regular archive, it’s sufficient to autobuild unstable packages from accepted, and leave the rest until the daily pulse run. This particularly means that uploads from accepted need to be moved into separate directories depending on the target suite, so that the buildds don’t accidently build a package for woody against the libc from sarge.

At a gross level, generalising to supporting autobuilding of multiple queues is therefore straightforward: extend the “accepted_autobuild” table to include information of what queue the files are from (which implies a name change is appropriate), extend the configuration file to allow you to specify which queues should get autobuild handling for which suites, and finally add the code to actually do this when packages are moved into and out of the appropriate queues. Hurray for simplicity!

But there’s one little bit of complexity still to bother us. We know we need different directories and Packages files for different suites, but do we need it for different queues as well? Worse, we need different chroots on each autobuilder for each suite, do we need different chroots for each suite for each queue? The latter could well be a complete showstopper.

One reason why we might need to treat the queues we’re considering separately is that they have different secrecy requirements: embaroged updates can’t be visible to anyone, even people working on the unembargoed security updates in parallel, and yet the buildd has to forward its logs on to someone to be signed. If embargoed and unembargoed packages are built in the same chroot, by the same buildd, from the same queue, how do the logs magically go to the right person?

That’s not necessarily easily answered; currently the hope is that it’ll be okay to have a single shared queue and set of buildds and chroots, and have all the logs forwarded to a single address, at which point they can be redirected to the appropriate security team members by procmail. The assumption this is based on is that there’ll already need to be some magic to do that for the builds themselves once they’re uploaded and need to be moved into either the embargoed or unembargoed queue, so the same thing should be possible with the build logs.

Pending further surprises though, that’s enough to move onto the next phases, which is generalising dak to cope with arbitrary queues.

The NIv2 Plot

So having found time to catch up satisfactorily on the implementation, time to get back into the blogging.

After working out what you actually want to do, the next step in implementing stuff, in my book, is to make sure you fully understand the context of the stuff you’re trying to change. In this case, the “unapproved” queue concept comes from some thoughts from quite a while ago that come under the heading “NIv2” for “new incoming v2”. For reference, the original “new incoming” was the switch to separate queues for new, byhand, and accepted packages, which was followed by a notable update shortly after to make accepted autobuilding possible.

That setup works, but has a few problems. One is that packages sit in accepted for quite a while (up to a full 24h), and occassionally in that time it can become inappropriate to actually add the package to the archive – for instance if it’s a build of a package that’s then removed from the archive. This results in an UNACCEPT, and requires manual intervention, since potentially it can result in weird inconsistencies, such as bug closures. Another is that files in the various queues aren’t tracked in the database, which can result in files (particularly upstream tarballs) being lost or left behind, and also requires some unpleasant special casing in the code. A final concern is that one of the bigger time sinks in the daily pulse is the apt-ftparchive scan, which is actually just doing work that’s already been done in order to allow packages in accepted to be autobuilt.

Put all those concerns together with the ones mentioned last week, and you get the impetus for NIv2. The principle of that is to put stuff in the queue into the database properly, simplify it, extend the queuing support to deal with different queues, and move stuff from accepted directly into the pool every quarter hour, though only pushing to buildds that frequently, leaving updates of Packages files and mirror pushes to still be daily, or every few hours or whatever.

This is what that bit looked like in the dak TODO:

[hard, long term] unchecked -> accepted should go into the db, not a suite, but similar. this would allow katie to get even faster, make madison more useful, decomplexify specialacceptedautobuild and generally be more sane. may even be helpful to have e.g. new in the DB, so that we avoid corner cases like the .orig.tar.gz disappearing ‘cos the package has been entirely removed but was still on stayofexecution when it entered new.

Obviously, it needed to get defined a little more clearly than that to work out how much the changes for the security queues I’m interested in depend on the other changes, and what they actually mean in detail. After some IRC discussion we came up with more or less the following summary of what NIv2 actually means:

How NIv2 should work.

q/accepted goes away; once packages are accepted, they get added straight into the pool and the database. At this point they’re added to a micro-suite for the buildds, and so apt-ftparchive can cache the package contents; and a symlink tree is made up so incoming.d.o works. The micro-suite might or mightn’t be a fully fledged suite in the database. Presumably it wouldn’t be visible in dists/ though. The micro-suite probably should contain packages uploaded to unstable for X hours, where X is 48 or 2x pulse-interval or similar.

This means UNACCEPT is impossible and apt-ftparchive takes less time in cron.daily.

q/byhand and q/new get new friends: q/sec-embargoed, q/sec-unembargoed, q/stable-updates. Rather than checking for “byhand” entries, or a lack of overrides; uploads get diverted to those queues if they’re uploaded to security or stable, and get moved out of there when a security team member or stable RM (ie, Joey) runs an amber/lisa-esque command.

This means sec-embargoed and sec-unembargoed need to be autobuilt.

This also means that proposed-updates can be a full suite, and point releases can be a matter of (a) adding proposed-updates to stable; (b) removing old versions from stable; (c) removing packages that have to be dropped; and users can put “stable + proposed-updates” in their sources.list reasonably sanely.

This also means that REJECTs from accepted (security) or the archive (p-u) aren’t needed.

It means the queue ends up looking like:

    unchecked      [uploads go here, insecure]

    holding        [secured]
    new            [waiting NEW processing]
    byhand         [waiting byhand processing]
    stable-updates [waiting stable RM approval]

    done           [.changes files for accepted packages]
    reject         [.changes, .reason and rejected stuff]

    bts_version_track [data for BTS]

with q/accepted disappearing and q/proposed-updates and q/old-proposed-updates being replaced by the stable-updates / oldstable-updates queues.

One of the nice parts about this is that it breaks down fairly neatly into three sets of changes: supporting additional queues, removing the accepted queue, and tracking queue stuff in the database. It does make those changes more or less dependent though – the new queues need to be autobuilt in the way accepted is at present; so it makes sense to add the queues before removing accepted so as not to lose the queue autobuild support.

The above actually can be simplified a little further – instead of managing a microsuite for changes since the last update, it’s possible to add accepted packages directly to the real suite; and generate the Packages files needed for autobuilding by querying packages accepted to unstable in the last 24 or 48 hours, simply by adding a field to track upload time for binaries as well as sources.

In any event, this thus makes generalising the queue autobuild support the next challenge.

Afraid of your Neighbour's Disapproval

Continuing on from the SCC stuff, which I should probably get in the habit of calling the mirror split, then…

At the same time as I’ve been trying to work out ways to fit the mirror split stuff into the AJ market concept, I’ve been pondering on and off what can be done to improve the security situation. My thinking’s probably still biassed by my long time wish for official security updates to testing, but I’m somewhat inclined to think that if we can figure out a way of integrating the testing security team and the official security team we might end up closer to a solution – at the very least that’d provide us with a decent number of people who can actively help out with security updates that aren’t secet and get a few more people familiar with the technical aspects of how security updates are released, who’ll then have less of a learning curve if they end up being added to the security team proper. And heck, even if that doesn’t work, it at least gives people running testing an easier url to remember for security updates.

Of course, integrating the teams isn’t that easy, or it would’ve already been done. So after thinking about it, I had a chat with Joey (aka Martin Schulze, aka the guy who’s issued 317 of the 340 security advisories in the past 12 months) about the problems:

<aj> so, the other thing i wonder is what you think of the testing-security-support split?
<Joey> need more input
<aj> as in separate host, funny domain name, not available on security.debian.org/, extra effort to mirror, etc
<aj> i think it sucks :)
<Joey> there are a lot of sucking things…
<aj> it’s true
<aj> so, i figure the reasons for the split are (a) you don’t want to add a bunch of people to team@ and hence vendor-sec, and (b) there’s no way to make it so they can use dak-security without being full team/vendor-sec members; and maybe (c) you don’t want them officially associated with debian security coz they’re newbies
<Joey> a) and b) are true
<Joey> However, with security.debian.org being a push mirror, it would be possible to let it be pushed from a second source into a different directory, such as /testing-security/
<aj> hrm
<Joey> However, that would require all architectures to be supported and not just some.
<Joey> all as in “all that are in testing”
<aj> ah, that’s (c) then – they don’t do enough for the official debian stamp
<Joey> ah ok

Assuming those really are the key factors (which might or might not turn out true, but they seem likely) then that’s something that can be worked on. The first point is completely out of my domain – I’ve never been on vendor-sec, and it’s not likely something that ought to be tweaked much anyway; vendor-sec’s a very closed list by design, and for most security work it’s probably a distraction (testing security as is currently operates without concerning itself with vendor-sec, AIUI); the second points purely technical; and the third point is a combination of technical issues and an issue of training, both of which will be hopefully helped a lot by integrating testing-security and security.debian.org.

So the easiest point of attack is obviously the second one, since it’s “just” a matter of expanding dak’s featureset.

The problem here is embargoed fixes – the security team needs to be able to prepare fixes for vulnerabilities revealed on vendor-sec, but then keep them secret from everyone not on vendor-sec until the publication date. Which is fine, except that dak takes an all or nothing approach to that sort of secrecy: secuirty uploads go into accepted, get built, and then the security team run “amber” to publish the update. But what that means is the accepted directory and its contents can only be visible to people with vendor-sec access, which means only people with vendor-sec access can do security updates.

But we do actually have a different queuing mechanism that allows us to keep some things secret, while not hiding everything: thanks to the crypto-in-main transition, the NEW queue acts like that, more or less, only allowing ftpmaster to see the contents of NEW packages until they’ve passed license inspections and the appropriate export notifications have taken place. And actually, we’d already considered extending that sort of behaviour to add an “unapproved” queue that would prevent uploads from entering proposed-updates without the stable release manager’s prior approval.

The idea then, would be to have a couple of queues for the security team: an embargoed queue, accessible only to those with vendor-sec access that would store embargoed fixed ‘til they’re ready to be released, and an unembargoed queue for preparation accessible by a wider group, for preparation of fixes that don’t need to be secret. Both queues would need to be autobuilt (and the build logs for embargoed uploads would need to stay restricted, while the build logs for unembargoed uploads would not), which is an added difficulty, since only accepted can be autobuilt presently. But otherwise, it seems basically plausible, both to myself and Joey.

Conveniently, Andrew Pollock’s been interested in throwing some money through the AJ Market to see what happens, and since the initial idea we were tossing around fell through due to his move to the US, he’s put up the dosh to make the above happen. Well, strictly the dosh to make me make the above happen, or something.

Anyway, I’m already running behind both in implementation and blogging about it, so back to the grindstone…

(Okay, perhaps it’s not quite the feature Mark Twain would most like added to dak, but it’s still nifty.)

Hacking dak

Who can resist a good rhyme? Or a bad one?

So this round of dak hacking turned out to make the AJ Market scheme another notch more confusing – hence the delay in blogging, and the teaser in my last post. The issue leading to the confusion is that the major item on the list to start hacking on was SCC, which unlike the projects I’ve undertaken so far, is more than a day’s hacking. In fact, due to the need to give mirrors a chance to adapt to the new system between it being implemented and actually used, it’s actually more of a multi-week task. And doing it as a one-day-a-week project would extend that into a multi-month task afaics. I guess that’s still better than never, but obviously it’s worth looking into alternatives.

Naturally, then, the first phase of a longer project like this is threefold. We’ll call it “the three P’s”.

Planning

My theory at this point was to come up with a plan for what to do, try figuring out how much work it’d take, and then see what sort of financial arrangements might be plausible – not involving me cutting a few weeks out of my life for spare change, but without making the whole thing an unleapable chasm from what the AJ Market’s currently managing either. I figured writing it up as a semi-formal proposal makes most sense:

Summary

Implementation of the outstanding mirror split proposal for the Debian archive to allow new architectures, particularly amd64, to be included in the archive.

Benefit

In spite (or perhaps because) of its simplicity, this project has been languishing for over two years, and is not currently being worked on; so at present it’s not even possible to estimate when it would otherwise be completed.

It is most notably preventing amd64 from being integrated into the normal Debian development environment, causing derived distributions to maintain amd64 specific patches themselves.

In the longer term, reducing the constraints imposed on the archive size may allow the introduction of additional suites, such as backports or volatile, as well as additional architectures; though significant further discussion on this would be needed.

Background

Since at least mid-2003 the Debian archive has been closed to new architectures due to the already large amount of space and bandwidth required to become a Debian mirror. At present, the archive uses some 158GiB of disk, and about 1GiB per day; additional architectures are expected to require approximately an additional 10GiB each, and there are likely around half a dozen architectures that will be considered for addition once the moratorium on new architectures is rescinded (incl amd64, armeb, sh variants, kfreebsd and possibly partial architectures for arch variants such as s390x and ppc64).

The primary work needed to fix this involves:

• ensuring the mirror network operates correctly when a majority of mirrors are partial; this reduces the impact on bandwidth and storage capacity
• optimising portions of the archive maintenance software, particularly apt-ftparchive; this reduces the load on the archive server
• providing appropriate guidelines on the qualification criteria new architectures need to meet in order to be added to the archive; this provides a limit on future increases, allowing growth to be appropriately controlled

Actual work

I expect there will be six phases to the project:

1. cleanup of the archive as it stands, and establishing a clear categorisation of its contents to define what a partial mirror by architecture or suite should officially contain
2. providing appropriate scripts to ensure mirror sites can easily comply with the previously defined expectations for partial mirroring
3. devise an appropriate structure for the new mirror network, that can easily incorporate existing mirrors, and coexist with the existing structure
4. provide information on the new structure to both mirror admins and users; assist with the transition, and resolve any problems found
5. ensure the archive management software is appropriately optimised, and that archive inclusion criteria have been debated and established
6. add new ports that have passed the qualification requirements to the archive

In theory, a couple of days for each of those sound plausible, so making that twelve days actual work (with a couple of week’s delay in between for mirrors to have some time to adapt to the new network). On the downside, twelve days at a day a week is over three months of real time, not counting the possibility of doing other things with the one day a week, or Christmas, or the aforementioned delay for mirrors. Yick.

So much for planning.

Preparation

So the next “p” is preparation. In this case that’s finally getting around to fix dak CVS, which has been slightly broken since May. The extent of the actual breakage was just the loss of the ChangeLog history, aiui (or at least, that was the unrecovered breakage), but the result of that was months of uncommitted changes on both ftp-master and security (and reportedly from Ubuntu’s dak installation too). The changelog for the first set of commits (not counting buildd changes from ftp-master, security changes or Ubuntu changes that haven’t made it to ftp-master) looks like:

        * tiffani: new script to do patches to Packages, Sources and Contents
        files for quicker downloads.
        * ziyi: update to authenticate tiffani generated files

        * dak: new script to provide a single binary with less arbitrary names
        for access to dak functionality.

        * cindy: script implemented

        * saffron: cope with suites that don't have a Priority specified
        * heidi: use get_suite_id()
        * denise: don't hardcode stable and unstable, or limit udebs to unstable
        * denise: remove override munging for testing (now done by cindy)
        * helena: expanded help, added new, sort and age options, and fancy headers
        * jennifer: require description, add a reject for missing dsc file
        * jennifer: change lock file
        * kelly: propogation support
        * lisa: honour accepted lock, use mtime not ctime, add override type_id
        * madison: don't say "dep-retry"
        * melanie: bug fix in output (missing %)
        * natalie: cope with maintainer_override == None; add type_id for overrides
        * nina: use mtime, not ctime

        * katie.py: propogation bug fixes
        * logging.py: add debugging support, use | as the logfile separator

        * katie.conf: updated signing key (4F368D5D)
        * katie.conf: changed lockfile to dinstall.lock
        * katie.conf: added Lisa::AcceptedLockFile, Dir::Lock
        * katie.conf: added tiffani, cindy support
        * katie.conf: updated to match 3.0r6 release
        * katie.conf: updated to match sarge's release

        * apt.conf: update for sarge's release
        * apt.conf.stable: update for sarge's release
        * apt.conf: bump daily max Contents change to 25MB from 12MB

        * cron.daily: add accepted lock and invoke cindy  
        * cron.daily: add daily.lock
        * cron.daily: invoke tiffani
        * cron.daily: rebuild accepted buildd stuff
        * cron.daily: save rene-daily output on the web site
        * cron.daily: disable billie
        * cron.daily: add stats pr0n

        * cron.hourly: invoke helena

        * pseudo-packages.maintainers,.descriptions: miscellaneous updates
        * vars: add lockdir, add etch to copyoverrides
        * Makefile: add -Ipostgresql/server to CXXFLAGS

        * docs/: added README.quotes
        * docs/: added manpages for alicia, catherine, charisma, cindy, heidi,
        julia, katie, kelly, lisa, madison, melanie, natalie, rhona.

        * TODO: correct spelling of "conflicts"

Ugh. Still, that’s enough to start work.

And the final “P”? Come on, be honest with yourself, you know what it’s going to be.

Procrastination

Okay, that’s not entirely fair; the irrelevant bit of work was actually on to TODO list before SCC (mostly because it was something that I could get done reasonably quickly) and in fact was this line of the above changelog:

        * dak: new script to provide a single binary with less arbitrary names
        for access to dak functionality.

All the various model/actress names have been getting more than a little confusing recently, with almost forty in the dak suite, and another two dozen or so in use elsewhere – and then there’s the fact that the whole hot babes thing is both a bit offensive, and getting a bit old. OTOH, you need something to rename them to. We ended up deciding on the “version control solution” and introducing a “dak” command that’d launch all the different little bits of functionality depending on arguments, in the same way cvs, svn, tla, bzr, darcs etc do.

The implementation’s kinda neat: we have a list of commands (like “ls”) and their description (“Show which suites packages are in”), along with the python module and function they’re in (“madison”, “main()”). That let’s us not actually have to change any of the other scripts immediately, and lets “dak ls foo” work the same as “madison foo”. It also means that down the track we don’t need to have separate modules for each subcommand, and that we can rename modules and functions without affecting the user interface.

Of course it also means that all the internal scripts haven’t changed to use the new names yet, leaving the new interface a bit underused, but hey, “dak ls” at least manages to be one character shorter than “madison”, so that’s a win!

To be continued…

Tiffani

So this week’s project was working on dak, in particular getting the tiffani implemention included. What’s tiffani, you ask? It lets you just download the changes to Sources.gz and Packages.gz files, instead of the whole damn thing – if you have main, contrib, and non-free for unstable in your sources.list, this means your apt-get update only needs to download 27.5 KiB instead of 2.85 MiB (for bz2) or 3.75 MiB (for gz). Two orders of magnitude saving’s not bad.

I’ve mentioned this previously a few times if you want to see the history, in January I noted some quick hacks which have now been obsoleted by Michael Vogt’s integration into apt (which is in a personal repository, not uploaded to unstable yet…), back in August last year I did a little promotion of the concept, referring back to my post in December 2003 that summarised the analysis done previously, which takes us back to March and April 2002.

The server-side implementation, tiffani, is thus a little python script that groks through the list of suites and components dak knows abouts, looks for Packages, Sources and Contents files, and creates a directory names something like Packages.diff which in turn contains an Index file and a number of gzipped patches, named after the time they were created. The process for updating an existing Packages file is to look for the matching SHA1-History entry in the Index file, download the corresponding patch, apply that patch to your existing Packages file, and repeat until your Packages file matches the one you were looking for (which you can determine by looking at the checksum in either the Release or Index file).

Anyway, tiffani’s been co-developed with Andreas Barth (aka Andi) after the following exchange back in November 2003 on the RM channel:

<aba> elmo: If I find some time to take a look at katies TODO-file, what would be most appreciated? Deleting udebs with melanie? “Right” handling of orig.tar.gz when jumping out of NEW / changing of the archive section? Something else?
<elmo> aba: the TODO list is a bit of a mess to be honest
<aba> elmo: So, what should I do if I want to help the ftp-masters with katie?
<elmo> aba: I wouldn’t necessarily do anything based off that unless it’s obviously correct/wanted. none of the really urgent stuff is even on there (e.g. testing-security changes)
<aba> elmo: If you tell me something urgent, I can put some time into it.
<aj> hrm, surely i have some wishlists
<aba> urgent wishlist? :)
<aj> actually, you could critique something for me if you’ve got half an hour free now?
<aba> “critique” means?
<elmo> tell him how crap his ideas are. as brutually as possible
<elmo> ;)

…

<aj> ooo!
<aj> aba: around?
<aj> aba: one useful thing you could work on that’s kinda katie related is implementing pdiffs into katie/apt-ftparchive/apt/whatever
<Kamion> aj: mvo’s been working on that over the last couple of days, see #128818 logs
<aj> apt side or katie side?
<Kamion> apt side
<aj> cool
<aj> aba: hacking up a katie-side implementation for that would probably be interesting, worthwhile and straightforward

(aj = me, aba = Andi, elmo = James Troup, Kamion = Colin Watson)

That ended up with me finding an old script I made up called update.py (from Dec 2003 apparently) that did the core patch and Index generation, Andi updating that to some more sane pythonic style and making it work on an actual archive, then both of us cleaning it up so it would actually work sensibly without handholding.

(For those playing along at home, the idea needing critiquing was trying to fix up dak’s idea of what should happen if a new version of a package is uploaded to stable or testing, when it hasn’t changed in unstable. At the time, dak would REJECT it, because it violated the requirement that you can’t downgrade packages between stable and testing or testing and unstable. That’s still somewhat broken, unfortunately)

TTBOMK, Tiffani’s named after Tiffani Amber Thiessen, apparently of Saved by the Bell and Beverly Hills 90210 fame. I’m not sure who of aba or elmo is more to blame for that one, though I guess it is at least a little more distinctive than update…

What Could Possibly Go Wrong?

James Troup and I are giving a talk at the Debian mini-conf at LCA in six minutes. We mostly finished the slides. What could possibly go wrong?

Quick Apt Updates

Continuing my “Today’s del.icio.us links” variant in my usual indolent fashion, an interesting post from the other week was this one on making apt-get update a bit quicker and more pleasant. It’s based on these ideas from a while ago, but now it actually works. (Well, actually it worked years ago when Ben Bell first implemented something along these lines, but that’s been dead for years, afaik. There’s some more discussion in Bug#128818 if you’re obsessive about reading background information, which, of course, you’re not)

Anyway, on the client side, you can make use of the whole concept with two scripts: “untiffani” and “apt-qupdate”. Note that the scripts below obsolete the ones in the mail cited above, which will no longer work.

Here’s untiffani:

Read the rest ...

Using Lisa

With the trend of posting daily del.icio.us links, I’ve been thinking of trying out a “daily mailing list posts”. Maybe I’ll get around to it sometime. Anyway, today’s mailing list post is this one about dodgy uploads of lib packages. It seems to be the season for it, since Matt Zimmerman did the same thing with libflac recently. Anyway, as that email mentions, I rejected the packages. That’s done with a tool from the “dak” suite called “lisa”, and everytime I use it, I tend to feel sorry for whichever poor sucker gets the message – heck, it’s not even enough to say “the package was rejected, sorry”, it’s instead shouted with a subject like “libofx_0.7.0-3_powerpc.changes REJECTED”.

So, with apologies to Ben Folds…

Read the rest ...

Reinventing the Wheel

Apparently the latest in a long line of folks reinventing wheels to make apt-get update more efficient is Steve McIntyre. I’ve blogged previously on the topic. I don’t really have much more to say than that; in comparison Steve’s proposal requires fancy downloading, changes to apt-ftparchive and changes to the Packages file format in order to handle removed packages. By requiring you to download the entire stanza for every updated package, it’ll also lose on efficiency in general, although it should gain on efficiency for people who update very infrequently.

Given pdiffs are conceptually more straightforward, can be trivially added to the archive by writing a couple of scripts, and have been successfully implemented in the past, I don’t see the point in reinventing this particular wheel instead of just building it as specced.

Incremental Packages.gz Downloads

One of the more annoying things about managing a Debian system is sitting around waiting for apt-get update to finish. On broadband it’s irritating, and on a modem it’s absolutely horrendous. The worst part is when it takes longer to do an update (get the information we need) than it takes to do a dist-upgrade (work out what to upgrade, and upgrade it).

Basically the problem is that we need to have a current list of available packages to work out what to upgrade to, but that list is absolutely huge, sitting at about 10MB uncompressed, or 2.0MB using bzip2 compression. On the upside we usually already have a list that was current not too long ago, and generally not all that much will have changed in the intervening time. So what we’d like to be able to do is just download the updates.

One way of doing this is using the rsync algorithm. Unfortunately that only works on uncompressed files, and it’s something of a brute force solution. It looks at a file in big chunks, so if you change one line, the entire chunk will be downloaded. That’s okay, and it’s reported to be quite useful, with downloads reduced to about 10% of what they were beforehand.

But we know what changes are likely, so we can optimise for that. Packages files are sorted by stanza (lines of non-blank lines separated by a single blank line), and usually only a few lines in any given stanza will change on an update (the Version: line might change, while the Description: lines don’t, eg). So we want a way of listing just the lines that change. The easy way to do that is with the diff --ed command, which produces a file saying things like:

6,7c
foo
bar
.

The above example says “replace lines 6-7, with two new lines saying foo and bar”. Brief and simple. It can be applied by saying (cat EDPATCH; echo w) | red - FILE. The patch program can also handle it, but has been flakey in the past.

So, how cool is that? Well, this isn’t a remotely new idea, and Robert Tiberius Johnson has done some analyses, back in the day. Basically, using the Packages files which were about two-thirds the size they are now, you can reduce the average apt-get update to be about 13kB per day since your last update. That’s as little as 0.7% what was previously required.