dak dsa

So the final implementation detail in the embargoing scheme is providing a tool to move stuff from the embargoed and unembargoed queues into the archive. The existing tool the security team use is called “amber” (after the inimitable Amber Benson). amber’s pretty simple: it takes a DSA number, and the .changes files you’re looking at; then asks for confirmation, accepts the packages into the archive, regenerates Packages and Release files, fills out a template advisory with details from the deb and mails that off, and uploads the files to ftp-master for inclusion in the next stable release.

There are a few problems with that. One is it doesn’t allow for rejections. Another is it doesn’t provide the security team with the opportunity to edit advisories while the packages are being prepared. Another issue is that the entire program has to run under sudo with full archive priveleges.

Enter “dak dsa” aka “newamber”. The new tool aims to do more or less the same thing, but with a little more style. First, it provides a small interactive interface, so that processing an advisory now looks like:

$ newamber DTSA-25-1 smb4k_*.changes
Create new advisory DTSA-25-1? y
Advisory: DTSA-25-1
Changes:
 smb4k_0.6.4-0.0etch1_alpha.changes
 smb4k_0.6.4-0.0etch1_hppa.changes
 smb4k_0.6.4-0.0etch1_i386.changes
 smb4k_0.6.4-0.0etch1_m68k.changes
 smb4k_0.6.4-0.0etch1_mips.changes
 smb4k_0.6.4-0.0etch1_mipsel.changes
 smb4k_0.6.4-0.0etch1_s390.changes
 smb4k_0.6.4-0.0etch1_sparc.changes
Packages:
 smb4k 0.6.4-0.0etch1 (alpha, hppa, i386, m68k, mips, mipsel, s390, source, sparc)
Approve, [E]dit advisory, Show advisory, Reject, Quit? 

Choosing edit will grab a copy of the template and run vi – the template will only be filled out when you approve the upload though, since the values might change before then. Accept will do more or less what old amber did; though instead of mailing the filled in advisory draft it’ll just leave it in the filesystem instead.

Of course, running vi (well, $EDITOR) generally means you can get a shell too, so running the command with full archive priveleges is a bit much – at least if you’re trying to have any sort of granularity to your security regime, which was, after all, the whole point of this exercise. So instead of running the entire command as the katie user, “dak dsa” instead has to escalate its own priveleges, in this case using sudo and specific options, such as sudo dak dsa -A -- foo.changes to approve foo.changes. Fortunately sudo and the apt argument parser are cooperative enough to allow “dak dsa” users to invoke “dak dsa -A – *” as katie, and thus have only the very limited access we’re trying for.

Obviously the above is taken from the testing-security team – it’s the same source and i386 packages, recompiled on other architectures by the security.debian.org testing autobuilders. It’s shown up a few flaws in the autobuilding for etch: (a) the amd64 autobuilder isn’t active; (b) the arm buildd can’t seem to find its chroot in between running apt-get install and apt-get remove; (c) the s390 buildd only works if the source is on ftp-master; (d) of the five m68k buildds that will take packages for security.debian.org updates to testing, only two will succeed (a400t and poseidon). There’s also the notable problem that the chroots for the functional buildds have gotten out of date and that builds break somewhat obscurely as a consequenece. One of the test updates is also failing on hppa due to space restrictions. And of course, the above list is after a chunk of other problems have already been fixed.

It’s worth noting that even if the above isn’t fixed for testing now; we’ll still need etch chroots for security.debian.org when we release, so those problems have to be dealt with at some point. And that the brokenness is the result of six months’ divergence from sarge; after a year and a half when etch releases – or the three years between woody and sarge’s release – it’s probably fair to expect worse breakage.

Anyway, that’s just about it from me on this topic. Micah Anderson from the testing-security group is currently checking out the unembargoed facility, and has redone a couple of DTSAs on security.d.o. So presumably those guys will start working out whether security.d.o is something they want to make use of, and if so, working out what changes/tweaks are necessary for that. Though, unsurprisingly, I also still have to do some committing to CVS…

Changing The Security Infrastructure

One of the most exciting things about working on Debian is that since it’s developed in the open, when you want to make changes everyone sees them, so depending on what you’re working on, the risk of breaking stuff can provide a real adrenaline rush, while you’re just sitting in your chair. So what better adventure sport could there be than hacking up Debian’s security support?

One idea that comes to mind: doing it with only a minimal idea how it works at the moment.

Read the rest ...

Security Infrastructure Changes

Delays suck.

I’m actually skipping ahead here, I’d meant to blog about updating the dak codebase into shape for whatever changes were coming up first, but it turns out I’m not in the mood for that. Besides which, most of this entry is preprepared. In the last few entries we went over the background for updating our security infrastructure to make it possible to have security team members that don’t worry about vendor-sec issues (process, overview, NIv2 aka NINJA, queue-build issues), all the while skipping fairly lightly over the question of what has to actually happen. That happens to be the way I prefer to work, lots of background tinkering, then spend as little time on actual implementation as possible; it has it’s drawbacks, but hey, I like it.

Anyway, the second last step is working out in detail what’s going to happen. The delay in question was trying to get the security team to sign off on it, but unfortunately they’re busy enough that that looks like it’ll being an indefinite delay, so I’m going with the “well, ftpmaster maintains the dak/katie infrastructure for the security team, so in the end it’s our call on what’s best anyway” line of thought, and just plunging in. Fortunately the current and final plan should make pretty minimal changes to how the existing security team operate anyway, even assuming a massive testing security operation happening in parallel with them – so the downside risk is already pretty minimal. (And it was, afterall, developed in consultation with Joey, and then shopped around a few folks who’ve been doing security work for testing and kernels)

What is that plan, you may well ask. It’s something like this:

First, let’s restate the idea:

Concept

Allow two teams to exist, the current security team as the “vendor-sec subteam”, and the full team which can expand to also include the testing-security people and others interested and able to help out with security work but who do not have access to vendor-sec.

Then, we want to think about the processes that are involved at a global level. First for uploads that can’t be made public until some later date, and can only be worked on by the “vendor-sec” enabled security team.

• Source upload to queue/unchecked (as now)
  • Upload is authenticated etc
  • Upload is moved to queue/embargoed
• Upload is autobuilt
  • Logs are sent to a procmail address
  • Logs are forwarded (by procmail) onto the vendor-sec subteam
  • Logs are signed and sent back
  • Binaries are uploaded to queue/unchecked
  • Binaries are moved to queue/embargoed
• Uploads (source + binaries) are approved from queue/embargoed by
• running amber
  • Uploads are moved into the archive
  • The archive is mirrored
• The approver issues an advisory

And likewise for issues that don’t need so much secrecy:

• Source upload to queue/unchecked-disembargo (new directory)
  • Upload is authenticated etc
  • Source+version added to unembargo table
  • Upload is moved to queue/unembargoed
• Upload is autobuilt
  • Logs are sent to a procmail address
  • Logs are forwarded (by procmail, after checking unembargo table) onto the full team
  • Logs are signed and sent back
  • Binaries are uploaded to queue/unchecked
  • Noting that the source is in the unembargo table, Binaries are moved to queue/unembargoed
• Uploads (source + binaries) are approved from queue/unembargoed by running amber
  • Uploads are moved into the archive
  • The archive is mirrored
• The approver issues an advisory

Processes are great, but they have to be supported by proper information handling, so it’s worth looking at what the above actually means for the on-disk structures, too. This is particularly important, since filesystem level security is mostly how we wall off vendor-sec stuff from other users of the system (whether they’re also working on security issues, or doing, eg, web page maintenance or something equally random). Voila:

queue/unchecked, queue/unchecked-dismebargo
    world writable
    accessible only by katie
    contents automatically cleared every 15m

queue/embargoed
    writable only by katie
    accessible by vendor-sec subteam
    amber allows packages to be ACCEPTed
    [...] allows packages to be REJECTed

queue/unembargoed
    writable only by katie
    accessible by full team
    amber allows packages to be ACCEPTed
    [...] allows packages to be REJECTed

queue/accepted
    packages only sit in accepted briefly while amber is running

And finally, to sum things up, this is what a member of the security team should expect to actually do to release an advisory:

• Upload package to queue/unchecked (or queue/unchecked-disembargo)
• Wait for build logs to be sent to your address
• Check and sign those build logs
• Check the final packages in queue/embargoed (or queue/unembargoed)
• Run amber over the packages to accept them
• Fill in the template advisory generated and send it to the -announce list

The differences between that process and the current one used by the security team is pretty minor, ttbomk: currently uploads sit in queue/accepted not queue/embargoed, and currently the security team ignore the template advisories amber generates preferring to make their own – aiui, this is mostly due to delays in actually receiving the advisory template since amber mails it instead of leaving it somewhere convenient on disk.

Anyway, that’s what the plan looks like. We should see how well it survives contact with the enemy, or at least reality, remarkably soon.

(Of course, I guess “political issues” could always foul it up completely anyway, particularly going by the fact that it’s been more or less the only problem raised by the folks I’ve passed this on to… Oh well, reconstruct that bridge when we get to it, I guess)

Queue Building

The biggest difference between the hypothetical security queues and the existing byhand and new queues is that they need to be autobuilt; it’s not really feasible to try saying whether a security update is acceptable if there’s still some possibility it’s not going to build on half our architectures. Additionally, it means that there are two queues that need to be built, rather than just one.

The way accepted autobuilding works isn’t too complicated, fortunately. There’s a directory where symlinks are made to uploaded packages and sources, and a Packages files (for all architectures) is generated, as is a Sources file, all of which are made available to buildds. They’re also added to the regular unstable Packages and Sources files and passed to quinn-diff and wanna-build. In order to avoid unnecessary breakage at transitional points (such as when a package moves from accepted to unstable proper), packages are kept in the accepted autobuild directory for a day or so afterwards. There’s also the issue of including the .orig.tar.gz from the archive along with the .dsc and .diff.gz from accepted for uploads that aren’t new upstream versions.

Security accepted autobuilding adds two extra wrinkles. The first is that packages being autobuilt can’t be generally visible to other users on the security host, which includes the web server that’s supposed to get the packages to the security buildds. So instead of symlinking, copies have to be made, and care has to be taken to ensure that doesn’t accidently leak information in the process. The second trick is that security needs autobuilding of multiple suites (oldstable and stable, and hopefully testing as well), whereas for the regular archive, it’s sufficient to autobuild unstable packages from accepted, and leave the rest until the daily pulse run. This particularly means that uploads from accepted need to be moved into separate directories depending on the target suite, so that the buildds don’t accidently build a package for woody against the libc from sarge.

At a gross level, generalising to supporting autobuilding of multiple queues is therefore straightforward: extend the “accepted_autobuild” table to include information of what queue the files are from (which implies a name change is appropriate), extend the configuration file to allow you to specify which queues should get autobuild handling for which suites, and finally add the code to actually do this when packages are moved into and out of the appropriate queues. Hurray for simplicity!

But there’s one little bit of complexity still to bother us. We know we need different directories and Packages files for different suites, but do we need it for different queues as well? Worse, we need different chroots on each autobuilder for each suite, do we need different chroots for each suite for each queue? The latter could well be a complete showstopper.

One reason why we might need to treat the queues we’re considering separately is that they have different secrecy requirements: embaroged updates can’t be visible to anyone, even people working on the unembargoed security updates in parallel, and yet the buildd has to forward its logs on to someone to be signed. If embargoed and unembargoed packages are built in the same chroot, by the same buildd, from the same queue, how do the logs magically go to the right person?

That’s not necessarily easily answered; currently the hope is that it’ll be okay to have a single shared queue and set of buildds and chroots, and have all the logs forwarded to a single address, at which point they can be redirected to the appropriate security team members by procmail. The assumption this is based on is that there’ll already need to be some magic to do that for the builds themselves once they’re uploaded and need to be moved into either the embargoed or unembargoed queue, so the same thing should be possible with the build logs.

Pending further surprises though, that’s enough to move onto the next phases, which is generalising dak to cope with arbitrary queues.

The NIv2 Plot

So having found time to catch up satisfactorily on the implementation, time to get back into the blogging.

After working out what you actually want to do, the next step in implementing stuff, in my book, is to make sure you fully understand the context of the stuff you’re trying to change. In this case, the “unapproved” queue concept comes from some thoughts from quite a while ago that come under the heading “NIv2” for “new incoming v2”. For reference, the original “new incoming” was the switch to separate queues for new, byhand, and accepted packages, which was followed by a notable update shortly after to make accepted autobuilding possible.

That setup works, but has a few problems. One is that packages sit in accepted for quite a while (up to a full 24h), and occassionally in that time it can become inappropriate to actually add the package to the archive – for instance if it’s a build of a package that’s then removed from the archive. This results in an UNACCEPT, and requires manual intervention, since potentially it can result in weird inconsistencies, such as bug closures. Another is that files in the various queues aren’t tracked in the database, which can result in files (particularly upstream tarballs) being lost or left behind, and also requires some unpleasant special casing in the code. A final concern is that one of the bigger time sinks in the daily pulse is the apt-ftparchive scan, which is actually just doing work that’s already been done in order to allow packages in accepted to be autobuilt.

Put all those concerns together with the ones mentioned last week, and you get the impetus for NIv2. The principle of that is to put stuff in the queue into the database properly, simplify it, extend the queuing support to deal with different queues, and move stuff from accepted directly into the pool every quarter hour, though only pushing to buildds that frequently, leaving updates of Packages files and mirror pushes to still be daily, or every few hours or whatever.

This is what that bit looked like in the dak TODO:

[hard, long term] unchecked -> accepted should go into the db, not a suite, but similar. this would allow katie to get even faster, make madison more useful, decomplexify specialacceptedautobuild and generally be more sane. may even be helpful to have e.g. new in the DB, so that we avoid corner cases like the .orig.tar.gz disappearing ‘cos the package has been entirely removed but was still on stayofexecution when it entered new.

Obviously, it needed to get defined a little more clearly than that to work out how much the changes for the security queues I’m interested in depend on the other changes, and what they actually mean in detail. After some IRC discussion we came up with more or less the following summary of what NIv2 actually means:

How NIv2 should work.

q/accepted goes away; once packages are accepted, they get added straight into the pool and the database. At this point they’re added to a micro-suite for the buildds, and so apt-ftparchive can cache the package contents; and a symlink tree is made up so incoming.d.o works. The micro-suite might or mightn’t be a fully fledged suite in the database. Presumably it wouldn’t be visible in dists/ though. The micro-suite probably should contain packages uploaded to unstable for X hours, where X is 48 or 2x pulse-interval or similar.

This means UNACCEPT is impossible and apt-ftparchive takes less time in cron.daily.

q/byhand and q/new get new friends: q/sec-embargoed, q/sec-unembargoed, q/stable-updates. Rather than checking for “byhand” entries, or a lack of overrides; uploads get diverted to those queues if they’re uploaded to security or stable, and get moved out of there when a security team member or stable RM (ie, Joey) runs an amber/lisa-esque command.

This means sec-embargoed and sec-unembargoed need to be autobuilt.

This also means that proposed-updates can be a full suite, and point releases can be a matter of (a) adding proposed-updates to stable; (b) removing old versions from stable; (c) removing packages that have to be dropped; and users can put “stable + proposed-updates” in their sources.list reasonably sanely.

This also means that REJECTs from accepted (security) or the archive (p-u) aren’t needed.

It means the queue ends up looking like:

    unchecked      [uploads go here, insecure]

    holding        [secured]
    new            [waiting NEW processing]
    byhand         [waiting byhand processing]
    stable-updates [waiting stable RM approval]

    done           [.changes files for accepted packages]
    reject         [.changes, .reason and rejected stuff]

    bts_version_track [data for BTS]

with q/accepted disappearing and q/proposed-updates and q/old-proposed-updates being replaced by the stable-updates / oldstable-updates queues.

One of the nice parts about this is that it breaks down fairly neatly into three sets of changes: supporting additional queues, removing the accepted queue, and tracking queue stuff in the database. It does make those changes more or less dependent though – the new queues need to be autobuilt in the way accepted is at present; so it makes sense to add the queues before removing accepted so as not to lose the queue autobuild support.

The above actually can be simplified a little further – instead of managing a microsuite for changes since the last update, it’s possible to add accepted packages directly to the real suite; and generate the Packages files needed for autobuilding by querying packages accepted to unstable in the last 24 or 48 hours, simply by adding a field to track upload time for binaries as well as sources.

In any event, this thus makes generalising the queue autobuild support the next challenge.

Afraid of your Neighbour's Disapproval

Continuing on from the SCC stuff, which I should probably get in the habit of calling the mirror split, then…

At the same time as I’ve been trying to work out ways to fit the mirror split stuff into the AJ market concept, I’ve been pondering on and off what can be done to improve the security situation. My thinking’s probably still biassed by my long time wish for official security updates to testing, but I’m somewhat inclined to think that if we can figure out a way of integrating the testing security team and the official security team we might end up closer to a solution – at the very least that’d provide us with a decent number of people who can actively help out with security updates that aren’t secet and get a few more people familiar with the technical aspects of how security updates are released, who’ll then have less of a learning curve if they end up being added to the security team proper. And heck, even if that doesn’t work, it at least gives people running testing an easier url to remember for security updates.

Of course, integrating the teams isn’t that easy, or it would’ve already been done. So after thinking about it, I had a chat with Joey (aka Martin Schulze, aka the guy who’s issued 317 of the 340 security advisories in the past 12 months) about the problems:

<aj> so, the other thing i wonder is what you think of the testing-security-support split?
<Joey> need more input
<aj> as in separate host, funny domain name, not available on security.debian.org/, extra effort to mirror, etc
<aj> i think it sucks :)
<Joey> there are a lot of sucking things…
<aj> it’s true
<aj> so, i figure the reasons for the split are (a) you don’t want to add a bunch of people to team@ and hence vendor-sec, and (b) there’s no way to make it so they can use dak-security without being full team/vendor-sec members; and maybe (c) you don’t want them officially associated with debian security coz they’re newbies
<Joey> a) and b) are true
<Joey> However, with security.debian.org being a push mirror, it would be possible to let it be pushed from a second source into a different directory, such as /testing-security/
<aj> hrm
<Joey> However, that would require all architectures to be supported and not just some.
<Joey> all as in “all that are in testing”
<aj> ah, that’s (c) then – they don’t do enough for the official debian stamp
<Joey> ah ok

Assuming those really are the key factors (which might or might not turn out true, but they seem likely) then that’s something that can be worked on. The first point is completely out of my domain – I’ve never been on vendor-sec, and it’s not likely something that ought to be tweaked much anyway; vendor-sec’s a very closed list by design, and for most security work it’s probably a distraction (testing security as is currently operates without concerning itself with vendor-sec, AIUI); the second points purely technical; and the third point is a combination of technical issues and an issue of training, both of which will be hopefully helped a lot by integrating testing-security and security.debian.org.

So the easiest point of attack is obviously the second one, since it’s “just” a matter of expanding dak’s featureset.

The problem here is embargoed fixes – the security team needs to be able to prepare fixes for vulnerabilities revealed on vendor-sec, but then keep them secret from everyone not on vendor-sec until the publication date. Which is fine, except that dak takes an all or nothing approach to that sort of secrecy: secuirty uploads go into accepted, get built, and then the security team run “amber” to publish the update. But what that means is the accepted directory and its contents can only be visible to people with vendor-sec access, which means only people with vendor-sec access can do security updates.

But we do actually have a different queuing mechanism that allows us to keep some things secret, while not hiding everything: thanks to the crypto-in-main transition, the NEW queue acts like that, more or less, only allowing ftpmaster to see the contents of NEW packages until they’ve passed license inspections and the appropriate export notifications have taken place. And actually, we’d already considered extending that sort of behaviour to add an “unapproved” queue that would prevent uploads from entering proposed-updates without the stable release manager’s prior approval.

The idea then, would be to have a couple of queues for the security team: an embargoed queue, accessible only to those with vendor-sec access that would store embargoed fixed ‘til they’re ready to be released, and an unembargoed queue for preparation accessible by a wider group, for preparation of fixes that don’t need to be secret. Both queues would need to be autobuilt (and the build logs for embargoed uploads would need to stay restricted, while the build logs for unembargoed uploads would not), which is an added difficulty, since only accepted can be autobuilt presently. But otherwise, it seems basically plausible, both to myself and Joey.

Conveniently, Andrew Pollock’s been interested in throwing some money through the AJ Market to see what happens, and since the initial idea we were tossing around fell through due to his move to the US, he’s put up the dosh to make the above happen. Well, strictly the dosh to make me make the above happen, or something.

Anyway, I’m already running behind both in implementation and blogging about it, so back to the grindstone…

(Okay, perhaps it’s not quite the feature Mark Twain would most like added to dak, but it’s still nifty.)