On freedom

One of the freedoms I value is the freedom to choose what you spend your time on and who you spend it with. And while I’ve spent a lot of time arguing that people in key roles in Debian still have those freedoms (hey, 2.1(1), don’t you know), reality these days seems to be otherwise. But hey, solving that quandry just requires a mail to DSA.

To folks on the core teams I’ve been involved with: it’s been a pleasure and an honour working with you; if not always, at least mostly. Best of luck, and I hope y’all accept patches.

Jigdo Downloads

Last month we had a brief discussion on debian-devel about what images would be good to have for lenny – we’re apparently up to about 30 CDs or 4 DVDs per architecture, which over 12 architectures adds to about 430GB in total. That’s a lot, given it’s only one release, and meanwhile the entire Debian archive is only 324GB.

The obvious way to avoid that is to make use of jigdo – which lets you recreate an iso from a small template and the existing Debian mirror network. I’ve personally never used jigdo much, half because I don’t usually use isos anyway, but also because the few times I have tried jigdo it always seemed really unnecessarily slow. So the other day I tried writing my own jigdo download tool focussed on making sure it was as fast as possible.

The official jigdo download tool, ttbomk, is jigdo-lite – which you give a .jigdo file, and the url of a local mirror. It then downloads the first ten files using wget, and once they’re all downloaded, it calls jigdo-file to get them merged into the output image. This gets repeated until all the files have been downloaded.

By doing the download in sequence like this, you miss out on using your full network connection in two ways: one during the connection setup latency when starting to download the next package, and also while jigdo-lite stops downloading to run jigdo-file. And if you’ve got a fast download link, but a slower CPU or disk, you can also find yourself constrained in that you’re maxing those out while running jigdo-file, but leaving them more or less idle while downloading.

To avoid this, you want to do multiple things at once: most importantly, to be writing data to the image at the same time as you’re downloading more data. With jigdodl (the name I’ve given to my little program), I went a little bit overboard, and made it not only do that, but also manage four downloads and the decompression of the raw data from the template. That’s partly due to not being entirely sure what needed to be done to get a speedy jigdo program, and partly because the communicate module I’d just written to deal with this sort of parallelism making that somewhat natural.

In the end, it works: from wireless over ADSL to my ISP’s Debian mirror, I get the following output:

Jigsaw download:
  Filename: debian-40r3-amd64-CD-1.iso
  Length:   675477504
  MD5sum:   d3924cdaceeb6a3706a6e2136e5cfab2
Total: 679 s; d/l: 586 MB at 883 kB/s; dump: 57 MB at 57 MB/s          

Finished!

which is only slightly short of maxing out my downstream bandwidth, taking a total of about 11m20s. Running jigdodl with a closer mirror works pretty well too, though evidently some of my more recent changes weren’t so great, because I’ve gone from 9153 kB/s on a 100 Mbps link down to 7131 kB/s or lower. The CPU usage also seems a bit high, hovering at between five to ten percent at 900 kB/s.

For comparison, running jigdo-lite on the same file took 17m41s, which is about 566 kB/s, with the overhead being about 6m20s. What that means is if I doubled my bandwidth to about 20Mbps, jigdodl would halve its time for the download to about 5m50s, while jigdo-lite would still have about the same non-download overhead, and thus take 12m10, which is still 69% of its original speed. Going from 10Mbps ADSL speed to 100Mbps LAN gets jigdodl down to 1m31s (13% of the time, with optimal being 10%), while jigdo-lite would be expected to still be about 7m51s (43% of its original time).

I suspect the next thing to do is to rewrite the downloading code to use python-curl instead of running curl, and thus downloading multiple files with a single connection, and tweaking the code so that it writes the file in order, rather than updating whichever parts are ready first.

Anyway, debs are available for anyone who wants to try it out, along with source in the new git source package format.

A New DPL...

In a couple of days, DPL-elect Steve McIntyre takes over as DPL, after being elected by around four hundred of his peers… Because I can’t help myself, I thought I might poke at election numbers and see if anything interesting fell out.

First the basics: I get the same results as the official ones when recounting the vote. Using first-past-the-post, Steve wins with 147 first preference votes against Raphael’s 124, Marc’s 90 and NOTA’s 19 (with votes that specify a tie for first dropped). Using instant-runoff / single transferable vote, the winner is also Steve, with NOTA elimited first and Marc collecting collecting 5 votes, Steve 4 and Raphael 2, followed by Marc getting eliminated with Steve collecting 50 votes, against Raphael’s 26.

So, as usual, different voting systems would have given the same result, presuming people voted in basically the same way.

NOTA really didn’t fare well at all in this election, with a majority of voters ranking it beneath all candidates (268 of 401, 53.5%). For comparison, only 18 voters ranked all candidates beneath NOTA, with 9 of those voters then ranking all candidates equally. (For comparison, in 2007, 312 of 482 voters (about 65%) ranked some candidate below NOTA, though that drops to 225 voters (47%) if you ignore voters that just left some candidates unranked. Only 98 voters (20%) voted every candidate above NOTA)

With NOTA excluded from consideration, things simplify considerably, with only 13 possible different votes remaining. Those come in four categories: ranking everyone equal (17 votes, 9 below NOTA as mentioned above, and 8 above NOTA), ranking one candidate below the others (13 votes total, 7 ranking Raphael last, 3 each for Steve and Marc), ranking one candidate above the others (66 votes; 30 ranking Steve first, 18 each ranking Raphael and Marc first), and the remainder with full preferences between the candidates:

     70 V: 213
     63 V: 123
     56 V: 132
     52 V: 231
     38 V: 312
     26 V: 321

The most interesting aspect of that I can see is that of the people who ranked Raphael first, there was a 1.8:1 split in preferring Steve to Marc, and for those who preferred Marc first, there was a 2:1 split preferring Steve to Raphael. For those who preferred Steve, there was only a 1.1:1 split favouring Raphael over Marc.

I think it’s fair to infer from that that not only was Steve the preferred candidate overall, but that he’s considered a good compromise canidate for supporters of both the alternative candidates (though if all the people who ended up supporting Steve hadn’t been voting, Raphael would have won by something like 26 votes (129:103) with a 1.25:1 majority; if they had been voting, but Steve hadn’t been a candidate, Raphael’s margin would’ve increased absolutely to 33 votes (192:159) but decreased in ratio to a 1:1.2 majority.

Dak Extensions

One of the challenges maintaining the Debian archive kit (dak) is dealing with Debian-specific requirements: fundamentally because there are a lot of them, and they can get quite hairy – yet at the same time, you want to keep them as separate as possible both so dak can be used elsewhere, and just so you can keep your head around what’s going on. You can always add in hooks, but that tends to make the code even harder to understand, and it doesn’t really achieve much if you hadn’t already added the hook.

However, dak’s coded in python, and being an interpreted language with lots of support for introspection, that more or less means there’s already hooks in place just about everywhere. For example, if you don’t like the way some function in some other module/class works, you can always change it (other_module.function = my_better_function).

Thus, with some care and a bit of behind the scenes kludging, you can have python load a module from a file specified in dak.conf that can both override functions/variables in existing modules, and be called directly from other modules where you’ve already decided a configurable hook would be a good idea.

So, at the moment, as a pretty simple example there’s an init() hook invoked from the main dak.py script, which simply says if userext.init is not None: userext.init(cmdname).

But more nifty is the ability to replace functions, simply by writing something like:

# Replace process_unchecked.py's check_signed_by_key
@replace_dak_function("process-unchecked", "check_signed_by_key")
def check_signed_by_key(old_chk_key):
    changes = dak_module.changes
    reject = dak_module.reject
    ...
    old_chk_key()

That’s made possible mostly by the magic of python decorators – the little @-sign basically passes the new check_signed_by_key function to replace_dak_function (or, more accurately, the function replace_dak_function(...) returns), which does the dirty work replacing the function in the real module. To be just a little bit cleverer, it doesn’t replace it with the function we define, but its own function with simply invokes our function with an additional argument to whatever the caller supplied, so we can invoke the original function if we choose (the old_chk_key parameter – the original function takes no arguments, so our function only takes one).

Right now, we don’t do much interesting with it; but that should change once Ganneff’s little patch is finished, which should be RSN…

Hopefully, this might start making it easier to keep dak maintained in a way that’s useful for non-Debian installs – particularly if we can get to the point where hacking it for Debian generally just implies changing configuration and extension stuff – then we can treat updating all the real scripts as a regular software upgrade, just like it is outside Debian.

The second half...

Continuing from where we left off…

The lower bound for me becoming a DD was 8th Feb ‘98 when I applied; for comparison, the upper bound as best I can make out was 23rd Feb, when I would have received this mail through the debian-private list:

Resent-Date: 23 Feb 1998 18:18:57 -0000
From: Martin Schulze 
To: Debian Private 
Subject: New accepted maintainers

Hi folks,

I wish you a pleasant beginning of the week.  Here are the first good
news of the week (probably).

This is the weekly progress report about new-maintainers.  These people
have been accepted as new maintainer for Debian GNU/Linux within the
last week.

[...]

Anthony Towns <ajt@debian.org>

    Anthony is going to package the personal proxy from
    distributed.net - we don't have the source... He may adopt the
    transproxy package, too.

Regards,

        Joey

I never did adopt transproxy – apparently Adam Heath started fixing bugs in it a few days later anyway, and it was later taken over by Bernd Eckenfels (ifconfig upstream!) who’s maintained it ever since. Obviously I did do other things instead, which brings us back to where we left off…

Read the rest ...

Been a while...

So, sometime over the past few weeks I clocked up ten years as a Debian developer:

From: Anthony Towns <aj@humbug.org.au>
Subject: Wannabe maintainer.
Date: Sun, 8 Feb 1998 18:35:28 +1000 (EST)
To: new-maintainer@debian.org

Hello world,

I'd like to become a debian maintainer.

I'd like an account on master, and for it to be subscribed to the
debian-private list.

My preferred login on master would have been aj, but as that's taken
ajt or atowns would be great.

I've run a debian system at home for half a year, and a system at work
for about two months. I've run Linux for two and a half years at home,
two years at work. I've been active in my local linux users' group for
just over a year. I've written a few programs, and am part way through
packaging the distributed.net personal proxy for Debian (pending
approval for non-free distribution from distributed.net).

I've read the Debian Social Contract.

My PGP public key is attached, and also available as
<http://azure.humbug.org.au/~aj/aj_key.asc>.

If there's anything more you need to know, please email me.

Thanks in advance.

Cheers,
aj

-- 
Anthony Towns <aj@humbug.org.au> <http://azure.humbug.org.au/~aj/>
I don't speak for anyone save myself. PGP encrypted mail preferred.

On Netscape GPLing their browser: ``How can you trust a browser that
ANYONE can hack? For the secure choice, choose Microsoft.''
        -- <oryx@pobox.com> in a comment on slashdot.org

Apparently that also means I’ve clocked up ten and a half years as a Debian user; I think my previous two years of Linux (mid-95 to mid-97) were split between Slackware and Red Hat, though I couldn’t say for sure at this point.

There’s already been a few other grand ten-year reviews, such as Joey Hess’s twenty-part serial, or LWN’s week-by-week review, or ONLamp’s interview with Bruce Perens, Eric Raymond and Michael Tiemann on ten years of “open source”. I don’t think I’m going to try matching that sort of depth though, so here are some of my highlights (after the break).

Read the rest ...

Exclusion and Debian

Oh yay, another argument about sexism. I thought we were over this. Aigars writes:

Trying to restrict what words people can or can not use (by labeling them sexist, racist or obscene) is the bread and butter of modern day media censorship. It is censorship and not “just political correctness”. While I would not want people trying to limit contributions to Debian only to “smart and educated white people” (racism) or “logically thinking males” (sexism), going the other way and excluding people from Debian because their remarks or way of thinking might offend someone is just offensive to me.

One: it’s not censorship for Debian to limit discussion of various things on Debian channels. It’s censorship when you prevent discussion of something anywhere.

Two: if you think excluding people is bad, then supporting jerks whose mysogyny repulses people isn’t compatible with that.

Three: doing something in someone else’s name, that isn’t supported by them, is wrong. If you’re in a channel called “debian-something” don’t act in ways that don’t match Debian’s goals. If you want to be free to go against the principles of the DFSG by, eg, discriminating against people or groups, make up your own name for a channel.

Asus eeePC

Okay, so any excuse to bring out the Laphroaig is fine by me, but the cute little eeePC is better than most. That it’s cute and popular is all very well, but what really makes my day is this is the first device I’ve seen that both doesn’t hide the fact it’s running Linux, and is available in maintstream stores in Australia. A random review from the Sydney Morning Herald:

You also don’t get Windows. Asus has adopted the free Linux operating system that’s been slowly yet steadily growing in popularity over the past decade. This keeps the cost down and makes better use of its relatively modest hardware, which would creak under the weight of Windows. It does mean not being able to use your favourite Windows software but, fortunately, the Eee PC comes with dozens of programs, including the familiar Firefox web browser and Skype for online phone calls.

That came out on the day the eeePC was released in Australia. Three days later, they did a followup:

Taiwan computer maker Asus might have underestimated the local demand for its diminutive Eee PC, as the $499 laptop is now virtually sold out in Australia.

From that, there’s a brief take on who’s actually buying them:

“We’ve had customers coming in buying two or three units for the family - the mix of customer has been probably novices more than the tech types,” he said.

It’s really good to see that Linux is getting credit in pretty much all those stories, no matter how mainstream. Xandros is pretty rarely mentioned, and I don’t think I’ve seen Debian mentioned yet. But even without the credit, it’s still pretty cool that with the eeePC and Dell’s trysting with Ubuntu, the real in-roads to pre-installed consumer Linux these days are building on Debian. I’d always expected that Red Hat or SuSE or someone with more commercial muscle would get through that door first (they’ve certainly had box sets more readily available!), but apparently elitist, freedom obsessed, techno-geekery actually works better, somehow. At least when there’s a company to put a smiley face in front of it all :)

The other interesting thing about choosing Xandros, is that it presumably means the eeePC is covered by Microsoft’s patent protection scheme – which means no baseless threats against eeePC users from Microsoft, but also that Microsoft’s probably getting a cut of whatever Xandros receives from each eeePC sale. Whether you think that’s a problem or not, the result is probably going to be that it’s the thin edge of the wedge: it’s removed Microsoft from blocking a Linux preinstall on consumer hardware, which means we can see that regular people like Linux systems enough that they sell out in days. And that means Linux systems are a fact of life, and if Microsoft try to stop them by making it difficult for you to sell Windows systems, well, that’ll just make it difficult to sell Windows systems – which is another win for Linux. And once you get to the point where the argument for paying off Microsoft doesn’t rely on keeping your OEM deal for Windows, but is just a matter of whether their patents are actually valid…

Oh, also nice from the promoting free software angle, is that the first thing you see if you decide to visit Asus’s eeePC site to check the gizmo is the news item:

2007.11.27 ASUSTek is committed to meet the requirements of the GNU General…

Add the fact that an eeePC running a Debian-derivative has access to all the software in the Debian repositories for no cost, and the new government’s planned rebates and investments in IT probably mean that parents can get close to a full refund, and you’re getting pretty close to a choice of spending nothing and getting a popular, reasonably functional and very portable laptop with all the software you could ever want, that just happens to be running Linux, or paying an extra few hundred or thousand dollars to get a Windows laptop, and then probably pirating whatever software you end up needing.

In any event, to my mind, that makes 2007 the year of the Linux on the desktop: everything from here is just a simple matter of quantity. What’s next?

Managing Debian Installs

For a while I’ve been trying to find some easy way to keep a few machines I admin behaving the way I want them too with minimal effort. They don’t really need much maintenance – but I would like something to help keep them all in sync. Basically, something like FAI, but much, much simpler – ideally something that takes five minutes to understand, and another five minutes to deploy; and leave the more complicated and powerful tools for when they’re actually needed.

I figured what I really want was just a simple way to make a meta-package – one that doesn’t really provide any functionality, just tells apt/dpkg what I want installed (via Depends), and what I don’t want installed (via Conflicts) and adds any extra configuration stuff or local scripts that I decide I want.

But doing that with a real Debian package is harder than I’m really comfortable with – I don’t want to have to worry about potential lintian errors, or rules files and debhelper commands, or writing a Makefile to get my files installed or whatever, I want something more trivial than that. Looking for meta-package creators, the only one I spotted that I thought looked likely was cdd-dev, described as “Custom Debian Distributions common files for developing meta packages”. Unfortunately it seems to just provide templates, which makes things quicker, but no less complex.

Fortunately equivs (“Circumvent Debian package dependencies”) is actually used for metapackages these days, according to its maintainer on IRC and its long description:

This package provides a tool to create Debian packages that only contain dependency information.

One use for this is to create a metapackage: a package whose sole purpose is to declare dependencies and conflicts on other packages so that these will be automatically installed, upgraded, or removed.

Another use is to circumvent dependency checking. […]

That turned out to work much better than I remembered (from whenever I last tried it – back in ‘99 I guess?), with the only drawback being that I couldn’t add files easily. But that’s just a matter of creating a patch to equivs, which I then won’t have to worry about again. So having done that, I can now create a metapackage to do whatever I want by creating a file like:

Suite: client
Section: misc
Priority: standard

Package: ajs-client-stuff
Version: 20071114.1
Maintainer: Anthony Towns <aj@erisian.com.au>
Description: Metapackage for aj's client computers
 Depends on necessary packages, etc.

File: /etc/apt/sources.list.d/client.list
 deb http://mirror.localnet/debian etch main contrib non-free
 deb http://mirror.localnet/debian etch-proposed-updates main contrib non-free
 .
 deb http://security.debian.org/ etch/updates main contrib non-free

File: postinst
 #!/bin/sh -e
 .
 apt-key add - <<EOF
 [output of gpg --armour --export $KEY]
 EOF
 .
 ##DEBHELPER##

debhelper kindly takes care of getting the permissions right for me, and equivs will generate a full source package if I tell it to, which I can just upload to mini-dinstall and have a regular Debian repository just by writing a text file and running equivs-build. And my metapackage can add dependencies, conflicts, apt sources, cronjobs, scripts, configuration files, documentation, or whatever I happen to want – which means I can make it automatically update itself, and thus install any dependencies or remove any conflicts, which then means that modifying the config on all the machines is just a matter of updating the metapackage. And new installs is (hopefully) just a matter of doing a standard install and then adding the metapackage. Perfect.

…even if it is really little more than a reinvention of rpm’s .spec files. :)

Some fun...

Something I’ve been meaning to play with for a while, inspired by a slashdot post the other day:

Hopefully I’ll be able to speed up the calculations enough to have it work on more than just .1% of the data in reasonable time (the above took three hours of CPU time to generate, sadly), at which point we might be getting somewhere.

Props to cairo and gifsicle for the viewable data dump.

GPLv3 and Debian

So with the GPLv3 and LGPLv3 finally out (hurray!) it’s time to actually pay attention to the practical consequences of an upgrade to the world’s premier copyleft licenses.

One of the things I hadn’t noticed, though I should have, is that the LGPLv3 is not compatible with the GPLv2 – that is, GPLv2 programs that used LGPLv2 libraries cannot make use of new versions of those libraries released under the LGPLv3 without also updating their license to GPLv3 (or “GPLv2 or above”, or LGPLv2 or LGPLv3 or something similar).

Fortunately one of the major LGPLv2 libraries out there – glibc – is reportedly going to remain available under the LGPL v2.1 but for other libraries, it could become a problem, and it’s something GPLv2 authors will need to pay attention to.

Also fortunately the opposite problem – GPLv3 apps using LGPLv2 libraries isn’t a problem, because the LGPLv2 explicitly allows redistribution and modification under the terms of GPLv2 or later, so you can take an LGPLv2 library and incorporate it in a GPLv3 application and just release the whole thing under the GPLv3.

Unfortunately, the LGPLv3 doesn’t have a similar clause, so when the GPLv4 comes out, if it ever does, we could easily find the GPLv4 apps can’t use LGPLv3 libraries, and GPLv3 apps can’t use LGPLv4 libraries.

Really these are side-effects of a fundamental change: up until now, almost everything we’ve distributed – and certainly all the low level stuff: the kernel, the C library, the compiler, the shell and basic tools – have all been under a GPLv2 compatible license. Some of them have been more flexible than that, due to being under a BSD or MIT license, or being under the LGPL or having a “GPLv2 or later” clause, but we’ve been essentially able to ignore all that and just treat everything as GPLv2. There are a few exceptions that we have to deal with – the occassional app that’s not GPL compatible doesn’t cause much of a problem, because things that depend on it don’t usually create copyleft issues, which just leaves checking that it doesn’t use any GPL-only libs, such as gnutls or libreadline. Openssl is a nuisance, in that it’s a popular, GPL-incompatible library, but it’s also reasonably easy to deal with on a case by case basis.

But with the GPLv3, that changes: as GPLv3 apps and LGPLv3 libraries get uploaded, the assumption that more or less everything is GPLv2 compatible goes out the window, and while some things are GPLv2 only (such as the kernel!) we also can’t assume everything is GPLv3 compatible.

Fortunately, both the GPLv2 and GPLv3 have what is known as the “system library” exception, which allows you to not bother distributing the source to your C compiler or C library or kernel with your app, because everyone’s assumed to have one of those already. Unfortunately, the GPLv2 version of that exception includes the phrase “unless that component itself accompanies the executable”, so Debian has always cautiously assumed the exception doesn’t apply to us, because everything we distribute in main “accompanies” everything else in main – kernel, C library, C compiler, etc. Equally unfortunately, while the GPLv3 avoids that problem, but ends up with a clause that’s not terribly comprehensible, despite a couple of rewrites during the drafting process.

What’s all that mean?

The big thing is that it means Debian’s going to have to put a bit more attention into the system library exception, both for GPLv2 and GPLv3 apps, because our current policy of trying to avoid needing it is probably not going to be sustainable. I’m not sure how in depth we’ll need to go for that; we’ve already contacted the FSF licensing folks for an explanation of the GPLv2 clause in relation to the CDDL, but it’s possible the FSF might prefer to be ambiguous in responding so as to keep their options open for defending or enforcing the GPL later, in which case we might need to get legal advice of our own as to what we should do.

The second thing is that in reviewing our understanding of the system library exception, we might be in a position to have a Debian GNU/OpenSolaris port, making use of the exception for the CDDL-licensed OpenSolaris kernel and libc, as Nexenta do. That seems to me to be almost certainly possible for GPLv3 (or “GPLv2 or above”) applications, and probably possible for GPLv2 apps as well.

If so, that’ll let us expand Debian to supporting four kernels in the near future: Linux, Hurd, FreeBSD and OpenSolaris, which will be pretty fun, in my book.

The downside, is that if we find an interpretation of the system library exception that lets us mix and match licenses comfortably, that same interpretation might be able to be used to let propietary software vendors mix and match GPLed software with non-free software, rather than just free software with different copyleft licenses – and thus defeat the entire point of copyleft. What’s worse, is that by following Debian’s lead, they’d have some moral authority in doing that, beyond just having a legal argument as to why it’s okay.

And that’s the rub.

(My thoughts on the license texts as posted to -legal are here, here, here and here.)

Some Random Notes from DebConf

debootstrap’s now team maintained under the debian-installer subversion repository; uploaders are Joey Hess, Frans Pop, Junichi Uekawa and myself. Rumours are Colin Watson might be joining in too. There’s a few changes committed, but an upload hasn’t been made yet – at least last I checked. Frans is applying pressure to bump the version to 1.0, as he seems to think it’s ready for production use.

ifupdown’s had a maintainer upload, albeit to experimental – consider it a response to the competitive threat of madduck’s netconf. Changes are pretty minimal – it now uses iproute instead of ifconfig, there’s a “-o” option for ifup that lets you say things like “ifup -o wireless_essid=DebConf7 eth1” instead of having to edit /etc/network/interfaces. There’s a couple of backwards-incompatible changes unfortunately, so some other development’s needed before it will be suitable for unstable.

In other red-letter news, the bzr archive for dak is up to date compared to what’s in use on ftp-master. Take advantage while you can! dak continues to stand for “debian-archive-kit”, not “debian-archive-kilt” however.

Chatting about the CDDL

Just prior to the “State of the Coffee Cup” talk on Sunday, Tom Marble sent a quick invitation to Steve Langasek, Sam Hocevar and myself to have a chat with Simon Phipps about the CDDL and some of the concerns that have been raised on the debian-legal list lately – particularly about choice of venue. Unfortunately in the ten or fifteen minutes between the mail and the meeting, we couldn’t find Sam, but we managed to grab Don Armstrong to join in instead.

It’s a couple of days later, and I’m going from memory, not notes, so this is paraphrasing. Simon’s primary concern was one of predictability – Debian accepted the Mozilla Public License back when it was released (and prior to Mozilla and other software tending to get dual licensed under the GPLv2) which is what the CDDL was based on and contains almost the exact same terms. Simon’s comment was that if you’ve accepted that then, reviewing new licenses from scratch and coming up with different results really makes it hard to work with you, and worse makes it look like you’re basing your conclusions more on who proposed the license rather than what the license actually says.

Don and Steve seemed to mostly appreciate that point, but still thought that choice of venue was harmful and unnecessary, and that it’d be nice if Sun – as pretty much the primary organisation currently behind any of these licenses – would take some steps to reduce that harm: limiting choice of venue to when both parties in a dispute are multinational, for example. Simon’s concern there was that updating the CDDL would require a lot of effort internally – getting buy in from all the groups in Sun using the CDDL, as well as getting the wording changes approved from the appropriate legal advisors, and convincing the lawyers of Sun’s commercial partners that the changes weren’t going to be a problem later; and committing to all that work when a new versions of the GPL is about to be released and it might be a better thing to work on getting the relevant software available under that, or even the GPLv2. On the other hand, Simon thought that making (binding) statements about Sun’s interpretation of the license seemed to (eventually) work reasonably well for the DLJ, so might be something that can be done again for this issue. And while that only counts for Sun, we can fairly easily ask if other authors of CDDLed stuff agree with Sun’s interpretation for their software, and only worry about possible problems if it turns out they don’t.

That’s pretty much where we ended up I think, with Simon suggesting that we work out what exactly needs to be clarified with Tom, and then get it passed back up to Simon to get signed-off on. And hopefully that’s what’ll happen over the next little while. :)

(The somewhat more interesting question of whether we can combine CDDL’ed OpenSolaris (in particular Solaris libc) with GPLed userspace tools such as dpkg to produce a Debian OpenSolaris distro is still up in the air – Sam’s currently waiting on a response from the FSF legal team on one of the questions that concerns Debian, aiui; but they’re currently flat out working on GPLv3 as you might imagine. Oh well – in the meantime, there’s always Nexenta!)

Dunc-tank Report Ideas

Okay, so with the “Bits from the DPL” talk that Sam and I shared now out of the way, that hopefully means finishing off the whole dunc-tank thing won’t imply any more conflicts of interest, potential or otherwise. And since we’ve now had the post-etch release team report, there shouldn’t be much to wait around for.

I’d been kinda hoping to be able to come to debconf with a pre-prepared report of similar quality to the debconf6’s final report (or debconf5’s for that matter); but while I was going back through the discussion threads, it just got harder and harder to find a way of writing the report that didn’t seem like it’d just inspire the same arguments again. I think that’s avoidable – I know some of the people who were against the way dunc-tank ended up have indicated there are other approaches they might support, and it ought to be possible to summarise the dunc-tank experience in a way that lets us have a better idea what to do in the future based on that experience, than have the same disagreements we’ve already had. Anyway, my best guess is to try crafting the report via blogging or put it on a wiki or something.

Anyway, there are three things that I thought we could usefully have in a report that wouldn’t devolve into the same old arguments (after all, if we at least have new arguments, that’s some sort of progress!).

First of all is trying to have a real understanding of what happened with etch – getting a release out is complicated with lots of potential blockers beyond just “release management” work: d-i, kernel, toolchain, CDs, builds, bug tracking, whatever. Presumably dunc-tank only ever had a chance of removing release management as a blocker, and since we can’t redo the etch release while changing parameters like dunc-tank or Vancouver, at least having as clear as possible understanding of which blockers happened when, and what other influences might have been important the release process seems like a good way of understanding the influences that were and weren’t important, so we can have a better idea what to do in future. I’m figuring a reasonable structure might be something like:

• Release blockers between July 2006 and April 2007
• Analsyis of bug discovery/resoution rates between July 2006 and April 2007
• Comparison of major statistics for etch with previous releases (freeze length, etc)
• Comparison of etch with Ubuntu’s edgy release (October 2006, roughly the time dunc-tank started), and feisty release (April 2007, roughly the same time as etch’s release)
• Summary of other release timliness projects: Vancouver proposal, release assistants, BTS version tracking and changes to RC bug monitoring, transition monitoring through the entire release cycle, binNMU changes, improvements in experimental support, BSP marathons
• Summary of release quality projects: DFSG-free content, LSB support, security support for testing, QA meetings, piuparts runs, whole archive rebuild testing

The theory is that the above should provide the information to make a good judgement of what (if any) effect dunc-tank (or the other projects) had on the release, without implying a preconceived conclusion of whether dunc-tank’s “good” or “bad” or whatever.

As well as looking at the release stuff in some depth, it seems to me to be pretty sensible to do a report on how dunc-tank worked financially. Anyone doing any similar sort of “let’s raise funds, then spend them on free software” is going to have similar issues in getting publicity, collecting funds, and actually spending them, no matter how they end up deciding on the harder questions of deciding who or what to fund. There’s not a lot to say there, and I don’t think there’s much that’s controversial there either, but there are a few problems and approaches to solving them that are worth reporting on.

I think that’s about as far as you can get trying to cleverly avoid the controversy though. And presuming we rule out the idea of choosing a position on some of the controversial questions and advocating for that (“dunc-tank was good, paying release managers was awesome, we should do it again for everyone!”), the best idea I’ve had so far is to just try summarising some of the controversial questions that remain open, which Debian, or dunc-tank, or whoever can then address in whatever way we like – by GRs, or further debate, or more experiments, or leaving it to other people to do for us, or whatever. So having the questions and some of the arguments for various answers and summarising what happened with dunc-tank seems like a plausible way to go. The questions I’ve got:

• Is paid work compatible with volunteer work?
  • How the release managers and assistants reacted to doing volunteer work while someone was paid to do similar work
  • Other developers – some donated; others resigned or reduced work
  • Dunc-bank and QA initiatives
• How much distance should there be between Debian and whoever’s funding development?
  • Involvement of DPL and high-profile developers
  • Involvement of SPI
  • Statements from press reports about the relationship
  • Votes to support/endorse/recommend against
  • Donors being confused as to where to donate funds
• How should work or developers to be funded be chosen?
  • Potential for favouritism
  • Potential for misalignment with Debian’s goals
  • Choosing based on least controversy or most benefit
• How should paid “volunteer” work be priced?
  • Covering living expenses?
  • Comparable rate to similar proprietary/commercial development?
  • “Market” rates?
  • Full-time versus part-time work
  • Contracts versus prizes or bounties or rewards

No promises as to whether they’re the best questions, the best sub-points, or that we’re going to be able to come to an answer on any of them. :)

Anyway, that’s my theory so far, no I need to grab some lunch before the next lot of talks – feel free to mail me, or talk to me in person, or blog responses or whatever…

Men in Skirts

Well, made it to Edinburgh, via a forty hour stopover in Singapore. Picked up a tie and kilt in the Debian tartan. Sadly Edinburgh in midsummer isn’t as warm as Singapore in summer, or even as Brisbane in winter, so there’s good odds my knees are going to be shaking as well as visible all to soon. Oh well, there’s reports that there might be scattered sunshine by Monday – fingers crossed.

The dak BOF’s been delayed until tomorrow (Saturday 3pm, Edinburgh localtime) so Andi and Steve can sit in. Bit weird having something that technical on Debian day, I guess, but hey, whatever. It’s scheduled for a 2hr session, but I’m hoping it’ll be pretty informal and hacky rather than slides and lecturing or whatever. I’m hoping people will arrive with some ideas on what cool things that dak ought to be handling, and we can spend the time actually making dak support as many of those things as possible. Some of the ideas I like are in the BOF proposal. It might also be worth having a look at the video (or audio) of Robert Collins’ talk (“…Release always?”) at the Debian miniconf at linux.conf.au earlier this year, particular related to some of the ideas about setting up separate GNOME or KDE or libstdc++ staging areas for getting major updates ready for unstable rather than mixing them all up in experimental. Anyway, feel free to plan on arriving or leaving anytime during the BOF, unstructured is just another word for freedom!

Yesterday I had a quick chat with Frans about getting debootstrap officially incorporated into the d-i subversion repo, so that it’s officially team maintained, and there’s a convenient central place for hacking on it. Given the recent discussion (while I was flying to debconf, in fact – how inconsiderate!) hopefully that’ll mean we’ll get a couple of cleanups there too.

Oh, also got informed that Eben Moglen will be giving a free lecture in Edinburgh on Tuesday after debconf (the 26th) courtesy of the Scottish Society for Computers and Law. Sounds interesting if you’re into the whole theory of law and the effects of modern technology thereon:

In this lecture, Professor Moglen considers how private legislation is replacing public law as the organising intellectual structure for software and the technology industries, with far-reaching social consequences and theoretical implications.

Not sure if he’ll arrive in time to drop by during DebConf proper or not.

Torrenting the Debian Archive

Continuing the theme from my previous post – the first and fundamental thing any distro needs to have, and thus the first and fundamental thing to think about disintermediating, is some way of distributing software. That might be burning CDs and sending them out to resellers or OEMs, or having your stuff available for download, whether by putting it up on your own server, hosting it via a dedicated high-reliability provider like akamai, or maintaining a network of mirror sites and keeping them up to date.

And, of course, anyone who wants to distribute their own software has to do the exact same thing – which either means doing it themselves, and not being able to take advantage of the scalability the distributor has already made work, or going through the distribution and dealing with the problems that entails, as well as the benefits.

In this case, disintermediation and decentralisation are pretty much one and the same thing: and decentralising content distribution is already a well understood problem: that’s what peer-to-peer is all about, and peer-to-peer distribution of, well, distributions is already very successful – at least when it comes to CD (and DVD) images. Which means that just about anyone can create a CD image, and distribute it to the world in exactly the same way a major organisation like Debian or Red Hat would – upload a torrent file, run a seed, and let the magic of BitTorrent do its thing. Scalability is no longer something you need a distribution organisation to manage, instead it’s built directly into the technology itself.

Unfortunately BitTorrent is designed for large static images – not something you update on a daily basis. And for better or worse, my preferred distribution (Debian testing, currently lenny) does update that frequently.

Fortunately, though, BitTorrent’s an open protocol and has dozens of open source implementations – so last year I was able to propose it as part of the Google Summer of Code, and was lucky enough to get a few responses and a student to work on it. It didn’t get much further than that unfortunately – the student lost internet access not long into the programme (and for most of its duration), and that was pretty much that. So when this year’s SoC rolled around, I didn’t really expect much, and didn’t put the idea up for consideration again, but lo and behold someone came through and asked if it was still possible and if there was any more information, and when I forwarded on some of the mails from the previous year we ended up with a second chance at the project.

So far it’s looking pretty good – we’ve had a lot more success at keeping in touch, and thanks to the extended schedule for the SoC this year we’ve been able to do a much better job of keeping on top of what’s going on. So much so, in fact, that there’s a first (alpha) release out before the SoC is officially due to start! Wonderful stuff!

What it does at the moment is allow you to take a Packages file (the stuff “apt-get update” downloads, which includes descriptions of all the packages that are available, how they inter-depend, and so forth), and from that information create a usable torrent from which to obtain the packages themselves which can then be used to share and distribute the packages.

There are two crucial steps in that: the first is allowing the torrent to work without requiring huge amounts of extra information to be distributed (which would introduce scalability problems of its own, instead of solving them), and the second is that the pieces that make up the torrent are selected in a way that matches the actual packages, so that when you upload a single new package, you are in fact only making a minor change to the torrent, rather than having it (on average) completely redefine half of the torrent (and again introduce scalability problems rather than solve them).

There’s more information on the DebTorrent wiki page and Cameron’s blog if you’re interested.

Anyway, it’s just an alpha release at the moment, which means while there’s code that does something, it’s not actually all that useful yet. The current plan is to next add some code to make it automatically prioritise packages based on what you’re actually using – so that rather than downloading all the debs it see, it’ll only download the ones you’ve got installed, or maybe only newer versions of the ones you’ve got installed, which should get us pretty close to the point where it’s actually useful for something.

The end result, of course, is to build a tool that you can point at a Debian archive, run it on a machine connected to the Internet, and you won’t have to do anything more to have a reliable, scalable and reasonably efficient means of allowing your users to distribute and update their systems. In this case, scalable means that if you end up with as many users as Debian or Ubuntu, your users will have a comparable experience, as if you’d arranged for a similarly comprehensive mirror network, without actually having to do the leg work.

And heck, presuming that works, it doesn’t even matter if no one else actually does that – it’s worth it even if it saves Debian or Ubuntu the effort of keeping track of a mirror network by hand.

There are interesting possibilities at smaller scales too, of course. :)

A conversation with jdub

A couple of months ago I was sitting in a north Sydney pub cradling a beer with the ineluctable jdub, the decidedly unparsimonious rusty and (if I’m accurately remembering who’d left and who hadn’t by this point) the irremediable mrd. As you might imagine, these sort of conditions are a perfect breeding ground for a particular type of discussion, and with Jeff’s departure from Canonical not that long beforehand accompanied by the then ongoing DPL elections, talk turned to the future of Debian, and in particular Jeff’s views on that future.

Obviously more than a few weeks have passed since then, so please imagine this as a dream sequence with a stylish TV fade-in accompanied by appropriate fine print caveats about paraphrasing and the perfidy of memory.

I think it’s fair to sum Jeff’s main point about the future of distros as one of “disintermediation” – that is removing, or at least minimising, the barriers between the authors and users of a piece of software, and in general getting closer and quicker collaboration between everyone involved and interested in developing and maintaining that software. Which means things like trying to remove the line between the Debian bug tracker and upstream bug trackers entirely, so that Debian users end up talking directly with the upstream maintainers, rather than having a Debian maintainer relaying information between the two. It also means making the path for updated packages to get to users quicker and smoother – so that as soon as a developer commits a new version of the software to version control, the next time users of that software “apt-get update” they see it, download it, and start using it. Or at least, they do if they’ve chosen to follow the bleeding edge.

Jeff had a bunch more ideas in that vein – like not bothering with a central archive, but having users collect the packages they’re interested in from upstream sites all over the place, standardising, reducing or removing the “control” information for packages so that creating the correct packages for 80% of free software (that uses ./configure in the usual way, eg) was a complete no-brainer, and perhaps viewing distributions not so much as the gatekeepers and central players of the open source world, but perhaps more as systems of providing resources, assistance and opportunities to software authors who deal with their users directly – and in so doing have more of an opportunity for their users to actively participate in the project’s development, and have more feedback (and control) of how their project is perceived by its users.

I’m not sure if I’m doing it justice – it’s a pretty radical notion, and as an upstream-type hacker, Rusty was pretty enthralled by it. As a distro-type hacker, on the other hand, I’d have to say I was daunted: getting rid of the middleman here means getting rid of Debian, which I must admit I have something of an attachment for… and yet, in principle at least, I can’t see anything but good sense in the idea – getting users and developers interacting more closely is important to improving the free software community, and while distros have historically been a force for making free software more accessible to people, it’s entirely possible we can now make it even more accessible now by, in some sense, doing away with distros. And if that’s the case, then we definitely should, because it’ll benefit both users and developers of free software, which is what we’re all about.

On the other hand, distributions (at least currently) aren’t just about collecting and compiling software which could be automated; it’s also about integrating it, standardising interactions between software, and in some cases acting as a buffer between users who want to use a bit of software and developers who for one reason or another don’t actually want to deal with those users (perhaps due to lack of a shared language or political or philosophical disagreements).

I think it’s also pretty interesting to view Gentoo through this light; that is, not so much as a distro that makes you compile everything from source, but rather as a distro that gives its users a system based on exactly what the developers produce, with the addition of only a comparably tiny amount of Gentoo-derived content. Maybe it’s just me, but Gentoo sounds a lot more interesting to me in that light.

So is that a foretaste of the future of binary free software distributions too? There’s a lot of things that would stand in the way of Debian managing that, but so far all I can think of is a thousand problems that should probably be solved anyway, and not even one that can’t.

I think, btw, that jdub had been reading Accelerando (also available as a Creative Commons licensed ebook) somewhat concurrently with thinking about this stuff, so perhaps it should be called “Distributions 2.0” or something. Accelerando’s a crazy-awesome near-future/singularity book by Charles Stross, who happens to live in the city that will host the forthcoming DebConf 7…

Can open source methodology make a movie?

Sometimes doing a Google News search for debian turns up some fascinating little gems. Today’s was this article:

Einfeldt says that the project also plans to sell copies of the film, or at least one of the versions of the film. Taking a cue from the Debian project, Einfeldt says that there will be several versions, starting with the edit codenamed “Buzz.” He says that there will be another version codenamed “Rex” that will be sold on DVD and through other avenues like Lulu.com that cater to self-distribution.

Some interesting quotes in the article from Postgresql hacker and SPI treasurer Josh Berkus, along with a link to the Digital Tipping Point wiki.

More DWN Bits

Following Joey’s lead, here’s some DWN-style comments on some of the stuff I’ve been involved in or heard of over the past week…

A future for m68k has been planned on the release list, after being officially dropped as a release architecture in September. The conclusion of the discussion seems to be that we’ll move the existing m68k binaries from etch into a new “testing-m68k” suite that will be primarily managed by m68k porters Wouter Verhelst and Michael Schmitz, and aim to track the real testing as closely as can be managed. In addition the m68k will aim to make installable snapshots from this, with the aim of getting something as close as possible to the etch release on other architectures.

A new trademark policy for Debian is finally in development, inspired by the Mozilla folks rightly pointing out that, contrary to what we recommend for Firefox, our own logos aren’t DFSG-free. Branden Robinson has started a wiki page to develop the policy. The current proposal is to retain two trademark policies – an open use policy for the swirl logo, that can be used by anyone to refer to Debian, with the logo released under an MIT-style copyright license, and left as an unregistered trademark; and an official use license for the bottle-and-swirl logo, with the logo being a registered trademark, but still licensed under a DFSG-free copyright license. The hope is that we can come up with at least one example, and hopefully more, of how to have an effective trademark without getting in the way of people who want to build on your work down the line.

Keynote address at OpenFest. Though obviously too modest to blog about this himself, Branden Robinson is currently off in Bulgaria, headlining the fourth annual OpenFest, speaking on the topics of Debian Democracy and the Debian Package Management System.

New Policy Team. After a few days of controversy following the withdrawal of the policy team delegation, a new policy team has formed consisting of Manoj Srivastava, Russ Allbery, Junichi Uekawa, Andreas Barth and Margarita Manterola.

Point release of sarge, 3.1r4. A minor update to Debian stable was released on the 28th October, incorporating a number of previously released security updates. Updated sarge CD images have not been prepared at this time and may not be created until 3.1r5 is released, which is expected in another two months, or simultaneously with the etch release.

Debian miniconf at linux.conf.au 2007. While it may technically not be supposed to be announced yet, there’s now a website for the the Debian miniconf at linux.conf.au 2007, to be held in Sydney on January 15th and 16th (with the rest of the conference continuing until the 20th). This year derived distributions are being explicitly encouraged to participate, so competition is likely to be high, and it’s probably a good idea to get your talk ideas sorted out pretty quickly if you want them to be considered!

Vote Early, Vote Often

A couple of comments on the ongoing votes.

The DFSG/firmware issue is a complicated one. For the votes that we’ve currently got open, I’m voting for futher discussion in favour of the DFSG#2 clarification – not because I disagree with requiring source code for all works in principle, but because I think we should be making sure we can make Debian work with full source for everything first, before issuing position statements about it; and I’m voting for “release etch even with kernel firmware issues” above further discussion and “special exception to DFSG#2 for firmware” below further discussion, because I don’t think we can handle the broader issue before etch, and I don’t think it’s a good idea to try to tie the exception to the non-existance of technical measures directly. I’m not really sure that’s a good enough reason to vote that option below further discussion, so I might change my vote on that yet.

There have been quite a few other proposals on the topic, including one from me that didn’t get sufficient seconds to be voted on, another from Frans Pop that was withdrawn due to procedural issues, a couple more from Sven Luther, and a new proposal from Sven and supported by the kernel team that’s a further refinement on the “release etch even with firmware issues” resolution currently being voted on.

I personally think we should spend some time after etch thinking a bit more deeply about this stuff. Personally, I think we should insist on source for everything, but that also means we need to have a clear explanation on why it’s good – even for firmware and font files and music and artwork – and it means we’re going to need to make sure we have a reasonable way of distributing it, and it means we’re going to have to make sure that we have a good way of distributing stuff that doesn’t meet our standards but that users still need or want; whether that’s drivers they need to do installation or get good graphics performance, documentation for their software, or whatever else. There’s a lot of real improvements we could make there – both in making the core of Debian more free and more useful, and making it easier for users who want to make compromises to choose what they want to compromise on and what they don’t want to compromise on. I really hope that once etch is done and dusted quite a few of those sorts of improvements will get done, both in technical improvements in Debian, and in good advocacy from Debian and other groups towards people who aren’t already making things as free as they potentially could be.

One the recall issue, I would have preferred to vote “re-affirm”, then “recall”, then “further discussion”, to say “I don’t think this creates a conflict of interest that can’t be handled, but I’ve no objection if other people think it does”. But since that isn’t what the ballot(s) turned out to be, I’ve voted “re-affirm” above further discussion on that ballot, and “recall” below further discussion on the other ballot.

I’ve voted the “wish success” option above “don’t endorse/support” option for two reasons – first, because the “wish success” resolution actually refers to “projects funding Debian or helping towards the release of Etch” in general, while the “don’t endorse/support” proposal specifically talks about projects I’m involved in (including non-Dunc-Tank projects) which seems kind of personal. There’s also the fact that I’d rather see more success and mutual support in the Debian community, even for projects I don’t personally like, than less. I originally voted the “don’t endorse/support” option below further discussion for those reasons, but then decided that that was silly – just as I would have been happy to vote for the recall above further discussion, it’s not really that big a deal either way, and fundamentally I think both options are essentially the same anyway: that any potential conflict of interest can be dealt with, and Debian and Dunc-Tank are fundamentally different projects. I was probably influenced in that a fair bit by the “not endorse/support” option being proposed and seconded mostly by people who actively oppose the idea, including Josselin Mouette, Samuel Hocevar, Pierre Habouzit and Aurélien Jarno.

But in the end, the outcome’s fine any which way – some people will continue disagreeing with the concept, others will agree with it, and everyone can keep contributing to Debian in whatever way they think’s best whatever the outcome. And like I said when running for DPL this year, while you are a lot more visible as DPL, it’s not actually that necessary to be DPL to get things done in Debian.