The recent Ubuntu Community Council marketing drivel about Mint … or how to put your foot into it with a run-up

Three days ago the Ubuntu Community Council released this document about the downstream distro agreement Canonical provides.

This is one of the worst documents i have seen released by the Council ever, it explains exactly nothing but tells you “Canonical is doing it right, trust them, we do too”. There is not a single piece of the technical background behind all this explained anywhere (while there is a lot to explain and if people read it they might even grasp why this agreement exists) or why it has nothing to do with licensing the ownership of anything to anyone, claiming debian copyrights, violating the GPL or any other such rubbish that gets claimed all around the net now. In this specific case i feel the Council has not done their homework … and why … oh why .. did you guys even have to put the word License into your headline at all. While the word might have some meaning to lawyers in this context, it must have been clear to you that this word will blow everything out of proportion .. the emphasis here needs to be that it is an agreement between both sides to protect the Mint users from technical breakage of their system …

So lets see if a proper technical explanation can probably calm the tempers a bit …

We’ll have to do some time travelling for this … come back with me to the year 2004 when a small developer team of less than 20 people grabbed the giant debian archive, made some changes to it, recompiled the whole thing and released it under the name Ubuntu.

Back then people read everywhere that this Ubuntu thing is based on debian and indeed when they found some third party packages built for debian (and not Ubuntu) they could sometimes even install them without breaking their system. Some people even went further and actually added debian repositories to their sources.list and were surprised to see their systems go up in flames.

Ubuntu packages aren’t at all the same as debian packages, Ubuntu might have made changes to some UI library where a binary symbol that offers the “draw rectangle” function actually draws an oval on your screen. Binary packages that were recompiled against this lib in the Ubuntu archive know about this change, binary packages from the debian archive that have been compiled against the debian version of the library will not know that, once they try to use the “draw rectangle” function, something unpredictable happens and the app falls over.

Today it is a known fact to most that you should not simply add a debian repo to your package sources, the info has spread across the net after 10 years, the generally established support answer if someone asks about adding plain debian binaries is “yes, you can and if you are lucky it even works, but if it breaks you have to keep both pieces”. The fact of binary incompatibility has caused lots of discussions and some anger on the debian side back then. “Why can’t you just do your work in the debian archive” was a question i remember hearing often from DDs back then. Ubuntu wanted to do stuff faster than debian and move ahead with some pieces. Debian is a rock solid tanker, it is gigantically big like a tanker but it is also as slow as one. Imagine that without Ubuntu doing this step things like developing upstart would not have been possible, imagine also that without Ubuntu moving forward with upstart and changing the init system there would most likely be no systemd today. Sometimes you need to move out of boundaries to get the whole thing to the next level.

Now lets fast forward back into presence again. Ubuntu has grown over ten years and there are many “in-archive” and also many “out-of-archive” downstream distros. It is facing something similar debian had to face back then. Downstream distros start to make changes out of the archive and provide their own binary packages for some bits. There is no problem at all with distros like Kubuntu, Xubuntu or Lubuntu here, their changes happen inside the Ubuntu archive packages in the archive will be linked against i.e. Kubuntu library changes. If it comes to the “out-of-archive” downstream distros there can indeed be the exact same problem that Ubuntu faced with debian back in 2004. These “out-of-archive” distros want to innovate the code, user experience or system design in a way that is either not appropriate for them to contribute back, or not appropriate for Ubuntu to take said contribution (because it breaks some existing functionality in Ubuntu or whatever). They feel they need to do their changes outside of the archive like Ubuntu felt back then with debian. Sadly these downstreams often enough do not have the resources to actually rebuild the whole archive so they start providing a mishmash of their re-done binary packages with the binaries from the Ubuntu archive which will leave you as a user with the same situation the early Ubuntu users.

For Mint you can find the list of changed packages on this page … you will see that for example libgtk is in the list, in case Mint decides to switch the “draw rectangle” function to actually draw triangles without bumping the ABI all Gtk apps that you use directly from the Ubuntu archive will possibly fall flat on their face. Users will be upset and blame either the bad Mint or the bad Ubuntu quality for their breakage. Now imagine what would happen if Mint decides to make any innovation of libc (like Valves SteamOS does very heavily with regard to debians libc), the library everything is linked against either directly or indirectly. Most likely the majority of original Ubuntu packages would just break and they would be limited to only use the packages listed in that linked document.

Lets do a short detour towards Trademarks now … Canonical owns the Ubuntu trademark, this means two things … one is Canonical can make money from it (yay, that luckily pays my salary so i can do paid work on my favorite OS all day !!!) … but that part is not involved in this agreement at all, nobody is asking Mint to pay anything. The agreement also does not mean that Canonical claims any ownership of any code from debian, violates the GPL or steals credit for any of the code in the Ubuntu archive.

Remember, I said owning the Trademark means two things … the second thing is that Ubuntu means a certain quality (be it good or bad is up to the beholder here, feel free to make your own judgement). If i install Ubuntu i can expect a certain level of quality and that the basic desktop largely works (yes, yes, i know there are enough bugs to make a lot of jokes over that statement). It also means that distros claiming to be “based on Ubuntu” inherit that archive quality of the binary packages. You can rely on the fact that the Kubuntu installer is using the same back ends and libs the Ubuntu installer uses for example. Bugs on the low level are shared and fixed for both. Now lets look back at the list of Mint packages and we will see that they provide their own “Ubiquity” package. I don’t know what changes are in there and I don’t know if it does not make the Mint installer completely incompatible to what Ubuntu, Xubuntu, Edubuntu or Lubuntu use, it will likely introduce different bugs and features into the binary resulting from that source. So this second part of owning the Trademark is about protecting the brand by guaranteeing a certain quality under this brand (which in turn indeed helps for the first part).

While the agreement with Mint kind of targets the second part here, it protects the reputation of Mint more than it does protect the Ubuntu reputation. It makes sure that Mint users will not have false expectations and that their systems will not suffer from technical breakage by Mint claiming it is 100% Ubuntu binary compatible … And while this whole agreement might technically be treated as License in front of a court (where it most likely will never go) IMHO the bigger info here is that there is an agreement between two distros to prevent Mint users from the same issues Ubuntu users faced back in 2004 with regards to debian. If this makes Canonical evil is up to you to judge, I personally think it does not and is a good thing for both sides in the end.

Please note that all the above is my personal opinion and should in no way be quoted as Canonical statement, I solely write all this as an Ubuntu developer independently of my employer (I would have written the same if i would work at Google and do Ubuntu development in my spare time). In case you want to re-share it on your news site, please get this differentiation straight !

… lots of Canonical in my mouth …

Getting online this morning was in interesting experience, seemingly some news sites picked up a two week old post to a mailing list thread from me to turn it into something that generates revenue for them …

… intrestingly even though the original post was linked in all of these articles, people seem to be more intrested in the interpretation of the reporters of the sites than to read the actual thread, putting potential words from Canonical into my mouth that i didn’t ever say.

I must say I find that pretty offending to me as an individual … yes, I do work for Canonical (still happily since nearly 9 years now and I love what I do and plan to go on to do so …) but please allow me to have my own mind and opinions, not everything an Ubuntu developer says is coordinated by Canonical, even if this statement might trash you conspiracy theories … (oh, and not every Ubuntu developer works for Canonical … unlike some people might want you to think, the Ubuntu dev community is healthy and happily chuggin along, with the new Ubuntu Touch community vibrantly growing)

What I wrote was my own personal opinion (that was actually the reason to use the word “personally” in my sentece about home banking, but I am not a native english speaker, so I might have misunderstood its meaning for all these years)

What I also did was to point to code evidence that shows that Linux Mint supresses security updates for certain software in its default setup … while it might be true that it is configurable and that it is only disabled for certain packages out of the box, this is still an evident fact and can be seen in the code, there is nothing to argue or discuss about (and as I learned now it seems to be part of the Mint philosophy since security updates seem to have caused them instabilities in the past)

Indeed I couldn’t keep my feet still and made the mistake to actually read comments on the different articles …

“He fears losing his job and needs to stir up stuff” … dude … after such a long time in an opensource company and getting headhunter offers regulary, you dont have to worry for your job … what I’m actually worried about is the undeserved badmouthing of Ubuntu based on FUD. Ubuntu is more than Canonical or its decisions, please don’t discredit the work of many many contributors out there … if you want to attack Canonical, do it, but pretty please take into account that Ubuntu is more than Canonical … Oh, and the stirring up part … I can tell you it isn’t any pleasure to be in focus like that for a side statement you made weeks ago …

“He wants to badmouth Linux Mint because they steal users from Ubuntu” … I seriously don’t care if users use Ubuntu, Xubuntu, Mint or elementaryOS, in fact it makes me proud to know they base on work I participated in (note that I maintained one of the first derivative distros of Ubuntu (edubuntu) for about two years nearly on my own) derivatives (and the work they feed back into the Ubuntu archive) are a big part of the Ubuntu ecosystem, why would any sane developer badmouth them ?

A big thanks to OMGUbuntu for fixing their headline … which initially suggested that I “advised” users to not use Mint because it is vulnerable, I never did, I just stated that I personally would not use it for online banking since I know they dont install all available security updates by default …

Seriously, I HATE raisins, I would never eat a cheesecake that contains raisins … did I ask anyone to not eat raisins or did I propose to stop producing them in the former sentence ? no, obviously I didnt … and I dont want the raisin farmers go jobless just because I dont like their product … why people did read something like this into my words in the mail that was quoted is really beyond me …

So lets see if we can get something constructive out of all this, obviously would I be a debian developer that had posted to some debian ML nobody would have picked it up … but since this trivial statement has drawn so much attention we can probably both benefit from it…

Clement Lefebvre’s statement

To me PERSONALLY suppressing any available security updates is a no-go and while Clem points out that it is configurable in Mint, I don’t belive my Mom or my sister would get along with that, they would just use the default. Which would leave them obviously vulnerable with some packages (wether the vulnerabilities are exploited or not, there are open security holes in your system after all) … obviously the practice to suppress these updates stems from bad experience with using Ubuntu provided security updates …

Hey Clem ! … so how about we take a look at this and improve that situation for you, obviously something in Ubuntu doesn’t work like you need it, Canonical puts a lot of time and money into improving the QA since about two years. I think it would be really helpful to sit down and look if we can improve it well enough for both of us to benefit (Ubuntu from your feedback and you from improvements we can do to the package quality) … wether you still want to suppress updating any packages or not even after we fixed the issue for you, is indeed your choice, but please dear press allow me to also still not use Mint for online banking then 😉

Quotes from the comments section in the above page:

… “Maybe it is time to re-evaluate whether security updates should be held back by default. Ubuntu have made steps to avoid regressions such as Phased Updates.” …
Clem: … “I’d be happy to have that discussion and look at the pros and cons post Mint16 release. It’s not a reaction to a particular incident though, it’s a difference in policy. We actually built the tools that would allow us not to make it trivial for people to apply changes blindly. There’s pros and cons to it, and that’s why it’s configurable.” …

So hey, as much out of bounds these press posts were for such a non-issue, it apparently caused some discussions and will possibly improve the situation for all of us in the end …

Oh, and btw … many people missed the actually interesting part in the mail thread … having Mate in debian and Ubuntu will definitely reduce the maintenance work for Mint since they can just pull it in from the respective archives (and it might bring Ubuntu another new derivative distro)

Important changes in Ubuntu Engineering

With the recent decisions of Canonical to do more upstream development like UnityNext, Mir and UbuntuTouch the
Ubuntu Engineering team took some fundamental decisions to help funding this development.

After a long, heated, team internal discussion it was decided that we no longer want to be a cost factor for
Canonical and that Ubuntu development will be split out into it’s own company under the umbrella of the
Ubuntu Foundation. The new company will be called “Ubuntu Engineering Ltd.” and will be seeking a listing at
worldwide stock markets within the next 5 years.

To make developing Ubuntu a more a more profitable thing for you as a developer and volunteer we worked out a
new upload concept which is just being put into place on launchpad right now.

* Each upload of a source package will be charged to your launchpad account with €0.05 (we picked the Euro here
simply because it is the more stable currency).
* For each successful build on one of the Ubuntu architectures (currently i386, amd64, armhf and powerpc) you
will get refunded with €0.01 per successfully built binary. An FTBFS (build failure) on any of these architectures
will cost €0.02.
* We are still in discussion with the debian project about the chargement for package syncs from debian and are
confident we will come to a conclusion with the DPL very soon (since syncs cost so much more we will likely be
charging slightly more than for a source package upload that goes directly into Ubuntu. Charging costs will likely
be handled inside debian directly)
* The refunds will immediately be turned into stock options at €1.00 per 1% of an Ubuntu Engineering stock or (at
your choice) into 0.5% of an album download from the UbuntuOne Music store.

You may also have heard about the recent decisions of the Ubuntu Technical board to provide a kind of rolling opportunity to follow the development release without being forced to upgrade your sources.list file.

We decided (in reminiscence to the release cycle this decision happened in) to stay with the R naming scheme for this and call the newly created rolling developer release the “Rolling Rouble”. The infrastructure for this will be in place within the next 10 days, if you are an Ubuntu Developer and plan to participate please update your sources.list by end of April 10th to point to “rolling-rouble”.

If you are a bug Triager you won’t be left out ! In agreement with the Ubuntu bug control team a similar yet to be fully defined set of features will also be put in place for bug triage. The same goes for blueprints where the refunding concept is already a bit more progressed (an accepted blueprint will cost €5.00, each fulfilled Work Item will be refunded with €0.01 in stock options or (at your choice) into 0.01% of an album download from the UbuntuOne Music store)

We belive this is an excellent business model for you as a developer, bug triager and blueprint creator and look forward to a bright future of Ubuntu development and full pockets for developers employed at Ubuntu Engineering Ltd.

In case you have any unanswered questions we will do our best to leave it like this if you mail our mailing list under: “Ubuntu Engineering Ltd.”

On behalf of the Ubuntu Engineering team
Sincerely yours, Oliver Grawert

The Bamboo Feeder – automating continuous ARM image tests

To minimize the need for manual image testing of our ARM images, the canonical QA team invited me to their team sprint this week in the Lexington test lab to help with setting up a fully automated infrastructure on a bunch of pandaboards.

If you know the pandaboard you might also know that it can boot from its mini USB port as well as from an SD card. As long as the SD is empty it will listen on USB for the first stage bootloader file (which then pulls u-boot via the USB connection).

The central design idea was to have a single machine with USB hub serving as an initial bootloader dispenser for all the other pandas (indeed we could have taken any kind of machine but there were some discontinued panda models around that we don’t use anymore so we picked one of these as the central server).

Once the bootloader is completely recieved and executed on the client pandas they default to do PXE booting and pull their kernel and initrd from a tftp machine over the network.

For this initrd i developed a small initramfs script (and matching initramfs-tools hook indeed) that simply streams the most recent Ubuntu image (location and name are configurable via a kernel cmdline parameter) from http directly to the SD card, mounts the first partition and dumps a bootloader configuration (similar configurable via a cmdline parameter) in place which has references to a remote debian-installer preseed file before it reboots the board into a fully automated installation.

In case there are issues all the pandas can be power cycled through a web-controlled power strip and indeed all of them are hooked up to a serial console server so you can access their serial console remotely in case your preseed file didn’t tell it to install sshd …

At the end of the install the beginning of the SD card gets zeroed so the panda thinks the card is empty and will boot via USB again (in case the kernel hangs on boot or something similar fatal happens u-boot luckily provides an erase command for MMC’s that can be executed via the serial control server).

Bamboo Feeder Schema

… So much for the theory …

Indeed we hit the first issue with the first step we tried to implement, USB booting only worked with one model of the pandas we had around, all the others simply didn’t pull their u-boot after the first stage bootloader was executed. Thanks to an impressing and tireless overnight debug and hack effort from John Rigby and Ricardo Salveti from linaro this issue was fixed on Tuesday. You can read Johns blog post about how to USB boot a panda on his blog, all fixes that were worked out should land in the quantal packages soon.

The next obstacle we hit was that TI apparently decided the panda should (like the beagleboard) be capable to be powered via the mini USB port (even though the current provided here is not sufficient to actually run the board unless you make sure that your kernel disables most of the power hungry devices on board) …

Read: we could not power cycle the boards remotely as long as the USB connection was in place …

I sat down and read up the USB specs. Theoretically the data transfer should not need the 5V connection to transfer its bits and bytes through the cable … so i opened an USB cable and just cut the power connection. Transferring data to my tegra2 netbook seemed to work this way (seems the port is always powered from the board side here), but sadly the pandas did not enable the mini port at all if there was no power applied to it from the other end …

Luckily there was a solution … the panda actually operates with 5V mains power so we spent our Tuesday with roaming around the nearby radioshack shops and buying barrel connectors (and sockets), shrink tube, a soldering iron and cable pieces. I then connected the power of the mini USB directly to the pandas main power connection
so that if you powercycle remotely the USB connection would be powered off as well.

Panda USB cable

Panda USB cable attached

Now that this bit was in place Carlos de Avillez (hggdh on IRC) who worked with me on this project and is a really awesome person to spend your time with (pay him a beer if you meet him at the next UDS !!) took all the puzzle pieces i had created and put them all in the right places on the different servers of the test lab, applied the
preseed, bootloader config and image location info … and off we went auto-installing our first panda … 😀

The whole system will soon drop its results into a public jenkins instance at so you will be able check installation results for every daily image build of our omap4 images (once Carlos integrated all this which is currently in the works). In the future there will likely also be automated SRU kernel tests and similar other things running under this setup.

This was a really exciting week and i had a huge amount of fun with the whole QA team. Given they are still a young team in canonical in their current form and just started their work to improve the overall quality of Ubuntu i guess you will soon see a massive increase in stability and bug-freeness. I expect once all their plans and tools are in place many bugs will be found long before a user hits them (and needs to report them) in a release.

Ubuntu on ARM, the best since sliced bread !!

It is that time of the year again where we’re nearing a a new Ubuntu release and as usual the Ubuntu ARM team can show off with support for a new architecture.

Have you already heard about the new shiny TI OMAP4 CPU ? If you haven’t yet and are interested in ARM stuff you surely will very soon. Ubuntu will additionally to the already known OMAP3 images release images for the OMAP4 architecture with the 10.10 Maverick release.

The shiny new pandaboard

Yesterday went online and you can sign up now for the early adopter program.
Ubuntu on the panda will be the best Ubuntu ARM experience you ever had thanks to collaborative efforts the Ubuntu ARM and TI developer teams made.

The good old beagleboard

With the new hardware release of the beagleboard XM that comes with 512M of RAM there finally exists a beagleboard that can run a full Ubuntu desktop really snappy making it more than just a a developer toy.

With 10.10 the Ubuntu ARM team also introduces the brand new type of preinstalled images. Once written to an SD card they expand themselves to the full size of the card on first boot and run the shiny new OEM config welcome screen (kudos to the installer team for that one)

But thats not all thats new in ARM land …

Are you an Ubuntu developer and interested in ARM … but never really started because carrying a bare board with you everywhere makes you recognizable as a geek immediately ?

There is help !!!!

About 4 weeks ago Toshiba brought the Nvidia Tegra2 based AC 100 to the stores in Europe it is a neat ARM Netbook that comes with a fully locked down android preinstalled …
Very fast people started to ask about Ubuntu or debian or any other Linux on the device … but indeed it comes with android …


Within a few days a developer community formed around teh AC 100 … and after a week the first possibility to gain root on the preinstalled android was found …

Now 4 weeks later the awesome guys from the #ac100 freenode IRC channel published a fully working Ubuntu ARM image for it … while still using the android kernel, the users pace is completely Ubuntu Netbook … with a few drawbacks (no sound, the HW buttons do not work yet and other little annoyances)
I’m confident these guys will soon figure out all remaining issues …

If you own such a Netbook, you can find the Ubuntu image and installation instructions at

Kudos to such an awesome community effort and as Ubuntu ARM developer I’m proud to see the first distro running on it is Ubuntu (this post is written on it, using drivel on the Ubuntu Netbook install)