The Bamboo Feeder – automating continuous ARM image tests

To minimize the need for manual image testing of our ARM images, the canonical QA team invited me to their team sprint this week in the Lexington test lab to help with setting up a fully automated infrastructure on a bunch of pandaboards.

If you know the pandaboard you might also know that it can boot from its mini USB port as well as from an SD card. As long as the SD is empty it will listen on USB for the first stage bootloader file (which then pulls u-boot via the USB connection).

The central design idea was to have a single machine with USB hub serving as an initial bootloader dispenser for all the other pandas (indeed we could have taken any kind of machine but there were some discontinued panda models around that we don’t use anymore so we picked one of these as the central server).

Once the bootloader is completely recieved and executed on the client pandas they default to do PXE booting and pull their kernel and initrd from a tftp machine over the network.

For this initrd i developed a small initramfs script (and matching initramfs-tools hook indeed) that simply streams the most recent Ubuntu image (location and name are configurable via a kernel cmdline parameter) from http directly to the SD card, mounts the first partition and dumps a bootloader configuration (similar configurable via a cmdline parameter) in place which has references to a remote debian-installer preseed file before it reboots the board into a fully automated installation.

In case there are issues all the pandas can be power cycled through a web-controlled power strip and indeed all of them are hooked up to a serial console server so you can access their serial console remotely in case your preseed file didn’t tell it to install sshd …

At the end of the install the beginning of the SD card gets zeroed so the panda thinks the card is empty and will boot via USB again (in case the kernel hangs on boot or something similar fatal happens u-boot luckily provides an erase command for MMC’s that can be executed via the serial control server).

Bamboo Feeder Schema

… So much for the theory …

Indeed we hit the first issue with the first step we tried to implement, USB booting only worked with one model of the pandas we had around, all the others simply didn’t pull their u-boot after the first stage bootloader was executed. Thanks to an impressing and tireless overnight debug and hack effort from John Rigby and Ricardo Salveti from linaro this issue was fixed on Tuesday. You can read Johns blog post about how to USB boot a panda on his blog, all fixes that were worked out should land in the quantal packages soon.

The next obstacle we hit was that TI apparently decided the panda should (like the beagleboard) be capable to be powered via the mini USB port (even though the current provided here is not sufficient to actually run the board unless you make sure that your kernel disables most of the power hungry devices on board) …

Read: we could not power cycle the boards remotely as long as the USB connection was in place …

I sat down and read up the USB specs. Theoretically the data transfer should not need the 5V connection to transfer its bits and bytes through the cable … so i opened an USB cable and just cut the power connection. Transferring data to my tegra2 netbook seemed to work this way (seems the port is always powered from the board side here), but sadly the pandas did not enable the mini port at all if there was no power applied to it from the other end …

Luckily there was a solution … the panda actually operates with 5V mains power so we spent our Tuesday with roaming around the nearby radioshack shops and buying barrel connectors (and sockets), shrink tube, a soldering iron and cable pieces. I then connected the power of the mini USB directly to the pandas main power connection
so that if you powercycle remotely the USB connection would be powered off as well.

Panda USB cable

Panda USB cable attached

Now that this bit was in place Carlos de Avillez (hggdh on IRC) who worked with me on this project and is a really awesome person to spend your time with (pay him a beer if you meet him at the next UDS !!) took all the puzzle pieces i had created and put them all in the right places on the different servers of the test lab, applied the
preseed, bootloader config and image location info … and off we went auto-installing our first panda … ๐Ÿ˜€

The whole system will soon drop its results into a public jenkins instance at so you will be able check installation results for every daily image build of our omap4 images (once Carlos integrated all this which is currently in the works). In the future there will likely also be automated SRU kernel tests and similar other things running under this setup.

This was a really exciting week and i had a huge amount of fun with the whole QA team. Given they are still a young team in canonical in their current form and just started their work to improve the overall quality of Ubuntu i guess you will soon see a massive increase in stability and bug-freeness. I expect once all their plans and tools are in place many bugs will be found long before a user hits them (and needs to report them) in a release.

Ubuntu on ARM, the best since sliced bread !!

It is that time of the year again where we’re nearing a a new Ubuntu release and as usual the Ubuntu ARM team can show off with support for a new architecture.

Have you already heard about the new shiny TI OMAP4 CPU ? If you haven’t yet and are interested in ARM stuff you surely will very soon. Ubuntu will additionally to the already known OMAP3 images release images for the OMAP4 architecture with the 10.10 Maverick release.

The shiny new pandaboard

Yesterday went online and you can sign up now for the early adopter program.
Ubuntu on the panda will be the best Ubuntu ARM experience you ever had thanks to collaborative efforts the Ubuntu ARM and TI developer teams made.

The good old beagleboard

With the new hardware release of the beagleboard XM that comes with 512M of RAM there finally exists a beagleboard that can run a full Ubuntu desktop really snappy making it more than just a a developer toy.

With 10.10 the Ubuntu ARM team also introduces the brand new type of preinstalled images. Once written to an SD card they expand themselves to the full size of the card on first boot and run the shiny new OEM config welcome screen (kudos to the installer team for that one)

But thats not all thats new in ARM land …

Are you an Ubuntu developer and interested in ARM … but never really started because carrying a bare board with you everywhere makes you recognizable as a geek immediately ?

There is help !!!!

About 4 weeks ago Toshiba brought the Nvidia Tegra2 based AC 100 to the stores in Europe it is a neat ARM Netbook that comes with a fully locked down android preinstalled …
Very fast people started to ask about Ubuntu or debian or any other Linux on the device … but indeed it comes with android …


Within a few days a developer community formed around teh AC 100 … and after a week the first possibility to gain root on the preinstalled android was found …

Now 4 weeks later the awesome guys from the #ac100 freenode IRC channel published a fully working Ubuntu ARM image for it … while still using the android kernel, the users pace is completely Ubuntu Netbook … with a few drawbacks (no sound, the HW buttons do not work yet and other little annoyances)
I’m confident these guys will soon figure out all remaining issues …

If you own such a Netbook, you can find the Ubuntu image and installation instructions at

Kudos to such an awesome community effort and as Ubuntu ARM developer I’m proud to see the first distro running on it is Ubuntu (this post is written on it, using drivel on the Ubuntu Netbook install)

Unleash the Beagles !!!

Over the last few weeks the mobile (mainly me) and the kernel team (mainly amit) of Ubuntu made some effort to bring up an image for the awesome TI OMAP3 Beagleboard.

Since we literally only had a few weeks and it is already been late in the release cycle when we started the images will have more rough edges than the other ARM images we provide. Also the board only has 256M of RAM so using a Netbook image, even though it is already slightly optimized for low-end systems and ARM usage it will surely use swap on your system if you try it out.

But hey, we made it after all and this will be the base for some really awesome OMAP images in the next releases.

A lot of thanks for endless patience with the “late guys” have to go to Steve Langasek, our release manager for taking the time to review code even after the freezes.
Other thanks for help and for sometimes poking the beagles nose into the obvious go to Loรฏc Minier and Colin Watson and to anybody else who made that happen in such short time … once again i’m happy to recognize the sholders of the biggest giants around under my feet !

So if you have a Beagleboard, go to any try out one of the images, if you dont have one yet, GET ONE !!!
Its fun and you can help out ubuntu on ARM !

PS: There is an untested kubuntu-netbook image here, i heard the kubuntu guys would be happy to hear if it works or not ๐Ÿ™‚

sudo make warm !

… or ‘how to spend a winter holiday to turn ubuntu into a heating control system’

about two years ago we got a huge solar thermal heating system for our house which worked fine for the first summer … until it overheated … we sadly had fallen out with the heating contractor since he screwed up a lot of unrelated stuff in the house making us very unhappy and we got never around to find a new guy for it (beside that i wanted to learn to handle all of the system myself)

our battery of solar panels

so after two winters where we rather heated the house with wood in the fireplace and the heating only worked at half power, this year i had to burn all my vacation days that stacked up over the year in december havin nearly a full month of holidays … alongside we had one week where it really got cold and i really didnt feel like chopping wood again ๐Ÿ™‚

so i decided to put up my scottish accent and spend as much time as needed with the three 1000 liter tanks and their pipes and valves in the engine room until i get that damned thing to fly at warp 10 ๐Ÿ™‚

the engine room

after a week of fiddling and studying i finally found out that the guy had wired up half of the system wrongly, valves that the control system triggered to open were closing instead, water flowed through pipes it wasnt supposed to, heat got routed into places where it wasnt stored etc …

so after two days of rewiring i suddenly had the heaters warm and cosy in the rooms, while the heating boiler wasnt running 24/7 anymore but only when heating up was needed and properly stored its heat in the big tanks, yay !!

the control unit is mounted in the worst place one can imagine in the basement room. you cant easily reach or see it. to see a sensor or valve value you have a little wheel you need to scroll up and down. you can only see about three values at a time on the display and indeed it’s usually three unrelated ones. so i spent my days scrolling this darn wheel up and down most of the time, standing on one leg in a very inconveninent position …

after i had the system back to normal and had a warm office again i started googling a bit and i found there is a usb datalogging unit for that box and the bill for teh system suggested that i even own it !

i found the little box after searching a day for it in another room in the basement hidden behind some pipes (no idea why the heating engineer had put it there or why he didnt bother to tell me it exists) … funnily it didnt work at all … guess what, the two cables running into it were indeed flipped …

after fixing that i actually got some data packets on my laptops USB port, yay !

googling even more i finally found that someone called Holger Rรถmer had written some linux software to properly recieve data from it and he uses it on debian and ubuntu !
(i’m about to package it but there are some issues like missing license files etc in the source tree before i can do it)

i spent another day to write a little script that generates html from the data in 5 second intervals, worked out an hydraulic plan in inkscape that i put in the back of that data and installed all that on a spare computer i had lying around with a minimal ubuntu and lighttpd.

the actual app

since i travel a lot and for the fun of it, i also wrote an rsync script that syncs the sensor and valve status data to a public place in longer intervals, so i can actually check the system while travelling the world (just click the picture above and you can actually see the current data)

so that was my little holiday adventure i wanted to share. if you have a similar system and are intrested in getting the script or the inkscape file to create your own hydraulic plan, feel free to contact me. i’m happy to share and i actually plan to move forward with that stuff up to a point where i can have proper mrtg graphs that you can combine to optimize the system, a desktop tool to do all that from your local machine and indeed i want to turn the website into an interactive webapp i will put on a touchscreen next to the heating so you can actually steer the system through it, but that will be work for another holiday season.

what really amazes me is that all this is possible with ubuntu and its open source tools. my little adventure wouldnt have been possible without all the work you guys do every day on the distro and on the thousands of open source projects. thanks so much to everyone whose software helped me to make that little project become reality …

or to say it in ubuntu: ‘I am warm as I am because of who you all are’ ๐Ÿ˜‰