To minimize the need for manual image testing of our ARM images, the canonical QA team invited me to their team sprint this week in the Lexington test lab to help with setting up a fully automated infrastructure on a bunch of pandaboards.
If you know the pandaboard you might also know that it can boot from its mini USB port as well as from an SD card. As long as the SD is empty it will listen on USB for the first stage bootloader file (which then pulls u-boot via the USB connection).
The central design idea was to have a single machine with USB hub serving as an initial bootloader dispenser for all the other pandas (indeed we could have taken any kind of machine but there were some discontinued panda models around that we don’t use anymore so we picked one of these as the central server).
Once the bootloader is completely recieved and executed on the client pandas they default to do PXE booting and pull their kernel and initrd from a tftp machine over the network.
For this initrd i developed a small initramfs script (and matching initramfs-tools hook indeed) that simply streams the most recent Ubuntu image (location and name are configurable via a kernel cmdline parameter) from http directly to the SD card, mounts the first partition and dumps a bootloader configuration (similar configurable via a cmdline parameter) in place which has references to a remote debian-installer preseed file before it reboots the board into a fully automated installation.
In case there are issues all the pandas can be power cycled through a web-controlled power strip and indeed all of them are hooked up to a serial console server so you can access their serial console remotely in case your preseed file didn’t tell it to install sshd …
At the end of the install the beginning of the SD card gets zeroed so the panda thinks the card is empty and will boot via USB again (in case the kernel hangs on boot or something similar fatal happens u-boot luckily provides an erase command for MMC’s that can be executed via the serial control server).
… So much for the theory …
Indeed we hit the first issue with the first step we tried to implement, USB booting only worked with one model of the pandas we had around, all the others simply didn’t pull their u-boot after the first stage bootloader was executed. Thanks to an impressing and tireless overnight debug and hack effort from John Rigby and Ricardo Salveti from linaro this issue was fixed on Tuesday. You can read Johns blog post about how to USB boot a panda on his blog, all fixes that were worked out should land in the quantal packages soon.
The next obstacle we hit was that TI apparently decided the panda should (like the beagleboard) be capable to be powered via the mini USB port (even though the current provided here is not sufficient to actually run the board unless you make sure that your kernel disables most of the power hungry devices on board) …
Read: we could not power cycle the boards remotely as long as the USB connection was in place …
I sat down and read up the USB specs. Theoretically the data transfer should not need the 5V connection to transfer its bits and bytes through the cable … so i opened an USB cable and just cut the power connection. Transferring data to my tegra2 netbook seemed to work this way (seems the port is always powered from the board side here), but sadly the pandas did not enable the mini port at all if there was no power applied to it from the other end …
Luckily there was a solution … the panda actually operates with 5V mains power so we spent our Tuesday with roaming around the nearby radioshack shops and buying barrel connectors (and sockets), shrink tube, a soldering iron and cable pieces. I then connected the power of the mini USB directly to the pandas main power connection
so that if you powercycle remotely the USB connection would be powered off as well.
Now that this bit was in place Carlos de Avillez (hggdh on IRC) who worked with me on this project and is a really awesome person to spend your time with (pay him a beer if you meet him at the next UDS !!) took all the puzzle pieces i had created and put them all in the right places on the different servers of the test lab, applied the
preseed, bootloader config and image location info … and off we went auto-installing our first panda … 😀
The whole system will soon drop its results into a public jenkins instance at https://jenkins.qa.ubuntu.com/ so you will be able check installation results for every daily image build of our omap4 images (once Carlos integrated all this which is currently in the works). In the future there will likely also be automated SRU kernel tests and similar other things running under this setup.
This was a really exciting week and i had a huge amount of fun with the whole QA team. Given they are still a young team in canonical in their current form and just started their work to improve the overall quality of Ubuntu i guess you will soon see a massive increase in stability and bug-freeness. I expect once all their plans and tools are in place many bugs will be found long before a user hits them (and needs to report them) in a release.