Drobe Launchpad | One-Step Compilation

One-Step Compilation

Published on 3rd Mar 2005, 15:01:56, source is drobe.co.uk
By Peter Naulls

Automatic building in action for RISC OS

The porting process on a large scale introduces many new problems - and also opportunities for automation and removal of tedium. In this article I discuss the ideas behind a RISC OS "autobuilder".

A builder Introduction
When I started the Unix Porting Project, it was clear that the volume of libraries and programs I would have to deal with would soon grow to be very large, and that manual building of even a small number of them would be time-consuming, tedious and error-prone. This would likely be compounded by the quick development of many of the libraries, meaning that new versions would have to be regularly built.

The solution to this is probably obvious - to automate as much of it as possible - although how to go about this perhaps isn't immediately obvious. Some of the techniques have been covered in previous articles about Porting Tools and later on, about Cross Platform Development. These tools (which have now incidentally, been integrated into GCCSDK), take the drudgery out the actual porting process and ensure a level of consistency.

Clearly that's great, but many packages require extra steps, patches or slightly unusual things to be done so they are built correctly or at all. Once we know what these extra things are, we don't want to have to redo them every time the package is rebuilt. And even when there are no special steps to be done, that still leaves the process of fetching the latest sources, unpacking, applying any internal patches, installing and/or zipping up to make a RISC OS application as well as the usual configure and build commands.

Designing an Autobuilder
When we talk about an "autobuilder", this is just a generic term meaning a system or program that builds some sources automatically with little or no human intervention when set up. There are many types of autobuilder, and they need to be designed for a specific purpose, so don't confuse the one I have developed with others you may have heard of, or tasks you think it might be applicable for.

The autobuilder I developed was designed simply to do one thing well - fetch package sources and apply a standard set of operations to them (as well as instructions specific to that package if need be) in order to produce a RISC OS port. Sources would be obtained from the Debian package repository - not only because I could use the Debian package update mechanisms and obtain source with a single command, but because it also covered over 95% of the programs that the UPP deals with. Also less common now, but at the time that the UPP began, it was common for Debian sources to contain fixes for ARM specific behaviour - these days, developers tend to be more aware of cross platform issues or fixes have long since made their way upstream.

This would exclude sources fetched from CVS or non Debian-packaged programs, but there are mechanisms for adding these later, and the advantage of being able to automate the rest was considerable; and those others wouldn't be any worse off. Naturally, you'd also have to be on Debian system for this to work.

The Autobuilder in Action
I started developing the autobuilder a few years ago, and it has had incremental improvements ever since. Recent changes include the ability to package up libraries so others don't have to build them themselves. This was covered in another drobe article. Any library that the autobuilder builds successfully is packaged up into formats suitable for use on Unix and RISC OS and immediately published on riscos.info.

Most recently of all, I've removed some of the assumptions about locations of files and programs so that it wouldn't be as tied to my system (it used to always build packages in /tmp, for example). This would potentially enable others to make use of this powerful system and contribute to it. It also meant I could contribute it to the GCCSDK, as I did recently, with at least the potential that it would work for others. As it stands, there are still probably a handful of issues that will cause problems if you are using it anywhere else. These, however, are minor and can be fixed with a little effort.

As it stands, the autobuilder contains instructions for building over 120 programs and libraries. Many of these are common libraries that you may well have heard of - zlib, libjpeg, libpng, libxml as well as programs published by the UPP - wget, whois, lincity. Others are obscure - either required by a few programs or have been built just because it was easy to do so.

At any given time, it's very unlikely that everything will build successfully. The presence of a package in the builder means that it probably built correctly at some time in the past. Breakages are often caused by normalisation of a package's build system (such as moving to autoconf) or a RISC OS patch no longer being correct. Usually, fixes are straightforward, but the volume of changes over such a large number of packages has been sufficient for me to not be able to keep up with it in addition to all my other work. I hope that others will now be able to contribute fixes, and improve mechanisms so that upstream changes are less likely to break our autobuilt packages.

Trying it out
If you've got GCCSDK already checked out of CVS and built then you will already have most of what you need to proceed. If you haven't done this, then you may wish to re-read the Cross Platform Development article. You should make sure you have the most recent GCCSDK sources by executing a "cvs update" too.

You will also need to check out the autobuilder module from GCCSDK CVS. e.g.:

cvs -d:pserver:anoncvs@gccsdk.riscos.info:/usr/local/cvsroot co autobuilder

The autobuilder presently presumes it's located in /usr/src, although there is a configuration line in the autobuilder/build bash script which you can change.

I can't promise that the following will complete correctly because of the previous caveats about the status of the system, but to give a flavour of how it works, change directory to somewhere that it won't matter if you make a mess (/tmp if you like, but not inside the autobuilder directory itself), and run the following command:

/usr/src/autobuilder/build -v -d cli/bzip2

In principle, this fetches the sources to the bzip2 program, unpacks it and builds it for RISC OS. The -v flag indicates to display progress, and the -d flag tells it not to delete the sources once it's done. For this particular program, extra instructions haven't been given in the "setvars" script to tell it how to create a zip of the result, but many packages do have these instructions.

And that's mostly it - it's meant to be that easy. There are many more packages you can try building. Some of these rely upon the X libraries which for various reasons haven't been included in the autobuilder, but there are instructions on riscos.info for building these manually. Adding a new library or program to the autobuilder can be tricky, although the resulting "setvars" script is usually very short. For well behaved libraries, often only a single line in the script is required telling it to be installed once it's built. For more information, refer to the manual which should have been checked out with the result of it.

The Autobuilder and RISC OS ports
One of the important points I've tried to impart to developers is that it's very easy to do a port - especially of some of the SDL games - and just about anyone, even with virtually no programming experience, can run my tools and produce a working binary and whack it in a zip and stick it on a website.

That might be ok - at least in the short term - but what happens if there's a new version upstream and a recompile is needed? Then the same manual process is required, and probably more effort to repeat the setup if someone else wants to build it to look at bugs or add features. That's why I'd like to encourage people to add ports to the autobuilder so everyone, including the porter, can benefit. By all means, stick your name on your work - neither the UPP nor the GCCSDK team wish to claim credit for your work - but such contribution will reduce a common problem that RISC OS currently has with Unix ports - there are multiple versions by different people spread around the internet.

Naturally, this presumes that the porter has access to a Unix system to run the tools (whether that be Linux, Cygwin under Windows or something else, and I fully appreciate that in many cases, that may neither be practical nor desirable. Sadly, RISC OS is still a long way from being able to run the types of tools required to directly run build systems for most of these programs. This is at least partially mitigated by having the system package libraries for RISC OS also. The NetSurf build instructions, for example, directs people to use at least some of these libraries, although it happens to have some unusual library requirements (mostly to reduce size) which means it also provides some of its own.

Fixing Autobuilder Deficiencies
As you've probably gathered, there are still a number of holes in the system that it would be good to fix. The most obvious is to be able to fetch from non-Debian sources, using wget with a fixed URL or from a CVS repository. The former is relatively straightforward to add, but there are down sides to this too. If we start fetching a specific URL, then that ties us to a specific version, and means manual checking and changes to the URL if we want to look for a newer version later on. CVS fetching is very easy too, with the caveat that we are potentially building a bleeding edge version (or one that might not even build at all). Sometimes is desirable to fetch a CVS version with a given date or tag.

It's also possible to arrange to fetch Debian sources on non-Debian systems - perhaps with ports of the Debian tools to that system. The particular advantage of Debian sources is that it's normally a known working piece of software (at least, under Linux) and that we fetch the latest version of that without doing anything special.

Other problems include assumptions about paths and no current support for build dependencies. Many packages require other packages to be built and installed first before they will build - for example, graphics libraries. In principle, you could arrange things so that when you directed a program to be built, it would first build all the libraries it depended on (and all the sub-libraries, etc.) before building the program. Finally, the system should be flexible enough to accommodate building any package that we need to - including the X libraries which are currently excluded. If there is a situation in which everything can be build with just a few commands (even if it takes a while), then we have a very powerful and easy to use system even for novices.

It's worth noting also that even though the main purpose of the system is to compile ports, there's no reason why it should be limited to that. If your original RISC OS program publishes its sources online, then the potential is there for them to be fetched, built and published by the autobuilder.

In Closing
I hope I've given you an idea of what the autobuilder is capable of, and I hope that developers doing their next port will consider adding their efforts to it. I'm sure it will prove to be a valuable resource.

Links
GCCSDK - GCC for RISC OS
riscos.info - RISC OS C programming information
Unix Porting Project

This article has been linked to, or is available in the following formats:

[Printable] [Digg this] [Blog search]

Design and concept (c) Fudgecake Design, 1999 - 2001. Content (c) The Drobe Team, 1999 - 2006. See www.drobe.co.uk for more information. For non-commercial personal use only.