RISC OS Shared Libraries - RISC OS News, Software and Information

RISC OS Shared Libraries

Published: 29th Sep 2003, 12:41:42GMT Source: drobe.co.uk
By Peter Naulls

Page 1 of 1

One for everybody

Not exactly breaking news, but we've been meaning to cover this for sometime, and it remains relatively new to RISC OS.

Ask for the list of things people would like to see in RISC OS - Preemptive Multi-Tasking, improved memory protection, better browsing, etc, and possibly high on the list will appear shared libraries.

Shared libraries are just that - they are chunks of code that contain library functions that can be shared by many programs running at once. One of the important features of shared libraries is that they can be upgraded independently of the programs that use them, avoiding relinking.

Currently on RISC OS, if you want to use a newer version of a library with a program, you need to create a new binary, which will be statically linked with it, meaning that all the library code is included in your program's RunImage. This is particularly unfortunate for programs using things like UnixLib, which can be frequently changed.

There is an existing shared library system on RISC OS, but it's never been extensively used outside of the SharedCLibrary (I only know of one other example). It works via a stubs plus module system - whilst ok as far as it goes for the SharedCLibrary, isn't really ideal in general use. The code needs to exist in a module, which isn't the best, and you need a new SWI base for every module. It's all a bit fiddly and inflexible. Perhaps more importantly, it precludes automated generation of shared libraries from existing Unix build systems.

Of course, Linux, Windows and just about every other non-trival OS has a shared library system - in the form of .dll or .so files, but RISC OS has missed out, perhaps for historical reasons, and the minimal memory requirements of most RISC OS programs (memory saving being another advantage of shared libraries).

To the Rescue

Steven Simpson has started on a shared library system for RISC OS, and you can see his efforts on his RISC OS Shared Libraries page.

It's somewhat experimental, and certainly in its early stages, but you are of course encouraged to try it out. Steven tells drobe.co.uk he based his efforts on the requirements of the One True Dynamic Linker, in a document drawn up by ex-RISC OS user, Mark Wooding. Dynamic linking is a subset of the behaviour required by shared libraries, and essentially means code is loaded upon demand.

Steven's developments were done using GCCSDK under Unix, which shows you don't need to develop natively. Unfortunately, GCC itself can't yet create suitable code for dynamic linking in RISC OS (the problems being essentially those required to allow modules with GCC).

Hopefully in future we'll see practical use of this. In particular, UnixLib would greatly benefit from being a shared library.

Links
RISC OS Shared Libraries

This article has been linked to, or is available in the following formats:

[Printable] [Digg this] [Blog search]

nunfetishist (+3.6) 29/9/03 1:53PM	The obvious issue with using the SharedCLibrary approach is that you can't just load a new version, as all the current applications that are running that are linked to it expect it to be in a certian place. Killing the old one and loading a new one results in programs pointing to something random in the RMA. You at least to be able to see the explosion happen if you tried to replace the SharedCLibrary if it had already been softloaded. (Programs bound the the one in ROM don't care of course, because it's in ROM and can't be removed.) mdw has done shared libraries for RISC OS before. The problem with his original linker is that it needed the application to start the show off - it wasn't automatic when the program was loaded. Perhaps it's time for an ELF loader module that handles this for us, and start building ELF executables instead of AOF ones? One of the other interesting issues is where to put the shared code. You want to try to avoid putting them in the RMA because of its size limitation on many machines, and to try to prevent fragmentation. Suggestions have been raised about putting it in a dynamic area, although there can be problems executing code in such areas because of their location (there are of course ways around this...) Shared libraries are one of the things that RISC OS is missing that it should be possible to implement without breaking everything, like PMT and further memory protection would. I hope it's a success.
nunfetishist (-0.9) 29/9/03 10:52PM	DWARF is the companion format to ELF that deals with debugging symbols.
Kinnison (+2.6) 30/9/03 12:00AM	ELF is clearly the right format to use. It already deals with the issues of dynamic linking, runtime relocation, etc, and it provides consistent debugging structures. There exists already a plethora of tools for dealing with the format, and thanks to the way Linux uses ELF, there's plenty of Linux and GNU code to crib from assuming you're not after selling this loader in the end. While mdws' (straylight's?) OTDL was a good idea back before ELF, noone should be considering it these days. I believe (although I may be wrong) that even mdw has said that ELF is more sensible than OTDL.
nunfetishist (+1.1) 30/9/03 12:02AM	Pace's 32bit C/C++ tools used to be able to create and process ELF files. I assume Castle's can too.
mckinlay (+1.1) 30/9/03 11:44AM	ELF on RISC OS would be superb. I'd suggest looking at the BSD ELF-loading code, though, rather than Linux's (it's considerably easier to read). /usr/src/libexec/rtld-elf on your friendly FreeBSD system (I assume it's somewhere similar on NetBSD).
mrchocky (-0.9) 30/9/03 1:05PM	What would be even better would be for someone to do some work on this, rather than everybody saying how they think it should be done -- Peter, drobe.co.uk
mrchocky (+0.1) 30/9/03 2:22PM	So, your view of Shared libraries is totally biased by your experience of them on Windows. You can't compare your perceived failings of Windows shared library systems with those of RISC OS, because the orgnaisation would be completely different. Who said they were "old fashioned"? Please justify this statement. Your argument about corruption and upgrades is a red herring. The same thing could happen already with RISC OS modules. As for "realiability", if there's a serious bug in one of the libraries, only one thing has to be upgraded, unlike recompiling everything. The serious bugs in Unixlib in the past which could lock the machine are just this - every program that uses it needs to be recompiled. You deserve no less than slap for scaremongering -- Peter, drobe.co.uk
Dougal (+1.1) 30/9/03 2:39PM	The benefits of shared libraries are beyond just space saving, though that is a major benefit. As Chocky points out, they provide nice upgrade paths without the need to recompile entire applications. They are in no way old hat. For bigger libraries, such as X11 GUI toolkits, they are the obvious right choice. I'm not a Windows user, so I have no idea what the state of play is with DLLs over there, but on UNIX they're alive and kicking. -- Dougal
jools (+0.1) 30/9/03 7:01PM	ELF, eh? It had crossed my mind, but AOF seemed to be mostly adequate, and it seemed that I would have to understand AOF anyway to read the compiler output. I shall investigate ELF in more detail...
nunfetishist (+2.1) 30/9/03 8:27PM	ELF's advantage is that it's designed for dynamic linking from the outset, is well understood, has a broad feature-set and pretty much everything non-Windows uses it. It seems to be the clear choice. I wonder how tricky it'd be to make a riscos-elf gcc...
mrchocky (-1.4) 30/9/03 8:50PM	...gcc has nothing to do with ELF. The assembler, drlink and libfile has to be modified. We look forward to your patches -- Peter, drobe.co.uk
jmb (-0.9) 30/9/03 9:09PM	1x Basic ELF loader module: [Link: www.ecs.soton.ac.uk]
mckinlay (+1.1) 30/9/03 11:39PM	*In reply to mrchocky:* Er, granted it won't support the same syntax syntax, but binutils for arm--elf has been around for donkeys years. To get started, you should be able to configure it and gcc just for 'arm-unknown-elf' which will give you a working compiler, assembler and linker - though no knowledge of RISC OS, obviously. The crt init code can be cribbed from NetBSD easily enough, and then it's a case of building a library containing stubs for all of the SWIs using a syntax gas understands. (Isn't there a library containing all the SWI stubs already, that's autogenerated or something - I'm sure I read about it somewhere?) Granted, you run into issues if you want to use any inline asm, but taking one step at a time would seem the sensible approach. (If the libc is shared and SWI calls aren't inline, then there's even the possibility of sharing binaries between RISC OS+elf and other arm--elf platforms, which would be nice).
mckinlay (+1.1) 30/9/03 11:42PM	In reply to myself: Of course, all of this assumes either cross-compiling, or getting binutils itself working on RISC OS. I must confess to not having a clue how difficult the latter is...
mrchocky (+1.1) 1/10/03 7:35AM	I'm unsure how you think that contradicts what I said. It doesn't. GCC itself never deals with binary formats. It's not GCC itself, it's the other tools - specifically, the ones I've listed above. I don't really think you have a very good idea of what's involved - I have no idea why and where this mystery library you name needs to be created, of what the point of those SWIs are. Porting bits of binuntils to replace the programs I named doesn't help you very much. You still need to be compatible with AOF, and the assembler needs to continue to understand the ARM (as in the company) assemhler syntax. -- Peter, drobe.co.uk
mrchocky (+1.1) 1/10/03 7:37AM	Finally, you can't share libraries with ARM Linux/NetBSD because the code generated for them doesn't contain checks for chunked stacks - plus if you take code from non-Debian, it will contain half word accesses which won't work on RiscPCs. -- Peter, drobe.co.uk
mckinlay (+1.1) 1/10/03 7:44AM	Erm, my point was that you can get away without having to have a toolchain which understands AOF if you have one capable of producing ELF objects, libraries and executables and the OS (or some helper) knows how to execute ELF binaries. Being compatible with ARM assembly syntax is still a problem in this scenario, obviously, but for many applications that's not going to be a problem. The library I was referring to was OSlib (I've since re-found it). The point was that if OSlib could be generated and compiled for arm--elf, using the gas assembly syntax, then there would be a baseline libc for building RISC OS applications atop of right away*, without any need for the huge task of porting the existing toolchain to ELF. [FWIW I do have an extremely good idea of what's involved in porting a toolchain to a new OS or binary format, having had to do exactly that in the past.]
nunfetishist (+1.6) 1/10/03 11:00AM	*In reply to Quatermass:* ""Who said they were "old fashioned"? Please justify this statement." " "Errr. I did. " And in what way are you qualified to say that? On the corruption front, I have several Unix boxes here that use shared libraries extensively, and have been working flawlessly, in some case for years. I can't remember the last time a file on disc got corrupted that wasn't directly my fault, or accidental. Of course, the file systems on Unix tends to be a whole lot more reliable than RISC OS' ones, but that's just another thing that needs to be replaced. (And yeah, I know that somebody could go poking around in the shared libraries out of curiosity, and break things, but then that's no different from poking around inside the box with a pin to see what sparks - you get what you deserve.)
Dougal (+1.1) 1/10/03 11:19AM	Quartermass: "I did" isn't a justification "...how the end User is suppose to handle the control of the libraries." - Well, ideally they shouldn't have to control them. Ideally they shouldn't have to care at all. I think one of the major problems of shared libraries, to which you refer, is the naming of the things with repect to different versions. I have no idea how Windows handles this, but I suspect badly, In UNIX like things, such as Linux, the library file name is appended with the version number, which makes life easier. In Mac OS X the framework system provides the name and version from insider the framework, so the filename doesn't matter - frameworks (the Mac OS X shared library packages) register a name, major version, minor version with the OS when installed - what the file name is doesn't matter. When an application wants a framework it gives the OS the name and the major version it wants (possibly also the minor version, but minor versions are meant to be interface compatible). At the end of the day, Windows demonstrated that shared libraries without proper thought to how they're named (by name I mean the more computing science point of view, rather than just the filename). UNIX libraries and Mac OS X frameworks demonstrate it can be done. Just because you've not experienced it done right doesn't mean it can't be. Shared libraries are more than an aid to the programmer, they're an aid to the user - it requires less memory, it allows applications to be upgraded by just upgrading the appropriate library, reducing file sizes of transfer, it allows the system provider to ship a bunch of libraries and application programmers to rely on them. -- Dougal
illudium (+1.1) 1/10/03 11:38AM	*In reply to quatermass:* "So what do Users currently do? RISC OS users are expected to take a supplied copy of !System and merge it with the one in !Boot on blind faith. Do these new modules get version checked or md5sum'd to ensure they are newer and intact?" Yes, they get version checked, that's exactly what the SysMerge tool is designed to do. If users are getting in a mess with toolbox and sharedclib releases then that's a (hopefully temporary) political problem with 2 companies developing in parallel, not a technical problem.
mrchocky (+1.6) 1/10/03 4:39PM	*In reply to quatermass:* Three people with different OS backgrounds gave you the same answer. I don't think there's anything for me to add. *In reply to Mo:* Erm all you want, but you haven't adressed the issues - we already have a toolchain that understands AOF - getting away without one makes no sense. I've already pointed out the exact changes that need to be done for using ELF on RISC OS (static linking at any rate) - you've not addressed these points at all. I don't see what the relevance of OSLib is at all - C programs are no used without a C Library - i.e. UnixLib or SharedCLib, not a mere C Library, which is what OSLib is. You may well have experience with porting toolchains, but you don't seem to have much experience with the existing RISC OS GCCSDK. -- Peter, drobe.co.uk
mckinlay (+1.1) 1/10/03 5:23PM	*In reply to mrchocky:* Where's the technical sense in having a toolchain which mixes and matches AOF and ELF? ELF isn't just an executable format, it's an object file format, too. On systems which support both a.out and ELF, you don't expect the same toolchain to support both; RISC OS has no reason to be any different. Would you care to explain the difference between "C Library" and "C Library"? I'm using the term in the traditional sense - a small layer which separates the OS system calls (which, in the case of RISC OS, are the SWIs), from the applications. Any other utility libraries are be built on top of those. A libc is a starting point - the most basic layer of abstraction. UnixLib and SharedCLib work at the layer above, don't they - they provide additional functionality above that made available by the OS itself, and as such they would lie above the libc. There's a very good reason why systems which support shared libcs do so - it allows a very small layer of abstraction to be introduced, with very little sacrifice - a layer which allows the semantics of the underlying system to change without breaking everything built on top of it.
mckinlay (+1.1) 1/10/03 5:29PM	*In reply to mrchocky:* Oh, and one last thing - I DID address the issue. nunfetishist's original post posed the idea of an ELF toolchain for RISC OS. You suggested modifying all the existing RISC OS tools to support ELF (in addition to AOF), whereas I suggested it might be easier just to use the ELF-producing gas, objdump, objcopy, ld, nm, ar and ranlib which are part of binutils already - the effort required getting a RISC OS-oriented runtime library working with the existing GNU tools being considerably less than the effort required to make the GCCSDK talk ELF.
mrchocky (+1.1) 1/10/03 6:00PM	Where's the technical sense in not? RISC OS has a vast array of stuff that's already AOF. Contrary to your final claim, adding ELF support to GCCSDK is not particularly hard, and if the extra binutils later work on newly produced ELFs, that's great, but not particularl relevant. On the other hand, creating a brand new toolchain for RISC OS makes little sense - you have to modify GCC to produce sensible code for chunked stacks (non trivial), and figure out what you want to do with the resulting ELF binary in terms of loading it, or converting to Absolute, as well as dealing with the masses of minor RISC OS details such as dealing with paths, etc, etc. And after all that, you have two toolchains for RISC OS which won't interoperate - certain to cause frustration - sounds like a pretty pointless exercise. I'm really not sure where you're going with the whole C library thing. A compiler for RISC OS must have a C library that implements the standard C functions - only Unixlib and SCL fill this role. OSLIb is neither here nor there, regardless of whether UL/SCL call SWIs or not. It would be nice if you understood the details of GCCSDK before making any further judgements about what should and should not be done. -- Peter, drobe.co.uk
mckinlay (+0.1) 1/10/03 6:09PM	I see. You've turned a potential suggestion into "making judgements about what should and should not be done", and completely failed to grasp the concept of a shared C library as exists on most other operating systems. Tell you what, though, I'll spend a couple of days fiddling with the de-facto standard toolchain (outside of RISC OS), producing the standard format binaries (outside of RISC OS) with a view to them being useful for producing libraries and executables useable on RISC OS, and see what happens. I'm at no point saying my approach is what SHOULD be done, I'm saying it COULD be done, and COULD be a better option. Don't be so unnecessarily defensive when the aim is to make it possible for folk to use their existing (and considered standard) toolchains to produce applications for RISC OS.
mrchocky (+1.1) 1/10/03 6:19PM	Interesting conclusion given that none of my above commentary was about shared libraies, but rather about the specifics of toolchains that don't particulary go near that issue. On the other hand, my above article did clearly talk about concept of shared libraries, so it seems I do understand it. If you want to throw a tantrum, choose an alternate forum. Please restrain yourself to addressing the technical and practical issues. But if you do want to help, then please do. Carry out your suggestion by all means - I think the information gleaned will be very useful in understanding the issues, but, for reasons already explained, it is unlikely to be a useful practical proposition. -- Peter, drobe.co.uk
4 comment(s) are below your moderation threshold. Login to view them.
Use the forum for more comments on this article