A Linux 'resource manager' V1.0

Disclaimer
Introduction
Sources
Some basics
Creating a resource filesystem
Accessing resources
What do resources mean?
Portability
Future stuff

Disclaimer

Much of what I've written below is somewhat KDE centric - this is mostly because I'm familiar with and used to working with KDE - however the resource manager package is intended to be used in other environments (Gnome?) if possible - it's written in C with C++ bindings, in some sense it could be the base of an extension of the common .desktop files (see below).

Introduction

For many years I worked in the Macintosh world (to be fair I did a large chunk of the A/UX kernel - so much of my work was Unix work) - anyway, one of the things that I've really missed is the idea of a self contained application - a file you can drag around on the desktop that contains all the information it needs to work in the system.

These days however Linux GUI apps seem to require an exact installation of files in lots of different places, and don't work well if all the different pieces aren't in just the right places - for example the following sorts things that a KDE app might need:

the binary of the app itself
a .desktop file describing the app, and it's icon
the icon to be used
one or more mimetype bindings (and their icons)
icons used in internal dialog boxes
fonts
sounds
html files for documentation
localization strings

The goal behind the resource manager project is to make it possible for an application author to create a binary which encapsulates all of this information within itself.

Sources

They're LGPL'd and are available:

Version 1.0

Simply unpack and type 'make' (it's all very primitive at the moment) 'make tests' will make a couple of test programs you must have a directory hierarchy rooted in 'testdir' within the current directory.

Some basics

The resource information is stored in a container which is embedded within a code file, for this first release only ELF files are supported (adding a new file format may only take an hour or so's work, maybe 50 lines of C, and I'm quite happy to help people who want to expand support to embrace other file formats). You can think of it as a tiny filesystem embedded within the program's binary file. It's stored in a non-loaded section (meaning that it's not mapped into memory when the program is run). Access to the files within the 'filesystem' are with a set of library routines with similar semantics (and parameters) to the C-library open/fopen/opendir families of routines.

The embedded filesystem is read-only - you can't update resources and write them back like you can on a Mac.

This release consists of the programs you can use to create a filesystem and embed it into a binary and a set of library routines you can use to access the contents.

Creating a resource filesystem

This part is REALLY easy - it's a 3 step process:

make a filesystem image using the makeresource tool
embed it into an object file using the linker
link it into your final binary.

The 'makeresource' tool is a simple program which creates a binary filesystem image from an existing file hierarchy. Basicly all you do is create a directory hierarchy on disk and execute something like:


	makeresource -nocvs root_directory -o res.fs

This will take all the files and directories in the directory 'root_directory' and recursively below it and put them into the file 'res.fs'. If you specify the '-nocvs' flag then directories named 'CVS' and their contents will be ignored.

The next step is to embed the filesystem image into a linker (.o) file - you can do this easily using your friendly local GNU linker. First create a file called something like 'script' and fill it with the text:


	SECTIONS { .resource (INFO) : { *(.data) } }

Now link your file with the command:

	ld -r -oformat elf32-i386 -o res.o -T script -b binary res.fs

Where res.o is the object file you are creating, and res.fs is the file you created in the first step. If you are using a non-x86 based system you'll have to replace the 'elf32-i386' string with something that's appropriate for your platform - 'objdump -i' will give you a listing of the file formats supported by your linker.

Finally just link the resulting resource file into your application as you would any other object file. Linking more than one resource file into a binary is not supported and will result in undefined results.

Here's a fragment of what you might toss in your Makefile:


app:	.... res.o
	gcc	.... res.o .... -lres

res.fs: <list of files in root_dir>
        ./makeresource root_dir -o $@
%.o:    %.fs
        ld -r -oformat elf32-i386 -o $@ -T script -b binary  $<

Accessing resources

Once you've made a binary containing resources how do you get at them? The library libres.a contains C and C++ bindings to the access routines. They look and work like familiar low level libc routines - with the added requirement that you have to specify where the embedded file system is when you first open.

C Bindings

You can access the C routines using:


	#include "resources.h"

Before you open a resource you need to choose which embedded filesystem the resource is stored in, you do this by creating a RES_ENV structure - you do this by calling res_open_env() and passing either a string which points to the code file or by passing NULL which opens the currently running binary:


	#include "resource.h"

	RES_ENV	*fs;

	fs = res_open_env(NULL);			// opens the current code file
	fs = res_open_env("/usr/local/bin/app");	// opens /usr/local/bin/app

res_open_env() will return NULL is the resource file doesn't exist, or has no internal resource filesystem.

When you are done with a resource filesystem close it with res_close_env(fs) - it won't actually go away until the last user of it is done with it. Note that in all the C routines below where a RES_ENV can be passed you can also pass NULL - if you do this it will create a temporary RES_ENV for the lifetime of the call - doing this a lot can be quite inefficient - it's better to call it once and destroy it when you're done.

open/read/lseek/stat/close

The following routines work like their libc analogs - they use a RES_FD* structure rather than a normal int file descriptor (warning a failed res_open returns NULL, not -1). Notice how the res_open routine requires a RES_ENV parameter to specify where the filesystem is located:


	RES_FD  *res_open(RES_ENV*, const char *pathname, int flags);
	int     res_close(RES_FD *fd);
	zsize_t res_read(RES_FD *fd, void *buf, size_t count);
	int     res_rstat(RES_ENV*, const char *file_name, struct stat *buf);
	int     res_stat(const char *file_name, struct stat *buf);
	int     res_fstat(RES_FD *filedes, struct stat *buf);
	int     res_lstat(const char *file_name, struct stat *buf);
	off_t   res_lseek(RES_FD *fildes, off_t offset, int whence);

res_rstat() works like res_stat - but allows you to pass in a filesystem pointer, res_stat() and res_lstat() open the current executing binary's resource filesystem and return information from it.

fopen/fread/fseek/fclose

These are analogs to the libc stdio routines - they use an analog to a FILE* called a RES_FILE* structure. Again the res_fopen() routine has an extra parameter to show where the filesystem is located:

	RES_FILE *res_fopen(RES_ENV *, const char *name, const char *mode);
	int     res_fclose(RES_FILE *stream);
	size_t  res_fread( void *ptr, size_t size, size_t nmemb, RES_FILE *stream);
	void    res_clearerr(RES_FILE *stream);
	int     res_feof(RES_FILE *stream);
	int     res_ferror(RES_FILE *stream);
	RES_FD  *res_fileno(RES_FILE *stream);
	int     res_fgetc(RES_FILE *stream);
	char    *res_fgets(char *s, int size, RES_FILE *stream);
	int     res_getc(RES_FILE *stream);
	int     res_ungetc(int c, RES_FILE *stream);  
	int     res_fseek(RES_FILE *stream, long offset, int whence);
	long    res_ftell(RES_FILE *stream);
	void    res_rewind(RES_FILE *stream);
	int     res_fgetpos(RES_FILE *stream, fpos_t *pos);
	int     res_fsetpos(RES_FILE *stream, fpos_t *pos);

opendir/readdir/closedir

These are analogs of the standard directory traversing routines you can use them to search for files within embedded resource filesystems.

	RES_DIR *res_opendir(RES_ENV *, const char *name);
	int     res_closedir(RES_DIR *dp);
	struct dirent *res_readdir(RES_DIR *dp);
	off_t   res_telldir(RES_DIR *dp);
	void    res_seekdir(RES_DIR *dp, off_t offset);
	void    res_rewinddir(RES_DIR *dp);

C++ bindings

You can access the C++ routines using:


	#include "cresources.h"

You start by connecting to a resource filesystem:


	#include "cresources.h" 
	res_env *fs = new res_env(NULL);		// connects to the current file
	res_env *fs = new res_env("/another/file");	// connects to one in /another/file

When you're done delete the pointer.

Next you can create one of 3 types of objects:

low level filesystem access
high level filesystem access
directory searching

These are basically analogs of the 3 types of libc accesses described in the C section above - you create them:


	
	res_env  *fs = ...			// as above
	res_fd   *fd = fs->open("a/b/c");	// open a file for low level reading
	res_file *fl = fs->fopen("a/b/c");	// open a file for high level reading
	res_dir  *dp = fs->opendir("a/b/c");	// open a directory for searching

Don't forget to delete these when you are done - you can happily delete the res_env after you've opened all the objects depending on it, the underlying data structures will go away when everything else has been deleted.

res_fd

A res_fd object supports:


	int res_fd::read(void *buf, size_t count);
        int res_fd::fstat(struct stat *buf);
        int res_fd::lseek(off_t offset, int whence);

res_file

stdio res_file supports:


        int  res_file::fread(void *ptr, size_t size, size_t nmemb);
        void res_file::clearerr();
        int  res_file::feof();
        int  res_file::ferror();
        int  res_file::fgetc();
        int  res_file::ungetc(int c);
        char *res_file::fgets(char *s, int size);
        int  res_file::fseek( long offset, int whence);
        int  res_file::ftell();
        int  res_file::rewind();
        int  res_file::fgetpos(fpos_t *pos);
        int  res_file::fsetpos(fpos_t *pos);

res_dir

directory searching object res_dir supports:

        struct res_dir::dirent *readdir();
        off_t  res_dir::telldir();
        void   res_dir::seekdir(off_t offset);
        void   res_dir::rewinddir();

res_env

res_env supports the following methods:

        int      res_env::stat(const char *file_name, struct stat *buf);
        res_fd   *res_env::open(const char *name);
        res_file *res_env::fopen(const char *name);
        res_dir  *res_env::opendir(const char *name);

What do resources mean?

The $64k question - so far in this document I've been very careful to address mechanism and no policy, i.e. provide a clean underpinning but not talk about how they are used. At this level resources are pretty much a blank slate - just like files in a files system - for example in a normal Unix filesystem /etc/passwd and /bin/sh are at one level (file system) just a pair of files containing a bunch of bytes - while at another (the system) they have very well understood functionalities.

This section is an attempt to show some possibilities and to start a dialog about how this might happen in and around the resource manager. This is not intended to replace existing functionality - but instead to allow files with the appropriate resources do the right sort of stuff.

A simple example

KDE icons live in one of a number of places (~/.kde/share/icons, $(KDEDIR)/share/icons, etc) either an application is installed by a user (and the icons go into their local .kde directory) or by root (and they go into the global directory used by everyone). If the KDE icon loader code was change to also look in the current binary's resource filesystem as well as in the other places then and applications icons would follow it around as it moves and don't need to be installed by root.

A more complex example

On the Mac when you drag a new application into a system the 'finder' (the desktop manager) looks at the resources within the file (if there are any) and uses the information it finds there to automatically extract the equivalents of mimetypes, .desktop files and their associated icons so that the application's icon appears correctly on the desktop, and that when you open the appropriate data files the application gets run.

Within KDE we could change the desktop manager to look at files to see if they have a resource file system and if they do look at a well-known place within the resource filesystem to discover prototype mimetypes and .desktop files, then unpack those to automatically allow a file to do its stuff (just as we share .desktop formats with Gnome we could also share this same info).

Portability

While my personal focus is Linux that's because that's what on my desk, most of this code is very portable - I'd like to port this code to as many platforms as possible.

I expect that the following issues are likely to show up as we port it elsewhere:

How do we open the current code file (under Linux we open "/proc/self/exe")
What about non ELF file formats - deep inside the code assumes that the embedded filesystem is just a range of bytes with the binary file - if you have a different file format that allows us to embed data then adding the code to find the section offset/size is probably pretty easy
how do we get the data into a code file if you aren't using the GNU linker

Future stuff

I'm working on an alternate interface - purely for experimental reasons .... basicly the idea is to push the internal file system into the kernel and then allow access through the /proc filesystem as /proc/pid/resources/.... It's a cute idea - but I'm in 2 minds about its usefullness - to be truely usefull it needs to be ubiquitous and and work in places that are not Linux - it also only allows access to applications that are currently running so the stuff provided here would still be needed.