The RISC OS File Repository opens | Drobe.co.uk archives

The RISC OS File Repository opens

By Chris Williams. Published: 2nd Aug 2005, 20:16:04 | Permalink | Printable

Finding those hard to reach applications

Press release

Wouldn't it be great if you were told when your favourite application has been updated? How about if you recall an old yet useful program that you once saw on a magazine cover disc? How would you find that?

Drobe webmaster Ian Hawkins has unveiled The RISC OS File Repository, an online software database for RISC OS software hosted on the drobe.co.uk server. The new website employs a custom web spider that crawls the 'net for new and old RISC OS applications, building up our database as it roams the web. Client software can connect to the File Repository to search for particular software and receive results in a simple to parse XML format.

This could allow users to automatically receive notifications of new and updated software, with a suitable client.

An RSS feed of recently found software is available. For software developers, documentation on how to connect to and use the File Repository is available. Application authors can also upload their software to be included in the database, and anyone can log in to help catagorize database entries. You can use your Drobe login details to access these features of the File Repository, although you do not need to log in to search the database.

Before overseeing the hosting and development of drobe.co.uk, Ian also administrated the popular AcornSearch website, which was later merged into the Drobe group. The RISC OS File Repository is based upon the AcornSearch system, which is billed as the most comprehensive search engine dedicated to the RISC OS platform.

For ideas, support or to suggest features, please just drop us an email.

Links

The RISC OS File Repository

Previous: Castle introduces Iyonix cube
Next: ROL tells Select users: A9 takes priority

Discussion

Viewing threaded comments | View comments unthreaded, listed by date | Skip to the end

Had a quick test and it looks very good.

Revin Kevin on 2/8/05 10:35PM
[ Reply | Permalink | Report ]

what a fantastic idea, great stuff! been wanting something like this for ages, well done :-)

leeshep on 2/8/05 10:39PM
[ Reply | Permalink | Report ]

Mr Hawkins sir this is a *very impressive development*, congratulations on a job well done.

AMS on 2/8/05 10:43PM
[ Reply | Permalink | Report ]

As the years roll by new repositories come into being and old ones fall away. Perhaps we should have a repository of repositories, with brief critiques ("all this stuff is 26-bit only", "no dates given"). My principal complaint about previous efforts is that insufficient information was given about: date of first insertion into database, what platforms will the item run on, how does the author ensure that the information is uptodate, date of last update. Nothing kills off a public spirited enterprise like this so much as being weighed down with obsolete stuff. Unfortunately squirrel mentality, which we all suffer from, keeps whispering in our ear "we might find a use for this again some day", when common sense says "chuck it".

GavinWraith on 3/8/05 9:14AM
[ Reply | Permalink | Report ]

I'm a bit concerned that this is a duplication of effort (ANS Filebase, Nutshells) and a quick test looking for my own software has discovered that it is (at the moment at least) already out of date! (I did a search for "HelpScan" and it presented version 0.12 (with "no description"). The latest version is 0.6.)

Adam

adamr on 3/8/05 9:16AM
[ Reply | Permalink | Report ]

adamr> You forgot 'Frax' in that list as well. The difference with this is that I finally got sick of outdated links on the other sites where you try and download the program and the link no longer works. With this effort:

1) You will always be able to download the program. 2) If you ever need a particular version (A pre 32bit version for example) then if it has been categorized, then you can download it. Multiple versions are not a problem. 3) The database can easily be extended to offer a 'OS Version' the program will run on if so desired. I will look at this in the coming week.

And btw, there is nothing stopping you from uploading and categorizing a new version of your 'HelpScan' (or any other) software to the database. Just login and click the upload link - thats what it's there for.

piemmm on 3/8/05 9:37AM
[ Reply | Permalink | Report ]

Should I upload entries for the 100 or so versions of software I maintain? No, I don't think so. The problem with this, and all recent file base efforts is that invariably, way too much manual effort is involved. Unless the much more serious problem of getting and installing RISC OS software is addressed (cue rant about packaging), then this can only every be a bandaid. But I've said all this before.

It's high time RISC OS development started thinking about long term solutions to problems rather than the "quick fix" and "I just felt like it" mentality that we still suffer from as a result of some of Acorn's attitudes towards OS development.

mrchocky on 3/8/05 9:42AM
[ Reply | Permalink | Report ]

mrchocky> "Should I upload entries for the 100 or so versions of software I maintain? No, I don't think so" You don't have to. The search engine should automatically find your software (some of which it already has) and add it to the 'this needs to be categorized' queue where people can collaboratively categorize it.

" Unless the much more serious problem of getting and installing RISC OS software is addressed (cue rant about packaging), then this can only every be a bandaid."

This is *ONE* of the reasons why the API exists so that people will be able to do exactly this kind of thing once someone writes a program.

"But I've said all this before"

And I listened.

piemmm on 3/8/05 9:46AM
[ Reply | Permalink | Report ]

It should find my software? Really? It's found NetSurf, FirefoxI, but not Firefox itself - oddly enough, hosted on your own server.

I can't see any of my own software there, but if you say so - I haven't looked through everything. But if it _has_, then it's clearly ignoring robots.txt, which on my site excludes everything but Google.

Having a quick look at the API, and ignoring the typos (5 results were returning for your query's (sic)), this addresses none of my concerns - it's just a query interface, which is all very well, but a completely different type of thing.

"And I listened". Without being even more blunt, what concerns of mine did you listen to? I didn't for example get any email from you about this requesting input.

Let's note some practical examples, and you can tell me which of these your database addresses or plans to address.

NetSurf requires no less than 5 other components (all used by other programs as well). How can users (or your system) determine that without quite a manual process of downloading it and checking though all the docs/start up error messages/following all the links on the NS page?

How can your database detemine which is the newest version of a piece of software given the varied naming and versioning schemes people use for versions of their software?

What if someone doesn't want their software listed (for whatever reason) or categorised differently because it might be a test version, etc.

What about the very tedious process of building up software and a boot sequence from scatch - there's around 10 different things you need to download before you can run many modern applications. An excellent way to scare off new users.

What about being able to download source to a program? License information? Removal of old versions of software on your system when you upgrade?

As I said, it's a bandaid. All the above issues I've named, a proper packing system could address. Much as I'd prefer not to be critical of new initiatives, it's hard to see how this offers anything particularly new over others that have gone before. RISC OS does not need more ad hoc solutions - fully thought out solutions will persevere much longer and ultimately be much less grief.

mrchocky on 3/8/05 10:28AM
[ Reply | Permalink | Report ]

mrchocky> "(5 results were returning for your query's (sic))"

Oh please. You chose to ignore the space between that and the search term enclosed in 'quotes'. I'll read the rest when I can be bothered if you're going to start being that petty.

piemmm on 3/8/05 10:32AM
[ Reply | Permalink | Report ]

Although I didn't quote the message exactly, I can't really make sense of what you're on about. If that's your only response, then who do you think is being petty here?

How about some serious and constructive debate? I've named some serious issues faced by RISC OS users, that are currently unaddressed by anything that exists. Ignoring them won't make them go away.

mrchocky on 3/8/05 11:06AM
[ Reply | Permalink | Report ]

piemmm:"You forgot 'Frax' in that list as well" Yes, and dmoz and PV's links database, the list goes on...

"And btw, there is nothing stopping you from uploading and categorizing a new version of your 'HelpScan' " Well, no, but given the number of links/database sites which potentially need updating I've given up and only make the effort to keep my entries on ANS Filebase up to date. Anyway, isn't the "unique" thing about your system that it's supposed to be automatic? Do you know why it is picking up an out-of-date version of HelpScan? Is there anything that can be done to improve it?

Adam

adamr on 3/8/05 11:40AM
[ Reply | Permalink | Report ]

adamr> "isn't the 'unique' thing about your system that it's supposed to be automatic?"

It is as much as is possible, however due to the extreme ways people 'document' their program there's only so much you can do to automatically scrape the required information from within the zip archive. The ability for everyone to collaboratively categorize programs is supposed to help alleviate this problem, but it isn't great.

The database contains several versions of helpscan, up to 0.5 i believe. this is one of the design features of it. When you originally did your search duplicates were automatically removed from the results, I have this turned off by default now and if you would like to redo your search you will find quite a few versions of HelpScan have been indexed. As to the reason why the absolute latest isn't on there yet (0.6?) is probably because the search engine just hasn't got around to refreshing your page yet.

"Is there anything that can be done to improve it?"

I'm reading peoples comments on here, and taking the constructive ones into consideration and already have a few ideas for how to improve things. As the site is fairly new (A month to think things through and a week of evenings to write the code) I have a fair amount of work to do to make it even better and fix other peoples concerns. Hopefully in a month or two it will have addressed most if not all of these.

piemmm on 3/8/05 12:02PM
[ Reply | Permalink | Report ]

I wonder if defining a simple XML format which people can put on their sites to describe what software their sites provide would help. These could be served as static files, with a standardised name and indicate exactly what is present, they could specify dependancies (care is probably required here), versions, compatibility, contact details and include links to other sites.

jamesp on 3/8/05 12:23PM
[ Reply | Permalink | Report ]

jamesp> Yes. I considered the file on website approach as well as several other options. In the end I decided that it is more correct for a file containing that kind of metadata to be kept inside the actual application directory (inside the zip archive / other distribution medium). There are a couple more documents to go up on the site once I have finished writing them, then people can comment/criticize them and hopefully agree on some concrete way that people will actually use.

piemmm on 3/8/05 12:30PM
[ Reply | Permalink | Report ]

I can see how this could be very effective.

I went to 'RISC OS search' the other day and typed in PDF it gave me a direct link to a zip of PDF112 (three versions out of date)

If only there where standards of how web sites should be organised. Are there any proposed?

CJE on 3/8/05 12:40PM
[ Reply | Permalink | Report ]

I imagine that if this system becomes completely comprehensive then I suppose the other program list sites will wither away, solving the multiple place problem - as google has pretty much done for internet search.

mrchocky, what exactly do you want? You seem to be complaining that it hasn't found programs, and that if it had it shouldn't have because of Robots.txt . You can't have it both ways. I'm sure packaging programs are a good thing - for programs which arn't entirely self-contained. Maybe the packaging system already under development could interface to this site? This could be very elegant indeed! Maybe the spider hasn't managed to trawl the whole web yet. I'd imagine that once it finds relevant files it looks at those sites more closely, and perhaps regularly.

Is the spider completely automatic? Or does it identify potential new files and permit some manual classification? This might solve mrchocky's versioning system issue.

From a quick look at the site, I have to say there does seem to be a bug or two. Clicking on a category gives you a blank page, and [link] reports a size for darkwood of 4k!

That said it seems to have found quite a lot of programs. If it did this entirely automatically it is impressive. Could you give us a description of how it works? Presumably you seeded the spider with a big site like Drobe and let it wander... but how does it extract the information? OK so if it finds a suitable archive I can see it decrypting it, looking for a directory structure with an !apps !sprites, !help and so on, but how does it know what they are, where to file them, or what is a useful summary? (My guess is that you do it manually.)

Loris on 3/8/05 12:41PM
[ Reply | Permalink | Report ]

I haven't complained that, please read what I wrote. Firefox is hosted on drobe so _should_ have been indexed. But my website excludes robots other than Google, so shouldn't have been.

I've made some of my specific wants quite clear - they can never be solved fully by such a system as is describe here. If you want more detail than I've already named, I refer to you articles on drobe on packaging.

Suggesting such a system might interface to it (and I can't really see how off the top of my head) is all very well, but you need specific examples.

With so few developers, and apparently even fewer users willing to get involved in one of the few activities they could make a real contribution to, it's important as much as possible be automated. The only practical way to do that is for the developers to provide pertinent information when creating archives - also automatically as possible. This can then be processed in whatever way by other systems.

mrchocky on 3/8/05 12:49PM
[ Reply | Permalink | Report ]

loris> ok, the filesize bug's been fixed, thanks for pointing that out

"...(My guess is that you do it manually.)"

Nope, the majority of it is completley automatic. You can see this for yourself...

First you have to login, then click the "Upload" button and chose a zip|arc|spk file containing an application. Once it has uploaded the repository will automatically extract as much as it can and make 'best guesses' as to what the application is, the sprites, help, version, description and author. These are then presented in a summary screen for you to amend/correct the guessed details (and choose correct categories) before they are comitted to the database.

piemmm on 3/8/05 1:00PM
[ Reply | Permalink | Report ]

What's the point of this?

We've had many sites doing a similar project before and they've all failed to be used and end up not being updated.

quatermass on 3/8/05 1:56PM
[ Reply | Permalink | Report ]

My main problem when looking for software with google (searching on; <app name> RISC OS), is all the links sites, or more accurately dead link sites, that completely swamp the results. The correct uptodate site is usually further down, but is far more difficult to find because of this.

Nice to see the prowl.org domain back online though.

druck on 3/8/05 2:10PM
[ Reply | Permalink | Report ]

mrchocky, I see what you meant now. But if your website excludes robots and the apps on it aren't present, doesn't that make your initial point redundant? I know piemmm said the spider had already found some of your files, but perhaps he meant something mirrored elsewhere, or which you also had a hand in, or mistakenly thought firefoxi was yours or something. In which case would you consider permitting his robot? Some of your points may be quite clear to those with the relevant knowledge, but maybe not to me. I didn't understand your point about the API being a query interface for example. Although I dabble in programming, I'm not well up on databases or web protocols, so please forgive me that. Regarding packaging, I read all the articles about it on Drobe, and after some discussion in the forums was entirely persuaded that it is a Good Thing. So I was being hopeful. Indeed, one of the problems with the packaging project (as I understood it) was the acquisition of the components. I'm not asking you to plan it all out - surely it is for the authors to arrange this. from [link] (last comment, by Graham Shaw in reply to me): "Actually, RiscPkg has already reached a state where it could install from a CDROM set. ... Other package transport schemes could be implemented as necessary, but if this left you having to manually download packages one at a time then you could reasonably question whether it was all worth the effort."

Grumbling that a system doesn't do what you want, then complaining when someone wonders whether it could be arranged to do so doesn't seem particularly reasonable.

piemmm, thankyou. That sounds like a good system to me.

Loris on 3/8/05 2:11PM
[ Reply | Permalink | Report ]

No, pie is well aware who authored FirefoxI, after all, there's a drobe article on it, and I voiced opinions on it. And to the best of my knowledge, my programs aren't mirrored anywhere. Yes, I will permit his robot _if_ he fixes it to honour robots.txt - I can't condone a robot that deliberately ignores such a thing, as he has insisted he's free to do in the past.

"Grumbling that a system...". I've not grumbled at all. I've pointed out that the system is a long long way from addressing the increasinly serious points that RISC OS users have in accessing files, regardless of how it might be "interfaced" in some arbitary and unknown manner. These are points I've been consistent about for a long time, already have most of a solution for, but which have been comprehensively ignored by most people for reasons I can only conclude as apathy. If pie wants to try implementing his own system, then it's unrealistic to expect it to escape scrutiny in this manner.

mrchocky on 3/8/05 2:52PM
[ Reply | Permalink | Report ]

I felt that RiscPkg ([link]) was a good solution to the problem of finding updated versions software. Should software developers now start supporting both RiscPkg as well as uploading versions of their software to this new repository? Did someone, once again, say "duplication of effort"?

The RISC OS developer-base seems to have difficulty understanding the meaning of collaboration.

jonix on 3/8/05 3:51PM
[ Reply | Permalink | Report ]

RiscPkg is a client. The File Repository is a database.

piemmm on 3/8/05 4:18PM
[ Reply | Permalink | Report ]

That's a worryingly naive simplification.

RiscPkg is certainly a client program, but it also maintains its own database as well as accessing external ones. These external databases are somewhat different in nature to what you're providing here. For one, they contain all the versioning, dependency information, etc, in a consistent manner, and would in practice be generated by developers from their software. If what your filebase was doing was collecting _this_ information, then that would be quite a different matter. Something akin to [link] (but more friendly and RISC OS oriented).

And before I hear anything more about "program that have dependencies", a large proporition of new or recently updated software _does_ have dependencies. Sometimes it's only the 32-bit SCL or a web browser to read the documentation, but there is also the toolbox, misc modules like SharedUnixLibrary, perhaps a specific OS version, etc, etc.

mrchocky on 3/8/05 4:38PM
[ Reply | Permalink | Report ]

Sounds like a plan... maybe that was the plan all along and too much words and emotions were spilled (again).

Jaco on 3/8/05 5:30PM
[ Reply | Permalink | Report ]

Sounds to me as if RiscPkg will be good at maintaining reasonably current software, while pie's repository will be better at helping people find legacy stuff for their legacy RISC OS.

Or am I being naive too?

steelpillow on 3/8/05 8:30PM
[ Reply | Permalink | Report ]

piemmm: If I click to "categorize" (sic) HelpScan it presents version 0.3 - even though version 0.5 is the latest it's found.

Adam

adamr on 5/8/05 4:48PM
[ Reply | Permalink | Report ]

Please login before posting a comment. Use the form on the right to do so or create a free account.

+++ Message board +++

Read messages or start a new thread

Search the archives

Today's featured article

Online banking with RISC OS
We thought this would be of high interest

28 comments, latest by devine on 25/9/05 8:30AM. Published: 1 Sep 2005

Random article

Wakefield 2007 live news and photos
The fun starts here

29 comments, latest by AW on 24/5/07 11:34PM. Published: 19 May 2007

Create a new account
Forgot your password?

Useful links

News and media:
Iconbar • MyRISCOS • ArcSite • RISCOScode • ANS • C.S.A.Announce • Archive • Qercus • RiscWorld • Drag'n'Drop • GAG-News

Top developers:
RISCOS Ltd • RISC OS Open • MW Software • R-Comp • Advantage Six • VirtualAcorn

Dealers:
CJE Micros • APDL • Castle • a4 • X-Ample • Liquid Silicon • Webmonster

Usergroups:
WROCC • RONE • NKACC • IRUG • SASAUG • ROUGOL • RONWUG • MUG • WAUG • GAG • RISCOS.be

Useful:
RISCOS.org.uk • RISCOS.org • RISCOS.info • Filebase • Chris Why's Acorn/RISC OS collection • NetSurf

Non-RISC OS:
The Register • The Inquirer • Apple Insider • BBC News • Sky News • Google News • xkcd • diodesign

© 1999-2009 The Drobe Team. Some rights reserved, click here for more information
Powered by MiniDrobeCMS, based on J4U | Statistics
"..your knowledge seems to be below par"
Page generated in 0.1322 seconds.