I don't think I had misunderstood what X was; the server runs on a device with a bitmap display, the clients run wherever they may be and communicate with the server using some form of networking protocol. That much, I've known since about 1985. I admit, though, I don't know if it provides a cut-and-paste mechanism, for example, or how it deals with video, but I don't believe that it doesn't affect the applications in some way.
One result of X's design is that if an application (client) writes to /tmp/fred, another application may not be able to access it, because its /tmp directory may be on another machine entirely. (That's what I meant when I was talking about a single directory tree.) That seem to me to be unecessary complication, when most people have only one personal computer.
Does ROLF need to run on anything without a framebuffer interface? I don't think so - that covers all current PCs with displays (AFAIK, but I'm sure at least 99% of them), and a lot of other devices, too, certainly every device I have in the house.
At the moment, the Xorg process is using approximately 15% of this 512MB machine's memory, what's that, about 76MB? The application itself is 1.5MB, to the Wimp's 84k. That leaves quite a lot of scope for optimising ROLF. The approach I'd like to take is to buffer the content of the slowest updated windows that are not frequently updated (e.g. vector graphics, or 3d renderers, rather than video or jpegs, maybe web browsers) in the graphics card memory, so that they only have to do the work once, when they need to, so that hopefully the application writers don't feel they need to do that themselves, which is counter-productive in smaller devices (using memory that isn't there).
At the end of the day, if what I want to work works, I'll be happy.