... jump right to motivation - part 3 ...
... this fundamental UNIX philosophy proved its robustness and reliabilty over the last decades.
A great range of devices and hardware, the cpu's, its kernel modules, processes, virtual ressources - everythings is attached to the filesystem
and accessible as an inode in the filesystem. This allows to address those system ressources as a filesystem path and use them for system calls.
Ken Thompson designed an implemented the original UNIX opertaing system at Bell Labs.
Later on he took this design principle a few steps further and was one of the creators of Plan 9.
In Plan 9 a process has not only a file associated to it in the filesystem - its representation is a folder containing files for information, arguments, status and controls.
Manipulating a process does not require system calls, it can be done by reads and writes on the provided files.
Accessing network ressouces is done in the /net folder via special directories for each protocol. /net/tcp /net/udp...
They build an interface to write control messages and read responses - the underlying hardware access, protocol handshakes etc. is managed by the OS.
When every system ressource has a representation in the filesystem, a strong and reliable permission system is needed to ensure security and allow or restrict
access to critical parts of the system. This is done by defining groups and assigning users to them. Each inode in the filesystem has ist own set of
reading, writing or executing permissions defined for a certain user or user group.
POSIX permissions are well estblished and work on all UNIX-like system
in the same way. This worked and still works over various generations of UNIX and Linux implementations independant of the underlying harware or file system driver
accesing them. Modern system can mount multiple disks/devices with different file systems transparently and seamless into on single file system tree. Distributed
file systems mount remote file system via network connections...
Numerous protocols were developed over the last decades, cryptographic components implemented etc. etc. one layer on top of the other.
Diving into the do's and don'ts of current system administration can be quite overwhelming and a big task to maintain nowadays...
But still - or even more - despite of all the heavy load and complexity :
[ everything is a file ] has broad shoulders and stands its ground.
Observing and studying the ever evolving fields in operating system development over the last 30 years with all improvements, its growing complexity, the envisioned possibilities
from then, the capabilities of today...
WOW! - we can do things nowadays, that were merely imanigable then.
The transparency of open source and open standars principles, the possibility to learn from other developers code and implementations, from their problem solving concepts...
And on the other hand - the growing (over)-complexity, tons of libraries, huge frameworks to manage all the protocols, gui and window managers...
Personally, i worked a lot with and on server applications. Starting with apache and perl, studying their internals - developing own tcp & http server applications,
implementing authentication and session handling for them, developing database applications - and then node.js arrived... i learned a lot from all of that.
And although there are numerous modules, libraries, packages for every task - at first i try to find out how is it done on OS-level,
how can it be implemented without the bells and whistles, but instead on solid ground...
Yes - that means to deal a lot with stdIO streams, buffered input and ouput, working with basic unix commands and system calls...
But one question was always there and never went away:
For a short thought experiment lets assume a few things:
- we ignore technical bits and pieces of transport layers and protocols
- we ignore encoding, cryptograghy endianess issues
- we pretend to send plain text messages between our services and processes
ELF executables
shell scripts
From a unix-like operating systems view, an executable file may have different forms.
On invocation, a script file will be passed to the specified scripting/shell environment, ELF executables
are directly passed to binary execution.
Each process has its stdIO (stdIn, stdOut, stdErr) assigned to some endpoints, the parent process or
a console/tty. ELF headers have additional information for which platform/cpu they were compiled, where inside the
file itself are the required binary and data blocks...
Without going into further technical details about how binary execution works:
After permission checks were successfull, the OS loads the file into memory, prepares the suitable execution context,
assignes a processID and creates an inode for the running process in the /proc path of the filesystem.
Next, the environment vars and arguments are provided for the created process, stdIO is routed as desired
and the execution starts running the main function of the program.
Additional arguments and information may be read in from stdIn, results be written to stdOut etc...
The running process may load files, process their contents and write them to the same or to another file...
On reaching the end of the main function, the process will exit and will be terminated by the OS.
By sending signals to the process, it may abort execution earlier etc.
In case the started process acts as a service or server, the main function is a (possibliy) infinite loop
waiting for requests on the bound socket/port and exiting only on predefined conditions.
... or what needs to be adapted?
Well, not very much! - stdIO are already streams (although they are bound to the execution environment
and a specific user and/or parent process)
writing/reading those streams requires knowledge of the process internals or the provided API,
sometimes accessing them through a system call or a binary interface.
[what if ??] a process would not only have its stIO streams bound to a specific user or parent process,
but also could expose additional streams for explicit data input/output or a stream to set/change
the internal status or variables of the running process or send commands directly to the process?
Very much like in Plan 9 a running process would then not only be a single inode in the filesystem,
it would be a directory containing a file/stream for each exposed data or command stream. Those streams
will have their own set of permissions (e.g. allowing only certain users to alter the state of the
process internals, but allowing publicly writing/reading of its data streams)
[what if ??] those stream endpoints can be passed around as an entity?
or as delegates with assignable permissions?
There may be dedicated data streams expecting a certain data format/protocol or responding on a specified one.
An additional stream may act as a management/status channel to the process. A user may send a message containing a
named key and the process will respond with the current value of the internal var - or the user may send a
message in the form of key:value and the process will set its internal var to that value.
Even more expenaded, the process may report its entire internal status (for example as an JSON object/string),
provide some functions/commands or minimal scripting functionality.
Does this sound familiar?
- like a service/server?
- or an interactive shell/interpreter?
Yes it does - every process would act on OS-level as a microservice or interactive component, consuming data and
responding with the processed results.
The contens of the streams may be dynamic, its representation on OS-level wouldn't. The process itself is
assigned to a specific user and its session context, but its streams may be passed arround as entities, even across
execution contexts. In addition, ownership of specific streams can be delegated.
No matter if dataflow to and from these streams are interactive or from files - the process digests data, processes them
as stream and writes them back to the desired stream. After all, files are nothing more than streams with a finite length.
If user processes act as microservice, could this work on system level as well?
[what if ??] my systems kernel could speak some sort of streaming protocol?
If there was some entry server - delegating system calls as streamed messages to the interconnected core modules?
And those system core modules would provide permanent data/command streams as interface, possibly exposing temporarily created individual
stream endpoints for a specific user?
If - per definition - everything is a streamable resource, the operating system itself should also be.
For every vital part of a running operating system there will be a specialized core module/process:
Another vital aspect of an operating system is security and integrity of data and execution
- on the other hand we want to access them by simply sending "messages".
Therefore we need some sort of "frontdesk", accepting and handling user requests
(maybe accepting different protocols and encodings) and passing them to the "backoffice".
Once a user request is accepted, it will be transformed to a common protocol and passed to
the core modules for processing.
First, the core modules should not be too talkative about their internals.
Second they should accept only authorized and syntactically complete requests.
To some extent, the system would work like elaborate web server applications do.
Most of the [what if ??]'s aren't genuinely new - for every part there are existing implementations.
Todays internet is in fact the one, big [everything is a streamable ressource] implementation.
As a living standard, which is ever growing and expanding, its defacto "standard" HTTP protocol,
all ist pitfalls and security breaches...
It showed all its strengths and vulnerabilities over the last decades and will serve as a blueprint
for the future do's and don'ts.
The system i'd like to see up and running can be implemented with currents tools and technologies as a
full-grown web server application on top of a unix-like system. Streamable system calls can be implemented
by wrapping C API systemcalls and exposing them to the webserver...
The drawbacks are at hand: the server application will inevitably become a bloated mess, consuming considerable
amounts of system resources and execution time - and most of them would be used to translate between the different
protocols and API's. Allowing users the execution of arbitary code, possibly containing
system calls, starting long runnging processes, managing file system accesses and session/authorization in an
per se stateless server environment has a lot of obstacles and tasks to master - especially security-related ones.
Whats missing - at least in my opinion - is a common ground to start from, a basic server/kernel which manages
permissions and auhorization throughout all system components, has a basic understanding of streams and
protocol negotiation and is as close to the basic operating system level as possible.
As one doesn't simply walks into mordor - noone, especially not me - is simply developing a new operating system.
Somewhere inbetween, there might be a viable way. As close to the underlying unix/linux system as possible -
regarding it as the systems "firmware" - and placing my system on top of that, implementing a dedicated
file system und user interface, decoupled from the underlying OS.
User interaction, whether via console, tcp/ssh or http(s), is always attached to a specific session.
Public users with no user account or users which are not logged in yet have an empty sessions with minimum rights.
This session instance is a first-class process, representing the users main execution environment. It is scriptable, holds the
current status, triggers/receives events, and provides interfaces to access the system components.
Further, all opened/connected streams, running processes and emmited/received events are bound to the session instance.
The system core itself holds a limited set of internal SessObj's, having elevated permissions. Access to those special SessObj's is restricted.
In some situations it may be desirable for a process to obtain resources from the system with other permissions and privileges as the original user.
In this case, a process will require an appropriate SessObj from the system and re-enter from this execution context creating requests from there.
(in the above diagram [IO] should have a fourth option - process delegates)
All system calls are handled through a ReqObj. For a user requesting resources from the system,
the ReqObj is created and handled by its SessObj. (note the green arrows)
Internal processing may create ReqObj's as well - they are also bound to a SessObj:
- if the internal request was created as a follow-up or nested invocation from a user request,
the sub-request will be bound to the original users own SessObj.
- if the internal request is purely internal induced, it will be bound to one of the special SessObj's of the system core.
The main bus is the primary entry point for new requests.
Its purpose is to provide access to the core modules by decomposing the request and assigning the parsed tokens
to its corresponding core modules and translates to their protocols.
It provides integrity checks for [ reqId & sessId ] and passes them to the core module interfaces.
The main bus provides the callback Handler for request cycling and is reponsible for request/message queuing and event propagation.
[ event, vfs, cmd, stream, sess ]
The systems core components are individual services/servers, each core component has its own interface based on a common protocol.
ReqId & SessId for these interfaces are provided through the main bus and cannot be manipulated through user interaction.
The core components themselves have an additional channel to the [ interop ] module. (note the red arrows)
More on each component... [ rwmax! - system components ]
The interop module and bus represents the systems backend channel.
It is not user accessible - its main purpose is to give the core modules a consistent and unified interface to the
internal pools and to split a top-level request into specialized sub-requests.
The interop module may respond directly to the calling instance OR by creating a new sub-ReqObj, looping it back
to the main bus and responding with the result of the sub-request.
[ reqPool, sessPool, procPool, streamPool, eventPool ]
- the pools themselves are not user accessible
- individual pool members are available by their respective Id through the provided core modules and interfaces.
- accessing pool members requires permission checks!
The users SessObj from above is directly accessible by its owner - viewing/accessing other/all sessions from
the sessionPool is only allowed for users with elevated privileges through the sessModules interface.
The same applies for requests, processes and streams - as all of them are bound to a specific session and user.
Some of the core modules have direct managment access to their corresponding pools. (not shown in the diagram above).
This managment access has higher privileges as the interop module and is done directly by the core module.
Nevertheless, managing the pool may emit events, which are then handled by the interop module and looped back
to the main bus.
Those pages were written in August 2023 as a Summary of the Ideas and Conecepts collected over several years. They serve as blueprint, ongoing development log and documentation for the development of the reference implementation.
Want to see it live and in colour ? Well, You actually do...
This entire Website is running on the latest implementation of the rwmax! server.
Although not everything is production ready for the intended use as Application Server
and You surely will encounter some ToDo Notes, placeHolders, inconsistencies and
glitches here and there...
The current Page content is included in the CoreIO's StdDocument and UI.
(try a long click on one of the links below or in the menuBar)
You may see (at least part of) the virtual file system at the 
[ fileViewer ].
If You like to 
[ login ] as PublicUser You may check out the
scripting abilities of the SessionObj in the 
[ console ] (Javascript ES5+)
The 
[ editor ] Component is CodeMirror 6
so, Yes!
- live and in colour !... and ready to hit the open internet ...
Although there is no download-and-install-ready package at this point, the
Server-Software rwmax! (vers. 0.5 - as of August 2024) is freely available under a
EUPL-1.2 License, compatible to many other open-source licenses (incl. GPLv3).
© Siggi Gross - June 2024 - mail@siggi-gross.de
[ Legal Notice | Impressum & Datenschutz ]
