The vfs core module provides a virtual filsystem and the interface to navigate its tree.
vfs is decoupled from the underlying unix system, builds its own root tree and stores on dedicated disks/partitions.
The module accepts request in form of a path/URI and additional arguments.
Standard operation is to find the specified resource while checking against the current users permissions.
Its standard return (without additional arguments) is the requested iNode. This iNode can later be passed to
a system command or its data streams accessed via handles.
With additional arguments, the module will return the iNode or an iNode tree in the requested representation.
This can be in string/text format, JSON format, or a HTML snippet which can be inserted into prepared documents.
The filesystem module will ensure that the user has the permission to operate on the requested resources.
The [ rwmax! - extended permission control ] follows established POSIX behaviour, and
extends on some parts to provide enhanced streaming capabilities.
The vfs module also provides an impementation and access to the database.
The database handles entries as structured elements, its data fields ar indexed by default and the
entire database is seamlessly integrated into the virtual filesystem.
For more detailed informations, see [ database ]
Status: working
- vfs.Driver & partitionTypes: raw, local, virtual - working
- vfs.Node - Pool & NodeTypes (including db) - working
- vfs mount - working
- vfs permissions - rwmax! working
- tree navigation - with permission checks - working
- stat, ls, mkFile, mkDir, fOpen, fRead, httpRead, httpProbe - with permission checks - working
- returns as string, JSON, HTML - working
- redirections (domain-, user-, group-, public-folder specific) - working
- create/deleteVirtual - working
- vfs commands - cp, mv, rmFile, rmDir - needs implementation & interface
- vfs managment - chOwn, chGrp, chMod - needs implementation & interface
- vfs versioning - prepared & partially working - needs interface
- the concept for disk storage (raw) is working,
writing to plain partitions needs further evaluation and implementation.
Status: working
- /fileViewer - working
- /console - vfs Object - working (some placeHoder functions)
The module provides an interface and is responsible for
instantiation, management & communictation from/to processes and sub-processes.
(mostly from the /bin folder)
For execution of code and scripts it provides js-based interpreters in two flavours:
Both versions come with a set of standard methods and properties for vfs, db and Stream handling, they can handle SysIds
and provide a basic Buffer implementation for binary Data handling.
The console of the SessionObj runs on an Intepreter instance.
Status: implementation of concepts
- creating Processes (node js) & stdIO/ReqObj/cmdReq protocol - concept is working
- creating Processes (C) & stdIO/ReqObj/cmdReq protocol - concept is working
- Interpreter - (inc. SysId & Buffer) & stdIO/ReqObj - working
- ScriptEngine - (incl. SysId & Buffer) - working | stdIO/ReqObj/cmdReq needs implementation
The session module is responsible for all user, group and authorization related tasks.
It provides an interface, accepting commands and arguments.
- creation & management of users - assignment to groups and domains
- login, logout, session creation & management
The sess module takes care of the sessPool, creates & manages SessObj's, session keys and Id's.
As user requests are bound to a specific SessObj, the module also manages (part of) the reqPool.
Through the interop module and bus, the sess module is responsible for all security checks
requested by the other core modules. It verifies users and sessions to be authentic and
valid.
Status: working
- mount & writeBack changes to /db/users.db, /db/groups.db, /db/hosts.db - working
- cooperation with [CoreIO], [Vfs] and [Stream] - working
- users: login, setPwd, logout - working
- groups: join, leave - working
- domains: join, leave - working
- sessions: createSess, destroySess, /console - working
- still some work on better encryption and key generation/handling (creation of client certs)
Status: working
- /login.html - working
- /console - query / WebSocket connection to SessObj - working
The stream module is responsible for direct stream access for the user.
Once a iNode from the filesystem or a running process is available to the user, their
streams can be accessed via connect(), fOpen()....
On execution, the core modules create stream / process handles, stored in their respective pools
AND bound to the users SessObj and permissions. Therefore a user can only access streams assigned
to his/her session via the provided handles.
The real work is then done by the stream module (read, write...)
(its main protocol is short and its implementation should be as lean and fast as possible)
The interface expects a valid stream handle, a symbol for the desired operation and one additional argument.
(see the system diagram above) the rest is pure data.
The module differentiates between r,s,p - accessing pools through the provided reqId,sessId,procId.
For all streams not beloning to one of these groups, a generic handleId points to the stream pool.
The stream module takes care for proper and complete streaming of data, it can determine wether the expected
data are character or binary based, and can provide some basic conversions. (from/to base64, urlEncode, ...)
For encrypted streams, the sessModule will provide the required keys.
encoding: character streams are expected to be utf-8 !!
Initialisation or alteration of the streaming mode, conversion/encryption can be managed through the modules command interface.
In case the stream handle is connected to a network, the module needs further initialisation and will take care
of the underlying network protocols (including protocol & content negotiation, mimeTypes etc).
Status: working
- assign Reader/Writer uIds/gIds - needs better implementation
- still - make it lean and fast !!!
Status: working
- /console - Stream Object - working (some placeHoder functions)
- Stream.info() - working
- Stream.read(), Stream.write()- working
- Stream.createReader(), Stream.createWriter() - working
- Sream.input, Stream.input.create(), Stream.input.subscribe() - working
- Stream.output, Stream.ouput.create() - working
- Stream.save(), Stream.saveAs() - partially working (placeHolder)
- vfs.fOpen -> StreamHandle - working (auto-subscribe/connect needed)
the event module is responsible for event triggering and handling.
its interface has 3 main modes:
depending on the type of source/destination stream, the event module will handle the following standard events:
as You may have noticed, the standard stream events follow a certain scheme with corresponding representations on the
emmit/receive sides. following those patterns the entire life cycles of a stream is represented:
- created -> init() -> ready -- start cycles -> normal execution can start...
in close cooperation with the main bus, the event module is responsible for the cycling of reqObj's, streams, processes and
provides the heartbeat for the entire system.
- read/writeData, lock/unlock, login/logout, next/busy,done -- continuous execution...
- close() -> closed / abort() -> aborted -> destroy() -> destroyed -- exit cycles...
events are not limited to the above standard events. they represent the minimum each stream will provide.
Signals, Warnings, Errors are also events, as well as user defined events.
events have an execution and propagation scope. events emmited at user level will be propagated up to the sessObj,
which represents the users top level of execution. most unhandeled events will be terminated at sessObj level,
eventually throwing an [unhandeled event exception].
others - especially those who are security relevant - will be handled at system level.
further, different types of events have priority levels assigned to them and act as interrupts.
lets start with an expample:
piping streams the easy way...
onEvt( [srcStr],"wData",[destStr],"rData" )
the above means:
- if the source stream emits an wData event, which means it has consumable data,
- emit an rData event to the destination stream to indicate to read them.
of course the streams need to be attached etc...
but this is what effectively will happen if You pipe two or more streams together with pipe() or plumb() via the cmd core module.
a similar mechanism is applied, when a stream is subscribe()'d
another example:
onEvt( [parentProcess],"closed",[childStr],"close" )
ensure child streams are closed if the parent exits.
a more generic way of registering event handlers uses:
onEvt ( [srcStr],"evt","cmd args...")
this way we can do the following:
onEvt ( [procStdOut],"closed","procStart ...")
which means:
if the stdOut of the process was closed (the process died), restart it.
as long as the event handler is registered, the process will be respawned again and again.
registered event handlers are assigned not only to the source and destination streams, but also to a user and session.
the eventPool is managed by the event and interop module, storing all registered event handlers.
if the event module receives an event, it will query the event pool and execute/propagate the found rules.
if further events are emitted, the interop module will initiate queing and propagation.
from a users perspective, event handling and registrations should be as easy as presented above.
given some basic understanding of the concept or similar scenarios like
Javascript Event bubbling, it should be easily
accessible and comprehensible.
from the implemetation perspective, a considerable amount of effort is needed.
for a simple "query the event pool and execute/propagate the found rules" a few different steps are required behind the scenes:
- permission checks on rules AND involved emmitters/receivers
- reqPool entries may have wildcards on uId/sessId ,streamId, or eventId
- some rules may represent a [catch-all except of] with multiple entries
- if multiple rules are found, their precedence needs to be sorted by priority
- emitted events (destroy) will wait for completion (destroyed)
- some events may invalidate or delay others
- some event constellations can render into infinite loops...
- existing concepts and code from BeapEvt and BeapControls as a starting point
- cycling and hook-in components from BeapWWW
- eventPool is an indexed and queriable "bubbling tree"...
- event->interop->mainBus loop strongly coupled with an internal cycle/next mechanism...
All internal Id's are 32 bits long: 8 bit header and 24 bit specifier.
Interpreted as uInt32, each Id type can have up to 16.7M entries.
| type | id | 32 bit | descr | |
|---|---|---|---|---|
| 8 bit | 24 bit | |||
| pool ids | sId | "s" | n n n | sessId |
| pId | "p" | n n n | processId | |
| hId | "h" | n n n | (stream)handleId | |
| user related ids | uId | "u" | n n n | userId |
| gId | "g" | n n n | groupId | |
| oId | "o" | n n n | domainId | |
| module ids | ":" | c c c | ||
| ":vfs" | virtual file system module | |||
| ":cmd" | command module | |||
| ":ses" | session module | |||
| ":str" | stream module | |||
| ":bus" | mainbus module | |||
| ":evt" | event module | |||
| ":IOh" | IO_Http - http(s) | |||
| ":IOn" | IO_Net - TLS | |||
The string & code representation for Pool and User related system Ids is in the form of
@u.00.00.01, with the first byte as character and the rest hex encoded.
Module system Ids by a colon and three characters. @:str
The Session & Script interpreters handle SysIds as literals.
In code, JSON or console input they are noted without quotes
and are identified by the initial @ character.
SysIds are a native datatype like bools or numbers, there is no SysId() constructor!!
(The ScriptEngine also accepts a [ 0i(32bit-hex) ] notation)
uId = @u.00.00.01; vfsId = @:vfs; resString = Stream.read(@h.00.00.01,64);
sysId.match(otherSysId) method.sysId.String and sysId.UInt32 properties, also an sysId.type property.sysId.isPublic() and sysId.isRoot() methods.
When passing Id's around, the system can determine its type by evalutating the first 8 bits,
from a developers point of view, these first 8 bits will be human readable.
24 bits (16.7M) for a specific Id type is sufficient for even a large running system.
When addressing different parts and/or ressources of the system as source or destination for
system calls, the entire 32 bits will prevent any "misinterpretation" of the passed Id.
The module Ids follow the same 32 bit schema starting with an ":" and all characters.
Althogh the list is pretty short, each core module has a unique 32 bit Id,
and therefore can be addressed unambiguously.
(see also the reqObj and mainBus protocol)
The above list is extensible for future implementations...
- "d"nnn -> deviceId
- "m"nnn -> memId (allocated memory blocks)
Status: working
- SessObj - inside interpreter - working
- SessObj - in/out of interpreter as JSON payload - working (test some edge cases)
- ScriptEngine - inside interpreter - working
- MainBus - as Req/Resp params - working
- MainBus/ModuleServer - as JSON payload - working
ReqIds are not SysIds per definition, but strongly related to them.
Each system component which creates request has its own 16-bit-counter, managing its own
requests and rIdx's in a round-robin-like manner.
Once a request is sent to the mainBus or the IO-Streams, its header has the following form:
'q0|c0' + srcId + rIdxThe combined srcId + rIdx (which is the SysId of the creating instance + its internal reqIdx) gives
16bits 32bits 16bits
@s.00.00.01|00.01. iNode Id's are 32 bits long (at this point) and follow a similar
"@" + 2characters + hex form as the SysIds. @f_.00.00.01
users will not have any interaction with the iNode Id's - they interact with streamIds...
the informations below show how Id's are currently assigned by the vfs module...
The headers are used in a similar way as for the system Id's, to
indentify/specify different parts of the filesystem.
This way, the unique id will give some information about its usage,
or where the iNodes are located or mounted to/from.
Status: working
- the vfs can address 2^24 (16.7M) iNodes per header group
- a future extension to 64 bits is planned (see below & iNode header )
- although not SysIds per definition, Interpreter and ScriptEngine handle NodeIds as such
- iNodes of the @f_ variant have a physical storage location.
- iNodes of the @fi variant may have a physical storage location (most of them won't).
- the vfs can address 2^56 (72.0P) iNodes per header
- the 56 bits may be split in meaningful blocks, depending on the storage type or location
- a variant of @f_.char.00.00.00... or @f_.char.char.00.00... may be possible
- - @f_.h.01.00.00... -> HD 01
- - @f_.u.01.00.00... -> USB 01
- - @f_.n.h.00.00... -> network (http)
- - @f_.n.f.00.00... -> nework (ftp)
- - ...
to match the corresponding system Id's...
- "@fv" + "s"nnn + 24bit -> sessId -> sessPool -> /sess
- "@fv" + "p"nnn + 24bit -> processId -> procPool -> /proc
- "@fv" + "h"nnn + 24bit -> handleId -> streamPool -> /tmp
- "@fv" + "u"nnn + 24bit -> userId -> /users/... (well known pathes/links)
- "@fv" + "g"nnn + 24bit -> groupId -> /groups/... (well known pathes/links)
- "@fv" + "d"nnn + 24bit -> deviceId -> /dev
- "@fv" + "m"nnn + 24bit -> memId -> /mem
- ...
- iNodes from this variant are the pure virtual parts of the vfs and have no physical storage location
- they hold the stream endpoints and buffers for their corresponding system Id counterpart
- the 24 bit may be unused/nulled, or have a distinct numbering to determine - for example:
- - the read/write buffers for a duplex stream
- - the indiviual endpoints of a multiplexed/subscribed stream
- - the stdIO endpoints of a process/session - 0x00 stdIn - 0x01 stdOut - 0x02 stdErr ...
- a single system Id can hold up to 16.7M iNodes/connection endpoints.
- not all system Id's have an iNode/vfs representation.
- most of these iNode Id's will have a very restrictive permission control or may only be available to the system itself...
- iNodes from this variant are virtual,
- they have a indirect physical storage location in form of the db's transaction files.
The iNode header is 1024 bits long, 64bit-aligned (and fractions of 64),
the first 512 bits are fixed, the second 512 bits are dependant on the file type...
the iNodeId is 64bits long,
permissions, user, group and creator ids are 32bits long,
dateTime in 64bit format.
| 64bits | 64bits | 64bits | 64bits |... | | 4x8bits | 32bits | 32bits | 32bits | 32bits | 32bits |... | iNodeId | rwmax! | userId | groupId | creatorId | dateTime |... | 64bits | 64bits | 64bits | 64bits | | 16bits | 48bit | 16bits | 2x8bits | 32bits | prev vers | next vers | | #of elems | size | #of links | vs&lock | locking | iNodeId | iNodeId |
an iNode can hold up to 2^16 (65k) members,
fileSizes up to 2^48 bytes (281.4TB)...
(in case of databases, the #of elems represents the subMembers of the class or treeMember,
the size denotes the numer of databaseIds pointing to the treeMember...)
an iNode can be linked up to 2^16 (65k) locations in the vfs,
2x8bits are reserved for marking the versioning (as current etc...) and locking (locking mode),
a 32bit field for the SysId of the locking instance,
and 2x64bits for iNodeId's for the previous and next version
the content of the second header block is highly dependant on the type of the file/iNode.
it mainly contains 16bit blocks of pointers to the iNodes data sections...
(in case of folders to the list of [iNodeId : iNodeName] of the child elements.)
...
the iNode doesn't know its own name, as it may be attached to multiple folders with
potentially different names. the names are stored in the "parent" folder iNode.
© Siggi Gross - June 2024 - mail@siggi-gross.de
[ Legal Notice | Impressum & Datenschutz ]
