Internally, the system has 4 different message Types :
All of them follow a Protocol with a common structure :
All of the following is handled internally !
From a client perspective, only the Destination, MsgReqData and the Payload are writable.
The lowest a client can acces ReqObjs (if permitted) is :
- via the MainBusInterface through MainBus.connect(...)
or MainBus.stream(...)
- via the ChildProcInterface through ChildProc.sendCmd(...)
The functions of the stdLib (vfs, db, Stream interfaces) will create the Requests and their arguments
by calling their methods and evaluating the specific function arguments.
The responses are JS/JSON objects as follows:
{ status:true, result:<jsVal>, [mimeType:<String>] }
(With the result according to the responseFormat)
{ status:false, descr:<String> }
(With the ERR-Code as start of the descr)
MainBus Requests & Responses (q0,r0)
Requests bubble UP the process tree from the current process to its ParentProcess
through its stdOut IO Stream - until they reach their top-most Parent.
(a SessionProcess in most cases) - and are then passed to the MainBus.
Responses travel the same way in opposite directions. (also through IO Streams)
Stream Connections (s0) take the same route to initialize themselves.
Their Response is a socket instance. Once established, they communicate through this
bi-directional keep-alive connection between the source and destination endpoints.
ChildProcess Commands (c0) communicate to a direct ChildProcess of the current one.
They travel DOWN the proccess tree by exactly ONE step through IO Streams.
Status: working
MainBusInterface & ModulServer - working
MainBus inside Interpreter - working (additional reqFormat 'v'->'j')
MainBus.stream() & Stream interface inside Interpreter - needs debugging
Status: TODO
ChildCmd inside Interpreter - proof-of-concept is working, needs implementation
ChildCmd - Process (nodejs) - stdIO/ReqObj/cmdReq proof-of-concept is working
MainBus & ChildCmd - Process (C-impl) - stdIO/ReqObj/cmdReq proof-of-concept is working
MainBus & ChildCmd - ScriptEngine - stdIO/ReqObj/cmdReq needs implementation
Each system component which creates request has a mainBus- or childProc-interface.
This interface has its own 16-bit-counter, managing and queueing requests by index.
The msgHeader is created by the interface of the current module, session or process.
It consists of a fixed 64-bit block, followed by Tracing and Destination Data block of variable length.
Commands to ChildPocesses have no Tarcing and Destination Data. (See there)
Once a request is created and sent to the mainBus or the IO-Streams, it has the following form:
'q0|c0' + srcId + rIdx | Tracing Data | uId + destId || 16bits 32bits 16bits | (hLen * 32bits) | 32bits 32bits ||
Depending on type the first character is set accordingly and the header length is initialized with 0.
(Sometimes request will be referenced as "q0" or "c0" ...)
The srcId
is set according to the current module, session or process.
The internal request counter assigns rIdx
's in a round-robin-like manner.
It revokes rIdx only on completion - once freed, the same rIdx will be reused only after
a full (16-bit) round again. Therefore it issues uniqe indices at any given time.
The msgHeader cannot be manipulated through user/client interaction.
By combining srcId + rIdx, it gives the entire ReqId
- its String representation in the form of @s.00.00.01|00.01
.
The combined reqId is unique at any given time.
ReqIds are not SysIds per definition, but strongly related to them. (See also SysId's )
applies to Requests and Responses to/from MainBus - see Request Cycles
Requests to MainBus contain the current userId and a destination as SysIds.
In some cases userId & destId may be re-evaluated or altered while a Request is
bubbling up the process tree. (Process delegates...)
Responses from MainBus have no Destination Data.
"c0" requests are commands to a direct childProcess of the current module, session or process.
As they are travelling exactly ONE step DOWN the process tree, they need no Destination or Tracing Data in their MsgHeader.
Requests to MainBus bubble up the process tree until they reach their SessionObj.
Initially the TracingData block of the Request MsgHeader is empty and hLen at 0 (q0).
While bubbling up, at each step the hLen Counter will be increased by one,
and the parent processes SysId will be added to the TracingData block.
Once a Request reached its SessionObj, the MsgHeader will/may be longer than in its original state.
In case the requests destination is one of the ParentProcesses, it may be intercepted early
and proccessed directly before hitting the Session or MainBus.
The accumulated TracingData will reflect the path travelled through the process tree. This way
the TracingData also build a "Chain-Of-Trust" towards the MainBus with its SessId as its last entry.
Note : Requests from Modules and top-level Processes (SessProc) have no accumulated TracingData,
as their srcId is allready trusted and they have direct MainBus access.
e.g : (a user process requesting the virtual files system)
'q0' + srcId + rIdx | Tracing Data | uId + destId || 16bits 32bits 16bits | (hLen * 32bits) | 32bits 32bits || q0 @p.00.00.04|01 | | @u.00.00.01 @:vfs || q1 @p.00.00.04|01 | @p.00.00.02 | @u.00.00.01 @:vfs || q2 @p.00.00.04|01 | @p.00.00.02 @s.00.00.01 | @u.00.00.01 @:vfs ||
The SessionProcess then stores this information, does some pre-flight permission checks,
adds/verifies the userId and passes the entire Request to the MainBus.
The MainBus will then connect to the corresponding CoreModule (& its Pools) to determine
the final destination of the Request and trigger its execution.
If Streaming is involved in the Requests (q0 with queryBuf || s0), a socket
between the endpoints is established and the rest of the communication sends/receives only
MsgReqData & MsgResData blocks until the Request is done and the socket is closed.
Otherwise, when the SessProcess received a result (or an Error) from MainBus,
the Request MsgHeader is converted to a Response MsgHeader and the same
procedure is executed in the opposite direction : bubble down towards (r0)
q2 @p.00.00.04|01 | @p.00.00.02 @s.00.00.01 | @u.00.00.01 @:vfs || --> pass to MainBus ... <-- on response from MainBus r2 @p.00.00.04|01 | @p.00.00.04 @p.00.00.02 || --> pass to @p.00.00.02 r1 @p.00.00.04|01 | @p.00.00.04 || --> pass @p.00.00.04 r0 @p.00.00.04|01 | || --> done !!
On reaching the Requests source, the response data are processed there
(identified by its rIdx
) and passed to the callback provided on creation.
The msgReqData is created by the interface of the current module, session or process.
It consists of a fixed 64-bit block, followed by the payLoad Data of variable length.
The payLoad Data consist of the queryStr, which is sent directly with the request,
and the queryBuf, wich is streamed when a connection between the endpoints is established.
Once a request is created and sent to the mainBus or the IO-Streams, it has the following form:
arg1 + arg2 + rFmt + qFmt | queryLen || char char char char | uInt32 ||
See below
'c'
char/string 'j'
js/json 'x'
xml/html 'i'
SysId 'b'
binary 'm'
mime/base64 'u'
,'s'
,'f'
[not implemented yet!] unsigned/signed/float (64bit wide) The response Format indicates in wich format the response should be sent. (if applicable)
If queryFormat is set, it declares the format of the queryStr. (default 'c')
The queryStr is limited by the transportBufSize. (global default 4096 / 0x1000 bytes)
Its byteLength is limited to transportBufSize -1 !
The queryBuf is calculated in full Blocks of transportBufSize (the last may not be fully used)
The maximum StreamData for queryBuf is limited to 2^32 (4GB) - transportBufSize.
When a Request has a queryStr AND queryBuf, the queryLen is calculated as follows:
[number of StreamData blocks, shifted by transportBufSize] + [queryStr byteLength] ------------------ = queryLen <uInt32>
When qFmt is "i" ("u","s","f") bLen for the queryStr is not a byteLength !
In those cases it is a unitLength - 32bit (64bit) !
e.g: format:"i", bLen:2 -> expecting 2 SysId's = 64bit/8bytes
Why <chars> ?
The whole MsgReqData header fits into one register.
There is no need for opaque enums to pack it denser etc...
By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the request will be processed.
Also - its Arguments internally handled as int8_t and separated as CmdType & SubType :
the decision tree in the code can remain modular and compact, allowing fall-through cases...
The msgReqData cannot be manipulated through user/client interaction once sent.
queryStrLength = queryLen % transportBufSize;
numberOfBlocks = int( queryLen / transportBufSize);
"!#"
- init()"!q"
- terminate()
"!+"
- add pool or vfs entry (on creation)"!-"
- remove pool or vfs entry (on termination)
"!!"
- exit()
"? "
- get Status information ( "created", "running", "ready" ...)"?#"
- get own SysId (procId, sessId, handleId)"?!"
- get serialized JSON representation"?*"
- get StatusField information"!*"
- set StatusField information*
will determine the StatusField to return
c
- creatorIds
- sessIdu
- userIdg
- groupId>
- writerId (streams & sess only)<
- readerId (streams & sess only)a
- accepted formats (streams only)"??"
- get all named StatusFields (as string, comma separated)"?="
+ StatusFieldName
in queryStr - get StatusField information for named Field"!="
+ StatusFieldName=...
in queryStr - set StatusField information for named Field"< "
- getRequest - get named user data ">?"
- get all registered function names (as string, comma separated)"> "
- cmdRequest - call a registered function \n
delimited">+"
- cmdRequest with Stream Data - call a registered function \n
delimited "< "
- read queryLen Data from Stream"> "
- write queryLen Data to Stream"<?"
">?"
- get read/write position (not on dynamic Streams)"<!"
">!"
- set read/write position (not on dynamic Streams)"<p"
- subscribe for passive reading mode (push connection)"<a"
- subscribe for active reading mode (keep-alive connection)"<~"
">~"
- establish active read/write connection on dynamic Streams'~a'
- abort a req/cmd which was allready sent
'~a'
Response will be returned'~~'
- switch to streaming mode
'~m'
- set mimeType & switch to streaming mode
'~!'
- unrecoverable exception while streaming
'~a'
Response will be returnedOnce a Stream between the source and destination of a Request is established,
streaming of data is done in blocks of (max) transportBufSize chunks over a socket.
While streaming, there is no distinction between msgReqData or msgResData headers,
the difference is rather: who is the sending / receiving end.
args | queryLen || char[4] | uInt32 ||
Note : no fmt in the header, the chars may be unused/nulled.
Messages sent over the socket have no msgHeader, they consist only of a series
of headers + thier payload Data followed by a confirmation
header from the counterpart to continue or end the transmission.
Switching to streaming mode may be triggered when:
- sending a Request with queryBuf Data [ '~w' '~n' ]
- sending a responseStr wich is longer than transportBufSize [ '~~' '~r' '~n' ]
- sending a Response wich has a responseStr & a resultBuf [ '~m' '~r' '~n' ]
- subscribing for active reading mode (keep-alive connection) [ '<a' ]
- establishing a keep-alive connection to a dynamic stream [ '<~' '>~' ]
'~r'
, '~w'
, '~n'
- streaming in progress
'~d'
- streaming done
'~d!'
- stream ended early
'~!'
- unrecoverable exception while streaming
'~a'
Response may be returned)'~q'
- close stream
'~q'
or '~d'
Response may be returned)Why <chars> ?
Again, the whole header block fits into one register.
By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the response will be processed
or what went wrong.
The msgResData is created :
- by the destination, when sending a response (or part of it)
- by the interface of the requesting module, session or process when a Request is done.
The msgResData consists of a fixed 64-bit block, followed by the payLoad Data of variable length.
The payLoad Data consist of the responseStr, which is sent directly with the final response,
or a resultBuf wich is streamed when the connection between the endpoints is established.
The Request is done when all Data are streamed to the resultBuf.
In case streaming ended early or with an exception, the receiving side may raise an errorCode.
When done, the msgResData has the following form:
status + rFmt | resLen || char[3] char | uInt32 ||
The msgResData cannot be manipulated through user/client interaction.
'c'
char/string 'j'
js/json 'x'
xml/html 'i'
SysId 'b'
binary 'm'
mime/base64 'u'
,'s'
,'f'
[not implemented yet!] unsigned/signed/float (64bit wide) The response Format indicates in wich format the response was sent.
It may be different from the MsgReqData.rFmt (if it was not applicable)
When rFmt is "i" ("u","s","f") bLen for the responseStr is not a byteLength !
In those cases it is a unitLength - 32bit (64bit) !
global.ErrCodes = [ /* ressources */ "ENE", // not existing "ENP", // no permission "ENR", // not ready "ENA", // not available/locked /* MainBus requests */ "ERS", // request syntax "ERM", // request missing (uId, sessId/moduleId, chain-of-trust) "ERV", // request not valid (destId, arg1, arg2, format) "ERP", // request permission (destId, arg1, arg2) "ERE", // request execution/evaluation error /* arguments - queryStr */ "EAS", // argument syntax "EAM", // argument missing "EAV", // argument not valid "EAP", // argument permission "EAE", // argument evaluation error /* functions - queryStr */ "EFS", // function syntax "EFM", // function missing "EFV", // function not valid "EFP", // function permission "EFE", // function execution error ]
Error responses ALWAYS have implicit format 'c' and
the responseStr contains their Error descriptions.
Client code can return global.ErrCodes, the E**
list is also user/client extensible.
(as long it starts with "E" and does not abuse the global.ErrCodes meanings)
By Convention they may be marked as EC*
(e.g. ECP for Client Permission Error)
When rFmt is 'm'
msgResData.mimeType & resObj.mimeType will also be set.
Why <chars> ?
Again, the whole MsgResData header fits into one register.
By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the response will be processed
or what went wrong.
e.g: CoreIO_Http makes assumptions by the ErrCode how to respond to http(s) requests.
ENE -> 404 - Not Found
ENP -> 401 - Unauthorized
EFE -> 409 - Conflict
© Siggi Gross - July 2024 - mail@siggi-gross.de
[ Legal Notice | Impressum & Datenschutz ]