[ /reqObj.html ]︙

table of contents

reqObj and mainBus protocol

page contents

Message Types

Internally, the system has 4 different message Types :

All of them follow a Protocol with a common structure :

All of the following is handled internally !
From a client perspective, only the Destination, MsgReqData and the Payload are writable.

The lowest a client can acces ReqObjs (if permitted) is :
- via the MainBusInterface through MainBus.connect(...) or MainBus.stream(...)
- via the ChildProcInterface through ChildProc.sendCmd(...)

The functions of the stdLib (vfs, db, Stream interfaces) will create the Requests and their arguments
by calling their methods and evaluating the specific function arguments.

The responses are JS/JSON objects as follows:
{ status:true, result:<jsVal>, [mimeType:<String>] } (With the result according to the responseFormat)
{ status:false, descr:<String> } (With the ERR-Code as start of the descr)

MainBus Requests & Responses (q0,r0)
Requests bubble UP the process tree from the current process to its ParentProcess
through its stdOut IO Stream - until they reach their top-most Parent.
(a SessionProcess in most cases) - and are then passed to the MainBus.
Responses travel the same way in opposite directions. (also through IO Streams)

Stream Connections (s0) take the same route to initialize themselves.
Their Response is a socket instance. Once established, they communicate through this
bi-directional keep-alive connection between the source and destination endpoints.

ChildProcess Commands (c0) communicate to a direct ChildProcess of the current one.
They travel DOWN the proccess tree by exactly ONE step through IO Streams.

implementation :

Status: working
MainBusInterface & ModulServer - working
MainBus inside Interpreter - working (additional reqFormat 'v'->'j')
MainBus.stream() & Stream interface inside Interpreter - needs debugging

Status: TODO
ChildCmd inside Interpreter - proof-of-concept is working, needs implementation
ChildCmd - Process (nodejs) - stdIO/ReqObj/cmdReq proof-of-concept is working
MainBus & ChildCmd - Process (C-impl) - stdIO/ReqObj/cmdReq proof-of-concept is working
MainBus & ChildCmd - ScriptEngine - stdIO/ReqObj/cmdReq needs implementation

msgHeader

Each system component which creates request has a mainBus- or childProc-interface.
This interface has its own 16-bit-counter, managing and queueing requests by index.

The msgHeader is created by the interface of the current module, session or process.
It consists of a fixed 64-bit block, followed by Tracing and Destination Data block of variable length.
Commands to ChildPocesses have no Tarcing and Destination Data. (See there)

Once a request is created and sent to the mainBus or the IO-Streams, it has the following form:

'q0|c0' + srcId + rIdx |      Tracing Data        |    uId   +  destId ||
16bits   32bits 16bits |    (hLen * 32bits)       |   32bits    32bits ||
"q","r","s","c"
messageType - <char>
hLen
header length - <uint8> (in SysId/32bit-blocks)
srcId
request creator - <SysId> (uInt32)
rIdx
request index - <uInt16>

Depending on type the first character is set accordingly and the header length is initialized with 0.
(Sometimes request will be referenced as "q0" or "c0" ...)

The srcId is set according to the current module, session or process.

The internal request counter assigns rIdx's in a round-robin-like manner.
It revokes rIdx only on completion - once freed, the same rIdx will be reused only after
a full (16-bit) round again. Therefore it issues uniqe indices at any given time.

The msgHeader cannot be manipulated through user/client interaction.

reqId

By combining srcId + rIdx, it gives the entire ReqId
- its String representation in the form of @s.00.00.01|00.01.
The combined reqId is unique at any given time.
ReqIds are not SysIds per definition, but strongly related to them. (See also SysId's )

Tracing Data

applies to Requests and Responses to/from MainBus - see Request Cycles

Destination Data

Requests to MainBus contain the current userId and a destination as SysIds.
In some cases userId & destId may be re-evaluated or altered while a Request is
bubbling up the process tree. (Process delegates...)

Responses from MainBus have no Destination Data.

ChildProc commands

"c0" requests are commands to a direct childProcess of the current module, session or process.
As they are travelling exactly ONE step DOWN the process tree, they need no Destination or Tracing Data in their MsgHeader.

Request Cycles

Bubble-Up-Phase

Requests to MainBus bubble up the process tree until they reach their SessionObj.

Initially the TracingData block of the Request MsgHeader is empty and hLen at 0 (q0).
While bubbling up, at each step the hLen Counter will be increased by one,
and the parent processes SysId will be added to the TracingData block.
Once a Request reached its SessionObj, the MsgHeader will/may be longer than in its original state.

In case the requests destination is one of the ParentProcesses, it may be intercepted early and proccessed directly before hitting the Session or MainBus.

The accumulated TracingData will reflect the path travelled through the process tree. This way the TracingData also build a "Chain-Of-Trust" towards the MainBus with its SessId as its last entry.
Note : Requests from Modules and top-level Processes (SessProc) have no accumulated TracingData,
as their srcId is allready trusted and they have direct MainBus access.

e.g : (a user process requesting the virtual files system) 

'q0'  +  srcId + rIdx  |      Tracing Data        |    uId   +  destId ||
16bits   32bits 16bits |    (hLen * 32bits)       |   32bits    32bits ||

 q0     @p.00.00.04|01 |                          | @u.00.00.01  @:vfs ||
 q1     @p.00.00.04|01 | @p.00.00.02              | @u.00.00.01  @:vfs ||
 q2     @p.00.00.04|01 | @p.00.00.02  @s.00.00.01 | @u.00.00.01  @:vfs ||

The SessionProcess then stores this information, does some pre-flight permission checks,
adds/verifies the userId and passes the entire Request to the MainBus.

MainBus-Execution-Phase

The MainBus will then connect to the corresponding CoreModule (& its Pools) to determine the final destination of the Request and trigger its execution.

If Streaming is involved in the Requests (q0 with queryBuf || s0), a socket between the endpoints is established and the rest of the communication sends/receives only MsgReqData & MsgResData blocks until the Request is done and the socket is closed.

Bubble-Down-Phase

Otherwise, when the SessProcess received a result (or an Error) from MainBus,
the Request MsgHeader is converted to a Response MsgHeader and the same
procedure is executed in the opposite direction : bubble down towards (r0)

 q2  @p.00.00.04|01 | @p.00.00.02 @s.00.00.01 | @u.00.00.01  @:vfs ||
--> pass to MainBus ...
<-- on response from MainBus
 r2  @p.00.00.04|01 | @p.00.00.04 @p.00.00.02 || --> pass to @p.00.00.02  
 r1  @p.00.00.04|01 | @p.00.00.04             || --> pass @p.00.00.04
 r0  @p.00.00.04|01 |                         || --> done !!

On reaching the Requests source, the response data are processed there
(identified by its rIdx) and passed to the callback provided on creation.

msgReqData

The msgReqData is created by the interface of the current module, session or process.
It consists of a fixed 64-bit block, followed by the payLoad Data of variable length.
The payLoad Data consist of the queryStr, which is sent directly with the request,
and the queryBuf, wich is streamed when a connection between the endpoints is established.

Once a request is created and sent to the mainBus or the IO-Streams, it has the following form:

 arg1 + arg2 + rFmt + qFmt  | queryLen ||
 char   char   char   char  |  uInt32  ||
arg1
cmd Type - <char>
arg2
cmd Subtype - <char>
rFmt
requested response Format - <char>
qFmt
query Format - <char> (optional)
queryLen
request length - <uInt32>
reqArguments

See below

reqFormats

The response Format indicates in wich format the response should be sent. (if applicable)
If queryFormat is set, it declares the format of the queryStr. (default 'c')

queryLen

The queryStr is limited by the transportBufSize. (global default 4096 / 0x1000 bytes)
Its byteLength is limited to transportBufSize -1 !

The queryBuf is calculated in full Blocks of transportBufSize (the last may not be fully used)
The maximum StreamData for queryBuf is limited to 2^32 (4GB) - transportBufSize.

When a Request has a queryStr AND queryBuf, the queryLen is calculated as follows:

  [number of StreamData blocks, shifted by transportBufSize]
+ [queryStr byteLength]
------------------
= queryLen <uInt32>
When qFmt is "i" ("u","s","f") bLen for the queryStr is not a byteLength !
In those cases it is a unitLength - 32bit (64bit) !
e.g: format:"i", bLen:2 -> expecting 2 SysId's = 64bit/8bytes
implementation :

Why <chars> ?
The whole MsgReqData header fits into one register.
There is no need for opaque enums to pack it denser etc...

By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the request will be processed.

Also - its Arguments internally handled as int8_t and separated as CmdType & SubType :
the decision tree in the code can remain modular and compact, allowing fall-through cases...

The msgReqData cannot be manipulated through user/client interaction once sent. queryStrLength = queryLen % transportBufSize;
numberOfBlocks = int( queryLen / transportBufSize);

Request Arguments

Interop : [ ! ]

Get/Set Status : [ ? ! ]

Initial Request Transmission : [ < > ]

Request Events : [ ~ ]

Request/Response Streaming

Once a Stream between the source and destination of a Request is established,
streaming of data is done in blocks of (max) transportBufSize chunks over a socket.

While streaming, there is no distinction between msgReqData or msgResData headers,
the difference is rather: who is the sending / receiving end.

   args  | queryLen ||
 char[4] |  uInt32  ||

Note : no fmt in the header, the chars may be unused/nulled.

Messages sent over the socket have no msgHeader, they consist only of a series
of headers + thier payload Data followed by a confirmation
header from the counterpart to continue or end the transmission.

Switching to streaming mode may be triggered when:
- sending a Request with queryBuf Data [ '~w' '~n' ]
- sending a responseStr wich is longer than transportBufSize [ '~~' '~r' '~n' ]
- sending a Response wich has a responseStr & a resultBuf [ '~m' '~r' '~n' ]
- subscribing for active reading mode (keep-alive connection) [ '<a' ]
- establishing a keep-alive connection to a dynamic stream [ '<~' '>~' ]

While streaming Data : [ ~ ]

implementation :

Why <chars> ?
Again, the whole header block fits into one register.
By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the response will be processed
or what went wrong.

msgResData

The msgResData is created :
- by the destination, when sending a response (or part of it)
- by the interface of the requesting module, session or process when a Request is done.

The msgResData consists of a fixed 64-bit block, followed by the payLoad Data of variable length.

The payLoad Data consist of the responseStr, which is sent directly with the final response,
or a resultBuf wich is streamed when the connection between the endpoints is established.

The Request is done when all Data are streamed to the resultBuf.
In case streaming ended early or with an exception, the receiving side may raise an errorCode.

When done, the msgResData has the following form:

 status  + rFmt  | resLen ||
 char[3]   char  | uInt32 ||
status
response Status - <char[3]>
rFmt
response Format - <char>
resLen
response Length - <uInt32>
The msgResData cannot be manipulated through user/client interaction.
resFormats

The response Format indicates in wich format the response was sent.
It may be different from the MsgReqData.rFmt (if it was not applicable)

When rFmt is "i" ("u","s","f") bLen for the responseStr is not a byteLength !
In those cases it is a unitLength - 32bit (64bit) !
resStatus
"OK "
success (rFmt is for responseStr)
"OK+"
success + has resultBuf data (rFmt is for resultBuf)
"ERR"
failure (generic)
"E**"
failure (specific)
global.ErrCodes = [
	/* ressources */
	"ENE", // not existing
	"ENP", // no permission
	"ENR", // not ready
	"ENA", // not available/locked
	
	/* MainBus requests */
	"ERS", // request syntax
	"ERM", // request missing (uId, sessId/moduleId, chain-of-trust)
	"ERV", // request not valid (destId, arg1, arg2, format)
	"ERP", // request permission (destId, arg1, arg2)
	"ERE", // request execution/evaluation error
	
	/* arguments - queryStr */
	"EAS", // argument syntax
	"EAM", // argument missing
	"EAV", // argument not valid
	"EAP", // argument permission
	"EAE", // argument evaluation error
	
	/* functions - queryStr */
	"EFS", // function syntax
	"EFM", // function missing
	"EFV", // function not valid
	"EFP", // function permission
	"EFE", // function execution error
]

Error responses ALWAYS have implicit format 'c' and
the responseStr contains their Error descriptions.

Client code can return global.ErrCodes, the E** list is also user/client extensible.
(as long it starts with "E" and does not abuse the global.ErrCodes meanings)
By Convention they may be marked as EC* (e.g. ECP for Client Permission Error)

When rFmt is 'm' msgResData.mimeType & resObj.mimeType will also be set.

implementation :

Why <chars> ?
Again, the whole MsgResData header fits into one register.
By inspecting ist first four bytes, which are human readable
quick decisions can be made, how the response will be processed
or what went wrong.

e.g: CoreIO_Http makes assumptions by the ErrCode how to respond to http(s) requests.
ENE -> 404 - Not Found
ENP -> 401 - Unauthorized
EFE -> 409 - Conflict

© Siggi Gross - July 2024 - mail@siggi-gross.de
[ Legal Notice | Impressum & Datenschutz ]

Loading Application ...