README
¶
(C) 2020 Geotab Inc
All files and artifacts in this repository are licensed under the provisions of the license provided by the LICENSE file in this repository.
Reference compression design
The design is context aware, i. e. it knows that the payload is JSON, and the set of key-value pairs that can be expected.
As the payload sizes are typically less than 100 characters, this is probably necessary to achieve high compression ratios.
Tests with standard compression algorithms like GZIP, LZW, and DEFLATE showed comparably low compression ratios.
The overall compression consists of the following compression features:
- Paths are replaced by a two byte index from the list of paths found in vsspathlist.json.
- Fixed keywords, such as "action", "path", "get", etc., are replaced by a one byte value.
- Timestamps are replaced by a four byte value.
- The value part for the keys "value", "requestId", "subscriptionId" are replaced by a binary value, if possible.
- JSON reserved characters are removed in the case their excistence in a certain position is a MUST according to the JSON format rules.
- Payload data that does not fall under any of the features above is left uncompressed.
Below follows further details about the compression features.
-
The list vsspathist.json is generated by the VISSv2 server at every startup. The input to the server comes from the "vss_vissv2.binary" file, which is in its turn generated by the VSS Tools, that take the VSS tree as represented by the vspec files from the VSS repo as input. This means that if the VISSv2 server starts up with an updated VSS tree, then the clients should be aware of this, and get a copy of the updated vsspathlist.json file from the server. This could be handled by a request to the server asking for this file, but the VISSv2 spec does not currenly support that.
The pathlist in the file generated by the VISSv2 server is ordered, so searches in the list can be optimised.
The list index is represented by two bytes, which means that a maximum of 65536 paths can be handled. The list only contains paths for leaf nodes, so this is the maximum number of leaf nodes that the VSS tree can have for this compression feature to work. -
The value replacing the keywords are the index from the kewordlist plus 128. The reason to add 128 is that an uncompressed UTF8 JSON string only contains byte values less than 128, see an ASCII table. The keywordlist original is found in the "common.go" source file. It is not expected to change. A client compression impl needs a copy of it.
-
The uncompressed timestamps follow the ISO8601 standard and has the format YYYY-MM-DDTHH:MM:SS.ssssssZ
This is compressed as follows:
- The least significant year digit is represented by four bits.
- The month is represented by four bits.
- The day is represented by five bits.
- The hour is represented by five bits.
- The minute is represented by six bits.
- The second is represented by six bits.
Subsecond usage is not supported currently.
The sum of bits above is 30 bits, which fits into 4 bytes as shown below.
Byte 1 | Byte 2 | Byte 3 | Byte 4
00yyyymm | mmdddddh | hhhhmmmm | mmssssss
The timestamp encoding has the limitation that when the year goes from 2029 to 2030, any messages that are received by a client after the new year has begun, that contain a timestamp from the previous year, will be incorrectly decoded as they will be assigned the year 2039 instead of 2029. A solution to this could be to use Unix time instead, but then the 2038 problem will be coming. Another would be for clients to check for this 10-year problem around the time when it becomes likely. Vehicles “should not” respond with too stale data, so a few months after 2030 begins should be max what is needed.
- The value types that are supported are integer, float32, and boolean. The first byte is and index from the keywordlist, which also contains this (+128).
The type indicated also sets the number of bytes used for the representation, as shown below.
The type "nuintx" means it is a negative uint value, as all integer values are represented as uints.
Type | #bytes
nuint8 | 1
uint8 | 1
nuint16 | 2
uint16v | 2
nuint24 | 3
uint24 | 3
nuint32 | 4
uint32 | 4
bool | 1
float32 | 4
The absolute value of integers are compared to the max value of the different types, and the smallest possible size is selected. Values that are not of any of these types are kept uncompressed.
- The following JSON reserved character usages can be removed as at decompression the JSON rules will infer their reinstatement.
- The leading and trailing curly brackets.
- The colon between key and value.
- The comma between JSON data entities.
- The fact that data that does not relate to any of the mentioned compression features is left uncompressed leads to that the usage of this compression does not apply any restriction to the syntax scope of VISSv2. It also enables the client to send uncompressed requests, where the response will be compressed.
The current compression solution does not support service discovery requests.
Documentation
¶
Index ¶
- Constants
- Variables
- func AddKeyValue(message string, key string, value string) string
- func AnalyzeValueType(value string) int
- func BackendWSdataSession(conn *websocket.Conn, backendChannel chan string)
- func CloseLogFile()
- func CompressMessage(message []byte) []byte
- func DecompressMessage(message []byte) []byte
- func ExtractFromToken(token string, claim string) string
- func ExtractPayload(request string, rMap *map[string]interface{})
- func FileExists(filename string) bool
- func FinalizeMessage(responseMap map[string]interface{}) string
- func FrontendWSdataSession(conn *websocket.Conn, clientChannel chan string, backendChannel chan string)
- func GenerateHmac(input string, key string) string
- func GetModelIP(ipModel int) string
- func GetRfcTime() string
- func GetServerIP() string
- func InitDataSession(muxServer *http.ServeMux, regData RegData) (dataConn *websocket.Conn)
- func InitLog(filename string, logdir string)
- func NextQuoteMark(message []byte, offset int) int
- func PathToUrl(path string) string
- func RegisterAsTransportMgr(regData *RegData, protocol string)
- func RemoveInternalData(response string) (string, int)
- func SetErrorResponse(reqMap map[string]interface{}, errRespMap map[string]interface{}, ...)
- func TrimLogFile(logFile *os.File)
- func UnpackFilter(filter interface{}, fList *[]FilterObject)
- func UrlToPath(url string) string
- func VerifyTokenSignature(token string, key string) bool
- type ClientHandler
- type ClientServer
- type CodeList
- type FilterObject
- type HttpChannel
- type HttpServer
- type HttpWSsession
- type PathList
- type RegData
- type TransportHubFrontendWSSession
- type WsChannel
- type WsServer
- type WsWSsession
Constants ¶
const CODELISTINDEXPATH = 4 // must be set to the list index of the "path" element
const CODELISTINDEXREQID = 1 // must be set to the list index of the "requestId" element
const CODELISTINDEXSUBID = 5 // must be set to the list index of the "subscriptionId" element
const CODELISTINDEXTS = 3 // must be set to the list index of the "ts" element
const CODELISTINDEXVALUE = 2 // must be set to the list index of the "value" element
const CODELISTKEYS = 10 // must be set to the number of keys in the list
const CODELISTKEYVALUES = 15 // must be set to the number of keys plus values in the list (excl value types)
const IpEnvVarName = "GEN2MODULEIP"
const IpModel = 0 // IpModel = [0,1,2] = [localhost,extIP,envVarIP]
Variables ¶
var ( // Trace *log.Logger Info *logrus.Logger Warning *logrus.Logger Error *logrus.Logger )
var AppClientChan = []chan string{ make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), make(chan string), }
the number of channel array elements sets the limit for max number of parallel WS app clients
var HostIP string
var Logfile *os.File
const LOG_FILE = "servercore-log.txt"
var MuxServer = []*http.ServeMux{ http.NewServeMux(), http.NewServeMux(), http.NewServeMux(), }
var TransportErrorMessage string
var Upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
}
Functions ¶
func AnalyzeValueType ¶
func BackendWSdataSession ¶
func CloseLogFile ¶
func CloseLogFile()
func CompressMessage ¶
func DecompressMessage ¶
func ExtractFromToken ¶
func ExtractPayload ¶
func FileExists ¶
func FinalizeMessage ¶
func FrontendWSdataSession ¶
func GenerateHmac ¶
func GetModelIP ¶
func GetRfcTime ¶
func GetRfcTime() string
func GetServerIP ¶
func GetServerIP() string
func InitDataSession ¶
func NextQuoteMark ¶
func RegisterAsTransportMgr ¶
* * registerAsTransportMgr: * Registers with servercore as WebSocket protocol manager, and stores response in regData *
func RemoveInternalData ¶
func SetErrorResponse ¶
func TrimLogFile ¶
* * The log file is trimmed to 20% of its size when exceeding 10MB. *
func UnpackFilter ¶
func UnpackFilter(filter interface{}, fList *[]FilterObject)
func VerifyTokenSignature ¶
Types ¶
type ClientHandler ¶
type ClientHandler interface {
// contains filtered or unexported methods
}
********************************************************************* Client response handlers *********************
type ClientServer ¶
type FilterObject ¶
type HttpChannel ¶
type HttpChannel struct {
}
type HttpServer ¶
type HttpServer struct {
}
func (HttpServer) InitClientServer ¶
func (server HttpServer) InitClientServer(muxServer *http.ServeMux)
type HttpWSsession ¶
type HttpWSsession struct {
}
func (HttpWSsession) TransportHubFrontendWSsession ¶
func (httpCoreSocketSession HttpWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)
type TransportHubFrontendWSSession ¶
type TransportHubFrontendWSSession interface {
// contains filtered or unexported methods
}
**********Server Core Communications *******************************************************************************
type WsWSsession ¶
type WsWSsession struct {
ClientBackendChannel []chan string
}
func (WsWSsession) TransportHubFrontendWSsession ¶
func (wsCoreSocketSession WsWSsession) TransportHubFrontendWSsession(dataConn *websocket.Conn, appClientChannel []chan string)
Source Files