# FlatBuffers and WAMP
## WAMP payload transparency mode
### Motivation
**tldr:**
* performance (passthrough of app payloads, general zero copy)
* support end-to-end encryption
* carrying proprietory binary payloads (MQTT)
* strict static typing of interfaces
---
WAMP supports both positional and keyword based request arguments and returns.
For example, the WAMP `CALL` message allows for the following three alternative forms:
1. `[CALL, Request|id, Options|dict, Procedure|uri]`
2. `[CALL, Request|id, Options|dict, Procedure|uri, Arguments|list]`
3. `[CALL, Request|id, Options|dict, Procedure|uri, Arguments|list, ArgumentsKw|dict]`
The actual application payload hence can take **three variants of app payload (XX)**:
1. `-`
2. `Arguments|list`
3. `Arguments|list, ArgumentsKw|dict`
This pattern repeats across **all** WAMP messages that can carry application payload, namely the following 7 WAMP messages:
* `PUBLISH`
* `EVENT`
* `CALL`
* `INVOCATION`
* `YIELD`
* `RESULT`
* `ERROR`
> Note: should the proposed new WAMP messages `EVENT_RECEIVED` and `SUBSCRIBER_RECEIVED` be introduced,
these also carry application payload, and would follow the same approach.
The approach taken (**XX**) allows for a number of useful features:
1. flexible support of popular dynamically typed serializers, namely: JSON, MsgPack, CBOR and UBJSON
2. allow arbitrary adhoc extensibility (as the router basically does not care about new app payloads)
3. transparantly translate the *application payload* between serialization formats used by different clients connected at the same time.
4. support optional router side application payload validation: both static, and dynamic (calling into user supplied payload validators)
However, a number of downsides have become apparent as well:
1. resource consumption: serialization/deserialization can eat significant chunks of CPU, and produce GC pressure
2. legacy (MQTT) and proprietory payloads that should simply be transported "as is" (passthrough, without ever touching)
3. as apps and systems get larger and more complex, the dynamic typing flexibility turns into a major problem: **the internal and external interfaces and APIs in a microservices based application must be relied upon and their evolution actively managed**
The latter does not mean an "either or" question. You can have important base APIs and external interfaces defined rigorously, using static, strict typing discipline, while at the same time have other parts of your system evolve more freely, basically allowing weakly and dynamically typed data exchange - for limited areas.
---
### Payload Transparency Mode
**Payload Transparancy Mode (PTM)** adds a 4th application payload variant to above **XX**
4. `[CALL, Request|id, Options|dict, Procedure|uri, Payload|binary]`
where the actual application payload takes this form:
4. `Payload|binary`
**PTM** can be used on a *per message basis*.
If PTM is used, then the following two attributes MUST be present in the `Details|dict` or `Options|dict` of the respective WAMP message:
* `payload|string`: Application payload type:
* `"plain"`: Plain WAMP application payload.
* `"cryptobox"`: Encrypted WAMP application payload. This is using WAMP-cryptobox (Curve25519 / Cryptobox).
* `"opaque"`: Raw pass-through of app payload, uninterpreted in any way.
* `serializer|string`: Application payload serializer type.
* `"transport"`: Use same (dynamic) serializer for the app payload as on the transport. This will be one of JSON, MSGPACK, CBOR or UBJSON.
* `"json"`: Use JSON serializer (for dynamically typed app payload).
* `"msgpack"`: Use MsgPack serializer (for dynamically typed app payload).
* `"cbor"`: Use CBOR serializer (for dynamically typed app payload).
* `"ubjson"`: Use UBJSON serializer (for dynamically typed app payload).
* `"opaque"`: Raw pass-through of app payload, uninterpreted in any way.
* `"flatbuffers"`: Explicit use of FlatBuffers also for (statically typed) payload.
* `key|string`: When using end-to-end encryption (WAMP-cryptobox), the public key to which the payload is encrypted
---
#### Payload PLAIN
When PTM is in use, and `Details.payload=="plain"`, the original application payload, one of
* `-`
* `Arguments|list`
* `Arguments|list, ArgumentsKw|dict`
is serialized according to the (dynamically typed) serializer specified in `serializer`, one of
* `"transport"`
* `"msgpack"`
* `"json"`
* `"cbor"`
* `"ubjson"`
> Note that serializers `"opaque"` and `"flatbuffers"` are illegal for payload `"plain"`.
---
#### Payload CRYPTOBOX
Write me.
---
#### Payload OPAQUE
Write me.
---
## FlatBuffers
FlatBuffers is a zero-copy serialization format open-sourced by Google in 2014 under the Apache 2 license.
Supported operating systems include:
* Android
* Linux
* MacOS X
* Windows
Supported programming languages include:
* C++
* C#
* C
* Go
* Java
* JavaScript
* PHP
* Python
---
## Building
1. Get [cmake](https://cmake.org/)
2. Follow [Building with CMake](https://github.com/google/flatbuffers/blob/master/docs/source/Building.md#building-with-cmake)
Essentially:
```console
git clone https://github.com/google/flatbuffers.git
cd flatbuffers
cmake -G "Unix Makefiles"
make -j4
./flatc --version
```
---
### flatc patches
The following patches are required currently for flatc: [this PR](https://github.com/google/flatbuffers/pull/4713) adds pieces missing for reflection in the flatc compiler.
---
## Usage
This folder contains a [Makefile](Makefile) that allows to compile all WAMP FlatBuffers schemata and generate binding code in various languages using
```console
make clean build cloc
```
Here is example output (a total of 30k LOC generated from 500 LOC .fbs):
```console
cpy2714_1) oberstet@thinkpad-t430s:~/scm/wamp-proto/wamp-proto/flatbuffers$ make clean build cloc
rm -rf ./_build
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/schema/ --binary --schema --bfbs-comments --bfbs-builtin-attrs *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/cpp/ --cpp *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/csharp/ --csharp *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/go/ --go *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/java/ --java *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/js/ --js *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/php/ --php *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/python/ --python *.fbs
/home/oberstet/scm/xbr/flatbuffers/flatc -o ./_build/ts/ --ts *.fbs
cloc --read-lang-def=cloc.def --exclude-dir=_build .
11 text files.
11 unique files.
2 files ignored.
github.com/AlDanial/cloc v 1.76 T=0.01 s (774.6 files/s, 113640.1 lines/s)
---------------------------------------------------------------------------------------
Language files blank comment code
---------------------------------------------------------------------------------------
FlatBuffers 7 314 372 535
Markdown 1 64 0 119
make 1 20 8 30
Windows Module Definition 1 0 0 5
---------------------------------------------------------------------------------------
SUM: 10 398 380 689
---------------------------------------------------------------------------------------
cloc ./_build
331 text files.
331 unique files.
2 files ignored.
github.com/AlDanial/cloc v 1.76 T=0.35 s (942.1 files/s, 132452.9 lines/s)
-------------------------------------------------------------------------------
Language fi