WASM Filter Plugins
Last updated
Was this helpful?
Last updated
Was this helpful?
(Wasm) is a binary instruction format for stack-based virtual machines.
Fluent Bit supports integration of Wasm plugins built as Wasm/WASI objects for input and filter plugins only. The interface for Wasm filter plugins is currently under development but is functional.
There are no additional requirements to execute Wasm plugins.
flb-wamrc (optional)flb-wamrc is a flb-prefixed AOT (ahead of time) compiler that's provided from .
For flb-wamrc support, you must install the LLVM infrastructure and some additional libraries (libmlir, libPolly, libedit, and libpfm). For example:
Currently, Fluent Bit supports the following Wasm tool chains:
Rust on wasm32-unknown-unknown
rustc 1.62.1 (e092d0b6b 2022-07-16) or later
on wasm32-wasi
v0.24.0 or later
13 or later.
To support AOT-compiled Wasm execution as filter plugins, build Fluent Bit with -DFLB_WAMRC=On.
Once compiled, you can see new plugins that handle wasm, for example:
The Fluent Bit Wasm filter assumes C ABI, also known as wasm32-unknown-unknown on Rust target and wasm32-wasi on TinyGo target.
Wasm filter plugins execute the function that has the following signature.
For C:
For Go (TinyGo):
For Rust:
The //export XXX attribute on TinyGo and #[no_mangle] attribute on Rust are required. This is because TinyGo and Rust will mangle their function names if they aren't specified.
Once built, a Wasm program will be available. Then you can execute that built program with the following Fluent Bit configuration:
The incoming raw strings from an IIS log are composed of the following fields:
date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken c-authorization-header
The output after the filter logic will be:
This filter approach offers several advantages inherent to programming languages. For example:
It can be extended by adding type conversion to fields, such as sc_bytes, cs_bytes, and time_taken. You can use this to validate data results.
It allows for the use of conditions to apply more descriptive filters. For example, you can get only all logs that contain status codes higher than 4xx or 5xx.
It can be used to define allow lists and deny lists using a data structure array or a file to store predefined IP addresses.
It makes it possible to call an external resource such as an API or database to enhance your data.
It allows all methods to be thoroughly tested and shared as a binary bundle or library.
These examples can be applied in a demo and can serve as an ideal starting point to create more complex logic, depending on your requirements.
To optimize Wasm program execution, there is the option of using flb-wamrc. This option reduces your runtime footprint and to be best performance for filtering operations.
This tool will be built when the -DFLB_WAMRC=On CMake option is specified and LLVM infrastructure is installed on the building box.
For further optimizations to the specific CPU, such as Cortex-A57 series:
Then, when AOT (Ahead Of Time) compiling has succeeded:
AOT compiling should generate CPU architecture-dependent objects. If you want to use an AOT compiled object on different architecture, it must align the target and target cpu for actual environments.
As described in the general options in the guide, Wasm support is enabled by default. Compile Fluent Bit with Wasm support, for example:
TinyGo and WASI SDK support Wasm target by default. When using Rust's wasm32-unknown-unknown target, you must install wasm32-unknown-unknown by using . Then, install the target components as follows:
For example, one of the sample should generate its filtered logs as follows:
Another example of a Rust Wasm filter is the filter. This filter takes the (with some custom modifications) and transforms the raw string into a standard Fluent Bit JSON structured record.