Does file input plugin support backpressure

Elastic Stack Logstash
1

We are using a file input plugin 4.4 latest version with logstash 8.17.0.

We have a kafka output plugin using a file input plugin using a persistent queue on a dedicated pipeline.
Does file input support backpressure ,ie if kafka output is blocked does it continue updating the sincedb register?

From our lab test it seems if kafka is not reachable ,sincedb does not get updated. Once the output is available it seems to update sincedb file again.

2

No, this is the oposite of back pressure, if the output is blocked, logstash will fill-up the queue, after the queue is full, it will apply back pressure on the input, in this case it will make the input stop getting new data as it cannot store it anymore on the queue and cannot published the current events.

This is how it should work, if Logstash cannot output the events, it should stop reading new events.

3

THank you for your swift response ! Yes you are correct certain inputs have documented back pressure features for example beats input stops/slows down when the output is blocked.

But is the same applicable for fie input in specific? There are contrasting answers on the net, that file input continues reading even if output is blocked and queue is full and keeps updating the sincedb registry.

There are also some discussions that suggest files stop reading once queue is full.

Official documentation on file input does not specifiy this behaviour explicitly.

4

Do you have an example of those answers? Never saw anything like this in years using Logstash.

Unfortunately there is no official documentation that lists how each inputs deals with back pressure.

But the file input only reads new lines after previous lines received an ack that they were processed by the filters and outputs, so if Logstash cannot process the events it is expected that the file input will not read new lines.

This works in the same way when using filebeat.

1 Like
5

I probably generalised it too soon based on some chatgpt responses, but below forums talk about how filebeat has clear backpressure and a kind of circuitbreaker mechanism and also log the backpressure onset, whilst certain other inputs do not support backpressure. They do not talk about file input in specific though.