Pipeline tests should assert that error.message is not present

[P1llus]

In the beats module pytests there was 2 checks added to the pipeline/system(?) test, one was checking for error messages in the logs, and another was to check for error.message being populated when it should not.

Currently pipeline tests will PASS if error.message is populated.

Do you know if we can add it in @andrewkroh ? If there is no time, could you maybe point me to the best location to add this functionality so maybe I can add a PR for it?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[mtojek]

The field error.message signals that something went wrong with processing. Currently we supports only positive tests and I'm not sure if it's a quick win to add it.

Could you describe your use case and justify why do you need negative tests?

[andrewkroh]

@P1llus Are you trying to implement negative testing? IIUC you want to assert that error.message is not present in the output of pipeline tests?

I recalled seeing failures caused by error.message. But it's the system tests that fail the test if error.message is present rather than pipeline. @mtojek WDYT about adding a similar check to the pipeline tests. Most (hopefully all) of the log pipelines have an on_failure handler that sets error.message if something unexpected happens.

elastic-package/internal/testrunner/runners/system/runner.go

Lines 213 to 214 in 9253ee3

	if message, err := hit.Source.GetValue("error.message"); err != common.ErrKeyNotFound {
	multiErr = append(multiErr, fmt.Errorf("found error.message in event: %v", message))

[P1llus]

The issue here is that if a pipeline processor has issues, it will populate a message in error.message, even though the pipeline actually failed, the tests passes.

I just want to be able to catch errors in pipelines, as we currently are not able to detect them, especially when we have large logfiles in pipeline tests it might be hard to find a single error in the expected output.

[P1llus]

This is how it currently is implemented for filebeat modules:

            assert "error" not in obj, "not error expected but got: {}.\n The related error message is: {}".format(
                obj, obj["error"].get("message"))

https://github.com/elastic/beats/blob/master/filebeat/tests/system/test_modules.py#L175

[mtojek]

@P1llus @andrewkroh I see your concerns. It's not about negative testing, but just feature parity between system tests and pipeline tests. In this case pipeline tests should behave the same way as system tests (spotted error = fail tests). I'm totally up for introducing this, most likely it was missed before (as for smaller test cases you can see them in git diff).

[ycombinator]

Yep, I'm ++ for introducing this check as well.

[mtojek]

I'm going with this one.