[CI] RemoteClusterSecurityEsqlIT testCrossClusterAsyncQueryStop failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6062 / rhel-8_platform-support-unix
• elasticsearch-pull-request #53217 / checkpart4_release-tests
• elasticsearch-pull-request #53073 / checkpart4_release-tests
• elasticsearch-pull-request #52965 / checkpart4_release-tests
• elasticsearch-pull-request #52925 / part-4
• elasticsearch-pull-request #52823 / checkpart4_release-tests
• elasticsearch-pull-request #52785 / part-4
• elasticsearch-pull-request #52753 / part-4

Reproduction Line:

./gradlew ":x-pack:plugin:security:qa:multi-cluster:javaRestTest" --tests "org.elasticsearch.xpack.remotecluster.RemoteClusterSecurityEsqlIT.testCrossClusterAsyncQueryStop" -Dtests.seed=8E517D83CCAB7558 -Dtests.locale=en-TZ -Dtests.timezone=Asia/Damascus -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <false>
     but: was <true>

Issue Reasons:

• [main] 4 consecutive failures in step checkpart4_release-tests
• [main] 8 failures in test testCrossClusterAsyncQueryStop (1.0% fail rate in 802 executions)
• [main] 3 failures in step part-4 (0.8% fail rate in 364 executions)
• [main] 4 failures in step checkpart4_release-tests (100.0% fail rate in 4 executions)
• [main] 7 failures in pipeline elasticsearch-pull-request (2.0% fail rate in 355 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 consecutive failures in step checkpart4_release-tests
• [main] 8 failures in test testCrossClusterAsyncQueryStop (1.0% fail rate in 802 executions)
• [main] 3 failures in step part-4 (0.8% fail rate in 364 executions)
• [main] 4 failures in step checkpart4_release-tests (100.0% fail rate in 4 executions)
• [main] 7 failures in pipeline elasticsearch-pull-request (2.0% fail rate in 355 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6062 / rhel-8_platform-support-unix
• elasticsearch-pull-request #53217 / checkpart4_release-tests
• elasticsearch-pull-request #53073 / checkpart4_release-tests
• elasticsearch-pull-request #52965 / checkpart4_release-tests
• elasticsearch-pull-request #52925 / part-4
• elasticsearch-pull-request #52823 / checkpart4_release-tests
• elasticsearch-pull-request #52785 / part-4
• elasticsearch-pull-request #52753 / part-4

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[smalyshev]

This seems to be the problem:

if (asyncListener == null) {
            // This should mean one of the two things: either bad request ID, or the query has already finished
            // In both cases, let regular async get deal with it.
            var getAsyncResultRequest = new GetAsyncResultRequest(asyncIdStr);
            // TODO: this should not be happening, but if the listener is not registered and the query is not finished,
            // we give it some time to finish
            getAsyncResultRequest.setWaitForCompletionTimeout(new TimeValue(1, TimeUnit.SECONDS));
            getResultsAction.execute(task, getAsyncResultRequest, listener);
            return;
        }

Looks like we get into situations where 1s is not enough - and properly, we shouldn't be needing this at all.

[smalyshev]

More coherent explanation: sometimes, the execution listener is completed before the result data is stored (since the listener completion seems to be detached from the store completion, because we hook execution listener after it has been wrapped by async manager). In such cases, we may try to return the result before it has actually been stored properly.