checkpoint_harmony_endpoint: fixes in CEL to control data flow

[chemamartinez]

Proposed commit message

Three changes have been made to improve handling of pulled data:
- Base subsequent request timeranges on the timestamp of the latest data received instead of the last requested range, to avoid losing events that could not be pulled in the previous iteration.
- Adjusted default request limits (results per search, and results per page) to avoid an overload on the API server that could lead to a timeout.
- When receiving a Canceled state[1] for the TaskStatusResponse, gratefully ending, clearing task ID and restarting the sequence for the same timeframe.

Ref:
- [1] https://app.swaggerhub.com/apis-docs/Check-Point/infinity-events-api/1.0.0#/Search%20Event%20Logs/getTaskStatus

Note

Each change is delivered in a different commit for better reviewing.

Note

All data streams have identical cel.yml.hbs files.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0b418f7

cc @chemamartinez

[efd6]

LGTM

Please wrap the commit message body text (also some typos fixed):

Three changes have been made to improve handling of pulled data:
- Base subsequent request time ranges on the timestamp of the latest
  data received instead of the last requested range, to avoid losing
  events that could not be pulled in the previous iteration.
- Adjusted default request limits (results per search, and results per
  page) to avoid an overload on the API server that could lead to a
  timeout.
- When receiving a Canceled state[1] for the TaskStatusResponse,
  gracefully ending, clearing task ID and restarting the sequence for
  the same time frame.

Ref:
- [1] https://app.swaggerhub.com/apis-docs/Check-Point/infinity-events-api/1.0.0#/Search%20Event%20Logs/getTaskStatus

The appearance of setting cursor fields to null to indicate absence is troubling, but that has existed since the package was originally written. It would be nice to fix it, but it can (should) happen in another PR.

[efd6]

This is allowed to be null, i.e. "events": null,. I'm not sure that it's clearer either way and the perf delta would be ~0, so no need to change this, just noting.

[elastic-vault-github-plugin-prod]

Package checkpoint_harmony_endpoint - 0.7.0 containing this change is available at https://epr.elastic.co/package/checkpoint_harmony_endpoint/0.7.0/

[chrisberkhout]

Looks good.

If we find Cancelled happening often, then next thing to try would be to limit the maximum time range.