Skip to content

[AWS] Moving edge processors to ingest pipelines, adding ECS version mapping and updating field definitions. #782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 23, 2021

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented Mar 9, 2021

What does this PR do?

This PR moves edge processing to ingest pipelines for AWS vpcflow and updates field definitions + ecs version mapping for the rest of the filesets.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.

Related issues

@P1llus P1llus added enhancement New feature or request Team:Integrations Label for the Integrations team Team:Security-External Integrations Integration:aws AWS labels Mar 9, 2021
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@P1llus
Copy link
Member Author

P1llus commented Mar 9, 2021

Currently the CI will fail on one of the logfiles used for pipeline testing because we use drop processors, I can either remove the loglines that matches the drop processors or we would need to add support for it in elastic-package check + the pipeline tests, ref: elastic/elastic-package#284

@elasticmachine
Copy link

elasticmachine commented Mar 9, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #782 updated

  • Start Time: 2021-03-23T09:37:49.893+0000

  • Duration: 19 min 36 sec

  • Commit: 4100841

Test stats 🧪

Test Results
Failed 0
Passed 245
Skipped 0
Total 245

Trends 🧪

Image of Build Times

Image of Tests

Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding the missing error.message fields.

@mtojek
Copy link
Contributor

mtojek commented Mar 10, 2021

Currently the CI will fail on one of the logfiles used for pipeline testing because we use drop processors, I can either remove the loglines that matches the drop processors or we would need to add support for it in elastic-package check + the pipeline tests, ref: elastic/elastic-package#284

As commented in the other issue, we'll add support for this, but please temporarily disable/remove these lines to make the CI happy.

@mtojek
Copy link
Contributor

mtojek commented Mar 15, 2021

jenkins run the tests please

@P1llus
Copy link
Member Author

P1llus commented Mar 15, 2021

jenkins run the tests please

I saw the error, I can add a fix for that right now, its the event.ingested is not set to dynamic for a specific logfile @mtojek . Its pretty hard to catch all log files, I ended up filing this: elastic/elastic-package#289

@P1llus P1llus merged commit 5c07017 into elastic:master Mar 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:aws AWS Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Convert AWS's edge processing to Ingest Node pipeline
5 participants