[aws][1password] Fix field conflicts

[legoguy1000]

What does this PR do?

Resolve field conflicts.

NOTE: The first three commits only contain formatting changes and can be verified via git-generate. It's recommended that you ignore those during the review.

Data Stream Changes

1password.item_usages

• event.created - keyword -> date (per ECS)

1password.signin_attempts

• event.created - keyword -> date (per ECS)

aws.elb_logs

• source.port - keyword to long (per ECS)
• tracing.trace.id renamed to trace.id (per ECS)
• Add missing ECS mappings for several destination, source, event, and http fields.

azure.auditlogs

• client.ip - keyword to ip (per ECS)

azure.eventhub

• azure-eventhub.offset - keyword to long
• azure-eventhub.sequence_number - keyword to long

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Closes [8.0] AWS & 1Password Integrations are creating Index field conflicts #2684
• Closes [azure] Mapping conflicts in Azure Fleet Integrations #2667

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-24T02:41:08.817+0000

• Duration: 38 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	436
Skipped	0
Total	436

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[andrewkroh]

/test

[legoguy1000]

@andrewkroh Should be gtg

[andrewkroh]

/test

[andrewkroh]

Thanks. I expanded the PR description with a list of changes to help the other reviewers.

[andrewkroh]

Looks like the aws.elb_logs needs a convert processor for the port values.

[1] parsing field value failed: field "source.port"'s Go type, string, does not match the expected field type: long (field value: 2817)

CI errors

[andrewkroh]

@elastic/integrations can you please review the AWS and Azure fixes.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[kaiyan-sheng]

Thanks for the fix!! @andrewkroh Is this considered breaking change for users who already have these integrations installed?

[andrewkroh]

@kaiyan-sheng I'd consider this a bugfix because the data didn't comply with ECS. Fleet will rollover their data streams given the change to the template and this will prevent mapping conflicts, but there will be a Kibana index pattern conflict for the between the old data and the new data. At least now the new data from these integrations will not conflict with each other in the logs-* index pattern.

[andrewkroh]

/test

[efd6]

/test

[andrewkroh]

I rebased this so that we can review the fixes separately from the formatting updates to the expected JSON pipeline test files.

[legoguy1000]

@andrewkroh thanks. Apologies for not staying on this, been super busy at work.

[andrewkroh]

No worries! We realize you have a two jobs working here and there 😉 .

[andrewkroh]

/test

[andrewkroh]

LGTM. Approving on behalf of elastic/security-external-integrations.

@elastic/integrations, can you please add your review for the aws and azure changes.

[kaiyan-sheng]

aws and azure looks good. Thanks!

[nicpenning]

How should we handle other field conflicts such as this? New issue?

event.duration is another one for the azure integration:

This is a long everywhere else besides the azure activity logs.

{
".ds-logs-azure.activitylogs-default-2022.03.23-000001" : {
"mappings" : {
"event.duration" : {
"full_name" : "event.duration",
"mapping" : {
"duration" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},

[legoguy1000]

I would do a new issue since this was already merged