security-service-integrations: retain event.original value upon pipeline errors

[efd6]

This is the security-service-integrations sub issue for #12045.

• add preserve_original_event in primary on_failure handlers ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12046 google_workspace,jamf_protect,ti_mandiant: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" #12108

• add preserve_original_event in cases of manual setting event.kind to "pipeline_error" ssi_all: add "preserve_original_event" tag to documents with event.kind manually set to "pipeline_error" #12109

• ensure no remove event.original processors remain ssi_all: do not remove event.original in main ingest pipeline #12076 okta: do not remove event.original in main ingest #12127

   for f in $(
   	(
   		for p in $(
   			yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
   			| grep -v -- ---
   		); do
   			find packages/$p -name default.yml
   		done
   	)|sort|uniq
   ); do
   	yq -o=json $f|quatch -l $f -p '{"processors":{"remove":{"field":["event.original"]}}}'
   done
  

  quatch

  • packages/bitdefender/data_stream/push_notifications/elasticsearch/ingest_pipeline/default.yml
  • packages/bitwarden/data_stream/collection/elasticsearch/ingest_pipeline/default.yml
  • packages/bitwarden/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/bitwarden/data_stream/group/elasticsearch/ingest_pipeline/default.yml
  • packages/bitwarden/data_stream/member/elasticsearch/ingest_pipeline/default.yml
  • packages/bitwarden/data_stream/policy/elasticsearch/ingest_pipeline/default.yml
  • packages/cisco_meraki/data_stream/events/elasticsearch/ingest_pipeline/default.yml
  • packages/darktrace/data_stream/ai_analyst_alert/elasticsearch/ingest_pipeline/default.yml
  • packages/darktrace/data_stream/model_breach_alert/elasticsearch/ingest_pipeline/default.yml
  • packages/darktrace/data_stream/system_status_alert/elasticsearch/ingest_pipeline/default.yml
  • packages/eset_protect/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
  • packages/eset_protect/data_stream/device_task/elasticsearch/ingest_pipeline/default.yml
  • packages/eset_protect/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/f5/data_stream/bigipafm/elasticsearch/ingest_pipeline/default.yml
  • packages/f5/data_stream/bigipapm/elasticsearch/ingest_pipeline/default.yml
  • packages/forcepoint_web/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/am_access/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/am_activity/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/am_authentication/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/am_config/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/am_core/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_access/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_activity/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_authentication/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_config/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_core/elasticsearch/ingest_pipeline/default.yml
  • packages/forgerock/data_stream/idm_sync/elasticsearch/ingest_pipeline/default.yml
  • packages/github/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml
  • packages/github/data_stream/dependabot/elasticsearch/ingest_pipeline/default.yml
  • packages/github/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
  • packages/github/data_stream/secret_scanning/elasticsearch/ingest_pipeline/default.yml
  • packages/gitlab/data_stream/api/elasticsearch/ingest_pipeline/default.yml
  • packages/gitlab/data_stream/pages/elasticsearch/ingest_pipeline/default.yml
  • packages/gitlab/data_stream/production/elasticsearch/ingest_pipeline/default.yml
  • packages/gitlab/data_stream/sidekiq/elasticsearch/ingest_pipeline/default.yml
  • packages/google_scc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
  • packages/google_scc/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/google_scc/data_stream/finding/elasticsearch/ingest_pipeline/default.yml
  • packages/google_scc/data_stream/source/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/access_transparency/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/admin/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/context_aware_access/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/device/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/drive/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/gcp/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/group_enterprise/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/groups/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/login/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/rules/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/saml/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/token/elasticsearch/ingest_pipeline/default.yml
  • packages/google_workspace/data_stream/user_accounts/elasticsearch/ingest_pipeline/default.yml
  • packages/imperva_cloud_waf/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/infoblox_bloxone_ddi/data_stream/dhcp_lease/elasticsearch/ingest_pipeline/default.yml
  • packages/infoblox_bloxone_ddi/data_stream/dns_config/elasticsearch/ingest_pipeline/default.yml
  • packages/infoblox_bloxone_ddi/data_stream/dns_data/elasticsearch/ingest_pipeline/default.yml
  • packages/infoblox_nios/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_compliance_reporter/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_protect/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_protect/data_stream/telemetry/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_protect/data_stream/telemetry_legacy/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_protect/data_stream/web_threat_events/elasticsearch/ingest_pipeline/default.yml
  • packages/jamf_protect/data_stream/web_traffic_events/elasticsearch/ingest_pipeline/default.yml
  • packages/jumpcloud/data_stream/events/elasticsearch/ingest_pipeline/default.yml
  • packages/keycloak/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/lastpass/data_stream/detailed_shared_folder/elasticsearch/ingest_pipeline/default.yml
  • packages/lastpass/data_stream/event_report/elasticsearch/ingest_pipeline/default.yml
  • packages/lastpass/data_stream/user/elasticsearch/ingest_pipeline/default.yml
  • packages/lyve_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/m365_defender/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/m365_defender/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/m365_defender/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
  • packages/m365_defender/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/mattermost/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/menlo/data_stream/dlp/elasticsearch/ingest_pipeline/default.yml
  • packages/menlo/data_stream/web/elasticsearch/ingest_pipeline/default.yml
  • packages/microsoft_defender_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/microsoft_defender_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/microsoft_exchange_online_message_trace/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/archive_search_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/dlp_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/message_release_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/threat_intel_malware_customer/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/threat_intel_malware_grid/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/ttp_ap_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/ttp_ip_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/mimecast/data_stream/ttp_url_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/netskope/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
  • packages/netskope/data_stream/events/elasticsearch/ingest_pipeline/default.yml
  • packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/okta/data_stream/system/elasticsearch/ingest_pipeline/default.yml
  • packages/opencanary/data_stream/events/elasticsearch/ingest_pipeline/default.yml
  • packages/panw_cortex_xdr/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
  • packages/panw_cortex_xdr/data_stream/incidents/elasticsearch/ingest_pipeline/default.yml
  • packages/ping_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/pps/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_access/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_cloud/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_cloud/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_cloud/data_stream/host/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_cloud/data_stream/host_profile/elasticsearch/ingest_pipeline/default.yml
  • packages/prisma_cloud/data_stream/incident_audit/elasticsearch/ingest_pipeline/default.yml
  • packages/proofpoint_tap/data_stream/clicks_blocked/elasticsearch/ingest_pipeline/default.yml
  • packages/proofpoint_tap/data_stream/clicks_permitted/elasticsearch/ingest_pipeline/default.yml
  • packages/proofpoint_tap/data_stream/message_blocked/elasticsearch/ingest_pipeline/default.yml
  • packages/proofpoint_tap/data_stream/message_delivered/elasticsearch/ingest_pipeline/default.yml
  • packages/pulse_connect_secure/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/qualys_vmdr/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
  • packages/rapid7_insightvm/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
  • packages/rapid7_insightvm/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
  • packages/santa/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one_cloud_funnel/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one/data_stream/activity/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one/data_stream/agent/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one/data_stream/group/elasticsearch/ingest_pipeline/default.yml
  • packages/sentinel_one/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/slack/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/snyk/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/snyk/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/snyk/data_stream/issues/elasticsearch/ingest_pipeline/default.yml
  • packages/snyk/data_stream/vulnerabilities/elasticsearch/ingest_pipeline/default.yml
  • packages/sophos_central/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/sophos_central/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/symantec_edr_cloud/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
  • packages/symantec_endpoint/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/symantec_endpoint_security/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/symantec_endpoint_security/data_stream/incident/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/action_history/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/client_status/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/discover/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/endpoint_config/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/reporting/elasticsearch/ingest_pipeline/default.yml
  • packages/tanium/data_stream/threat_response/elasticsearch/ingest_pipeline/default.yml
  • packages/teleport/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_io/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_io/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_io/data_stream/scan/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_io/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_sc/data_stream/asset/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_sc/data_stream/plugin/elasticsearch/ingest_pipeline/default.yml
  • packages/tenable_sc/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
  • packages/thycotic_ss/data_stream/logs/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_abusech/data_stream/malwarebazaar/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_abusech/data_stream/malware/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_abusech/data_stream/threatfox/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_abusech/data_stream/url/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_anomali/data_stream/threatstream/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_cif3/data_stream/feed/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_crowdstrike/data_stream/intel/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_crowdstrike/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_cybersixgill/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/apt/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/botnet/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/cc/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/domains/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/files/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/ip/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_eset/data_stream/url/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_maltiverse/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_misp/data_stream/threat_attributes/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_misp/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/tines/data_stream/audit_logs/elasticsearch/ingest_pipeline/default.yml
  • packages/tines/data_stream/time_saved/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_otx/data_stream/pulses_subscribed/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_otx/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_rapid7_threat_command/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_rapid7_threat_command/data_stream/ioc/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_rapid7_threat_command/data_stream/vulnerability/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_recordedfuture/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_threatconnect/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
  • packages/ti_threatq/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
  • packages/trellix_edr_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/trellix_epo_cloud/data_stream/device/elasticsearch/ingest_pipeline/default.yml
  • packages/trellix_epo_cloud/data_stream/event/elasticsearch/ingest_pipeline/default.yml
  • packages/trellix_epo_cloud/data_stream/group/elasticsearch/ingest_pipeline/default.yml
  • packages/trendmicro/data_stream/deep_security/elasticsearch/ingest_pipeline/default.yml
  • packages/trend_micro_vision_one/data_stream/alert/elasticsearch/ingest_pipeline/default.yml
  • packages/trend_micro_vision_one/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/trend_micro_vision_one/data_stream/detection/elasticsearch/ingest_pipeline/default.yml
  • packages/vectra_detect/data_stream/log/elasticsearch/ingest_pipeline/default.yml
  • packages/wiz/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/wiz/data_stream/issue/elasticsearch/ingest_pipeline/default.yml
  • packages/zerofox/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml
  • packages/zeronetworks/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/zscaler_zpa/data_stream/app_connector_status/elasticsearch/ingest_pipeline/default.yml
  • packages/zscaler_zpa/data_stream/audit/elasticsearch/ingest_pipeline/default.yml
  • packages/zscaler_zpa/data_stream/browser_access/elasticsearch/ingest_pipeline/default.yml
  • packages/zscaler_zpa/data_stream/user_activity/elasticsearch/ingest_pipeline/default.yml
  • packages/zscaler_zpa/data_stream/user_status/elasticsearch/ingest_pipeline/default.yml

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

Confirmed no removal of event.original in any pipeline.

$ (for d in $(
        (
                for p in $(
                        yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
                        | grep -v -- ---                                                                                              
                ); do                           
                        find packages/$p -name ingest_pipeline        
                done                                              
        )|sort|uniq
); do
        for f in $d/*.yml; do
                yq -o=json $f|quatch -l $f -p '{"processors":{"remove":{"field":["event.original"]}}}'
        done
done)|wc -l
0