Tags: logstash-plugins/logstash-input-elasticsearch
Tags
Merge pull request #233 from mashhurs/esql-support ES|QL support: - introduces query_type params, accepts dsl or esql option. - adds ES|QL executor to execute ESQL query and parse/map response to event validations - make sure LS (8.17.4+) supports ES|QL (new elasticsearch-ruby client) - make sure connected ES is greater than 8.11+ - query isn't empty or meaningful that starts with command syntax - if query_type is esql, make sure we accept meaningful inputs and do not allow response_type, index, etc.. DSL related params - informing if query isn't using METADATA which adds _id, _version to the response entries - informing ineffective params such as size, search_api, target if users configure ES|QL results field names in a dotted format. The plugin reproduces nested (example {a.b.c: 'val'} => {'a':{'b':{'c':'val'}}})
ES|QL support (#235) * ES|QL support - introduces query_type params, accepts dsl or esql option. - adds ES|QL executor to execute ESQL query and parse/map response to event validations - make sure LS (8.17.4+) supports ES|QL (new elasticsearch-ruby client) - make sure connected ES is greater than 8.11+ - query isn't empty or meaningful that starts with command syntax - if query_type is esql, make sure we accept meaningful inputs and do not allow response_type, index, etc.. DSL related params - informing if query isn't using METADATA which adds _id, _version to the response entries - informing ineffective params such as size, search_api, target if users configure ES|QL results field names in a dotted format. The plugin reproduces nested (example {a.b.c: 'val'} => {'a':{'b':{'c':'val'}}}) * Update CHANGELOG.md
Introduce cursor tracking akin to jdbc input (#205) Provide field value tracking, persisted to disk on each search_after page. Adds `:last_value` and `:present` placeholders, allowing the plugin to inject the cursor value and now-30 seconds, respectively, in the query string. Useful to track new data being written to an index or series of indices. Works best with nano second precision timestamps added by Elasticsearch's Ingest Pipelines. --------- Co-authored-by: Joel Andritsch <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Karen Metts <[email protected]>
Introduce cursor tracking akin to jdbc input (#205) (#234) Provide field value tracking, persisted to disk on each search_after page. Adds `:last_value` and `:present` placeholders, allowing the plugin to inject the cursor value and now-30 seconds, respectively, in the query string. Useful to track new data being written to an index or series of indices. Works best with nano second precision timestamps added by Elasticsearch's Ingest Pipelines. --------- Co-authored-by: Joel Andritsch <[email protected]> Co-authored-by: Rob Bavey <[email protected]> Co-authored-by: Karen Metts <[email protected]>
Tolerate failure (#183) * test setup: ensure presence of /etc/protocols * test setup: actually run secure_integration tests When SECURE_INTEGRATION is speicified, the (non-secure) `:integration` specs are excluded, so we cannot have the `:secure_integration` specs wrapped in a context flagged as `:integration`. * test setup: regnerate test certs (and add regen script) * test setup: give ES the full cert chain In order for the `ca_trusted_fingerprint` specs to work with the CA's fingerprint, ES needs to be configured to present a cert chain that includes the CA. * resilience: prevent failures from crashing plugin When an Event cannot be created directly from the hit, or when the docinfo cannot be merged into a non-hash field in the hit, emit an Event tagged with `_elasticsearch_input_failure` that contains the JSON-encoded hit in `[event][original]` instead of crashing. * add link to changelog * remove orphan method from refactor
Backport support for custom headers (#217) Co-authored-by: flexitrev <[email protected]> Co-authored-by: Karen Metts <[email protected]>
Set x-elastic-product-origin header for ES requests (#211) * Set x-elastic-product-origin header for ES requests This commit updates the `Elasticsearch::Client` used to make requests to ES to send along a header identifying the request as originating from an internal component. * Prep 4.20.5 release Update version and add CHANGELOG entry.
PreviousNext