Elasticsearch Package #1365

sayden · 2021-07-28T22:24:52Z

This PR migrates the elasticsearch filebeat and metricbeat module into a package.

I'll open it for review but I wouldn't be surprised if some thing aren't totally correct because the migration process for this particular package was very error prone with many manual steps.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.

Reviewers checklist

The migration of this package required some big amount of custom stuff, I'm gonna write it here:

This module had a huge amount of aliases at the module level. Those that weren't relevant for a particular metricset have been manually removed.
Aliases are related to Stack Monitoring UI stuff so they have been removed from the docs as they are irrelevant for users. The downside is that I had to manually write them in the _dev/build/docs/README.md to avoid that they are rewritten during the build proccess. This means they'll need to be updated manually.
All references to metricbeat, filebeat, metricsets and filesets have been removed from the docs

Screenshots

elasticmachine · 2021-07-28T22:40:39Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-12T10:01:26.321+0000
Duration: 12 min 32 sec
Commit: 8d8cd47

Test stats 🧪

Test	Results
Failed	0
Passed	31
Skipped	0
Total	31

Trends 🧪

elasticmachine · 2021-08-04T12:05:30Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-08-04T12:05:31Z

Pinging @elastic/integrations-services (Team:Services)

mtojek

Left few nit-picks to fine-tune. I guess you can pull it in and add more changes/tests in iterations.

mtojek · 2021-08-05T09:07:09Z

packages/elasticsearch/manifest.yml

+        title: Collect Elasticsearch ccr, cluster_stats, enrich, index, index_recovery, index_summary, ml_job, node, node_stats, pending_tasks and shard metrics
+        description: Collecting ccr, cluster_stats, enrich, index, index_recovery, index_summary, ml_job, node, node_stats, pending_tasks and shard metrics from Elasticsearch instances


I suggest to transform it to a more human readable form and rephrase titles and descriptions :)

I left simple titles now, like "Collect elasticsearch logs" or "Collect elasticsearch metrics". The description has a longer text but still human readable and I wrote some descriptions on the data_streams themselves 👍

mtojek · 2021-08-05T09:07:56Z

packages/elasticsearch/manifest.yml

+license: basic
+categories: ["elastic_stack"]
+conditions:
+  kibana.version: ^7.10.0


Please bump up the constraint to 7.15. We don't want this package to be available for earlier versions.

Okay. I wrote ^7.15.0 now, I really didn't know which was is the correct.

mtojek · 2021-08-05T09:14:24Z

packages/elasticsearch/manifest.yml

+name: elasticsearch
+title: Elasticsearch
+version: 0.1.0
+release: beta


maybe mark it as experimental as it's still early stage and lot of stuff to do.

Oh, I read somewhere that all packages must be released as beat and that's why I changed it from experimental to beta. No prob, I revert it back to experimental.

Bump version to 7.15 Rewrite title and descriptions for logs and metrics

sayden · 2021-08-10T10:35:06Z

I have also updated the screenshot to show the changes requested

mtojek

A couple of nit picks

BTW I saw that every data stream starts with "Elasticsearch ..." prefix. I'm not sure if it isn't redundant as all these data streams are only related to Elasticsearch.

mtojek · 2021-08-10T11:18:16Z

packages/elasticsearch/data_stream/index_recovery/manifest.yml

@@ -0,0 +1,15 @@
+type: metrics
+title: Elasticsearch index_recovery metrics


nit: index recovery

mtojek · 2021-08-10T11:26:19Z

packages/elasticsearch/data_stream/cluster_stats/manifest.yml

@@ -0,0 +1,7 @@
+type: metrics
+title: Elasticsearch cluster_stats metrics


nit: cluster stats

sayden · 2021-08-11T15:13:44Z

A couple of nit picks

BTW I saw that every data stream starts with "Elasticsearch ..." prefix. I'm not sure if it isn't redundant as all these data streams are only related to Elasticsearch.

Good point I was mostly following the naming of other modules I took as example.

sayden added 5 commits July 28, 2021 10:24

Migration execution result

f47d694

Add docs and fix stuff

26c7ea2

Follow the stuff on the tips README.md file

bf08393

Fix the stuff that appears on the tips

6470c27

Update some docs

5a32159

sayden added enhancement New feature or request Team:Integrations Label for the Integrations team Team:Services (Deprecated) Label for the former Integrations-Services team labels Jul 28, 2021

sayden self-assigned this Jul 28, 2021

sayden added 13 commits July 29, 2021 17:54

Those files shouldn't be created if no policies apply

bbf88ea

Fix readmes

8832b40

Remove docs that weren't part of a policy

1181b59

Original readme

9edf22b

Remove aliases from README.md files

1468541

Remove duplicated/unrelated package level fields

c7cc7cd

Update sample events, remove references to metricbeat

d706e51

Update readme

40dc50e

Add changelog

8a49400

Update some descriptions

2dbb81f

Fix format of package

5ecf9b4

Fix failing tests

ce6a86e

Format packages

7480b97

sayden marked this pull request as ready for review August 4, 2021 12:05

mtojek self-requested a review August 5, 2021 08:35

mtojek approved these changes Aug 5, 2021

View reviewed changes

Change release to experimental

8252e0a

Bump version to 7.15 Rewrite title and descriptions for logs and metrics

mtojek approved these changes Aug 10, 2021

View reviewed changes

Rewrite descriptions. Added datastore to categories

8d8cd47

sayden merged commit b338600 into elastic:master Aug 12, 2021

jasonrhodes mentioned this pull request Aug 13, 2021

[Stack Monitoring] Manually test package-driven collection elastic/kibana#108543

Closed

3 tasks

sayden deleted the migration/sm/elasticsearch branch August 26, 2021 16:09

matschaffer mentioned this pull request May 19, 2022

ES package logo is missing pieces #3379

Closed

andrewkroh added Integration:elasticsearch Elasticsearch New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

consulthys mentioned this pull request Jan 29, 2025

[Elasticsearch]: Ingest pipeline created to process Elasticserver logs truncates log messages #12501

Closed

		title: Collect Elasticsearch ccr, cluster_stats, enrich, index, index_recovery, index_summary, ml_job, node, node_stats, pending_tasks and shard metrics
		description: Collecting ccr, cluster_stats, enrich, index, index_recovery, index_summary, ml_job, node, node_stats, pending_tasks and shard metrics from Elasticsearch instances

		@@ -0,0 +1,15 @@
		type: metrics
		title: Elasticsearch index_recovery metrics

		@@ -0,0 +1,7 @@
		type: metrics
		title: Elasticsearch cluster_stats metrics

Elasticsearch Package #1365

Elasticsearch Package #1365

Uh oh!

Conversation

sayden commented Jul 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Reviewers checklist

Screenshots

Uh oh!

elasticmachine commented Jul 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

Uh oh!

elasticmachine commented Aug 4, 2021

Uh oh!

elasticmachine commented Aug 4, 2021

Uh oh!

mtojek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayden commented Aug 10, 2021

Uh oh!

mtojek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayden commented Aug 11, 2021

Uh oh!

Uh oh!

sayden commented Jul 28, 2021 •

edited

Loading

elasticmachine commented Jul 28, 2021 •

edited

Loading