[Azure Logs] Clarify the Storage Account container role in the integration #5604

zmoog · 2023-03-21T09:51:06Z

What does this PR do?

Extend the Azure Logs documentation on the Storage Account. We want to make sure users understand why it is needed and how to set it up to deploy this integration successfully.
Add a small new "Requirements and setup" section in each integration page that links to the general Azure Logs page, containing the requirements and detailed setup instructions.

Motivation

We received some questions about the role of the Storage Account in the Azure Logs integration. At least one user—probably more—couldn't find the needed information on our public-facing documentation.

Here are the questions:

- Do we need to set up and Azure storage account for the integration to work?

- Is it enough with just having an event hub?

- Why is recommended setting up an storage account and how does it help?

- What can happen if we stop using the storage account with the integration?

- If we need to use the storage account, what configuration elastic recommends? Hot or cold storage, how long to store the data in the storage, etc

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Screenshots

Here is a screenshot of the link to the general Azure Logs page that contains requirements and setup information.

elasticmachine · 2023-03-21T10:06:44Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-24T11:30:17.639+0000
Duration: 13 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	125
Skipped	0
Total	125

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-03-21T10:06:55Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`10/10`)	💚
Files	86.364% (`19/22`)	👎 -13.636
Classes	86.364% (`19/22`)	👎 -13.636
Methods	83.333% (`155/186`)	👍 50.0
Lines	84.935% (`2802/3299`)	👎 -15.065
Conditionals	100.0% (`0/0`)	💚

zmoog · 2023-03-21T12:34:15Z

Hey, @alstolten, I drafted an update to the storage account documentation. We probably have better coverage of this topic now, but I'd love to hear WYT.

Should we:

Add more details to existing information?
Cover aspects we currently do not cover (partially or at all)?

Let me know anything that comes to your mind.

zmoog · 2023-03-21T12:35:33Z

You will see the changes twice, but this is normal. The integration stores the doc in source and rendered forms.

alstolten · 2023-03-23T08:37:05Z

Hey @zmoog LGTM. I believe we can take it from here and adjust for more user feedback once collected. Thanks for the changes!

packages/azure/_dev/build/docs/eventhub.md

packages/azure/_dev/build/docs/springcloudlogs.md

dmathieu · 2023-03-23T08:55:09Z

packages/azure/docs/README.md

@@ -69,7 +67,7 @@ To learn more about Event Hubs, refer to [Features and terminology in Azure Even

 The [Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) is a versatile Azure service that allows you to store data in various storage types, including blobs, file shares, queues, tables, and disks.

-The Azure Logs integration uses a Storage account container to store and share information about the Consumer Group (state, position, or offset). Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.
+The Azure Logs integration requries a Storage account container to work. The integration uses the Storage account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and share it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.


Suggested change

The Azure Logs integration requries a Storage account container to work. The integration uses the Storage account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and share it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.

The Azure Logs integration requires a Storage account container to work. The integration uses the Storage account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and shares it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.

I'm not seeing these typos being resolved in the latest diff.

packages/azure/docs/adlogs.md

packages/azure/docs/application_gateway.md

packages/azure/docs/eventhub.md

packages/azure/docs/springcloudlogs.md

alstolten · 2023-03-23T11:29:49Z

packages/azure/_dev/build/docs/eventhub.md

@@ -2,7 +2,7 @@

 The Azure Eventhub Input integration allows users to collect events from Azure event hubs.

-The azure-eventhub input functionality is based on the the event processor host (EPH is intended to be run across multiple processes and machines while load balancing message consumers more on this here https://github.com/Azure/azure-event-hubs-go#event-processor-host, https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host). State such as leases on partitions and checkpoints in the event stream are shared between receivers using an Azure Storage container. 
+The azure-eventhub input functionality is based on the the event processor host (EPH is intended to be run across multiple processes and machines while load balancing message consumers more on this in the [Azure event-hubs-go doc](https://github.com/Azure/azure-event-hubs-go#event-processor-host) and [Azure event-processor doc](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-event-processor-host)). State such as leases on partitions and checkpoints in the event stream are shared between receivers using an Azure Storage container. 


Two minor suggestions:
based on the the event -> remove one "the".
message consumers more on this in the -> message consumers, see

I've added it to the latest commit; please let me know if the updated version works for you.

Tries to answer to the following questions: - Why is recommended setting up an storage account and how does it help? - What can happen if we stop using the storage account with the integration? - If we need to use the storage account, what configuration elastic recommends? Hot or cold storage, how long to store the data in the storage, etc

All requirements and setup instructions are now consolidated in the general Azure Logs page available at: https://docs.elastic.co/integrations/azure Each specific integration page now links to the general page.

Co-authored-by: Damien Mathieu <[email protected]>

zmoog · 2023-03-23T11:45:08Z

@dmathieu I took the opportunity of this review to make additional changes to the Azure Logs integration docs.

Now each integration page is aligned with the following broad structure:

description and purpose
additional info (depending on what is available right now)
requirements and setup

Here are a couple of examples:

zmoog · 2023-03-23T11:51:56Z

This documentation requires revision, but the scope is broader than this PR. I am addressing the most critical changes here and saving the rest for another PR.

packages/azure/_dev/build/docs/README.md

dmathieu · 2023-03-23T13:57:56Z

packages/azure/docs/README.md

@@ -69,7 +67,7 @@ To learn more about Event Hubs, refer to [Features and terminology in Azure Even

 The [Storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) is a versatile Azure service that allows you to store data in various storage types, including blobs, file shares, queues, tables, and disks.

-The Azure Logs integration uses a Storage account container to store and share information about the Consumer Group (state, position, or offset). Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.
+The Azure Logs integration requries a Storage account container to work. The integration uses the Storage account container for checkpointing; it stores data about the Consumer Group (state, position, or offset) and share it among the Elastic Agents. Sharing such information allows multiple Elastic Agents assigned to the same agent policy to work together; this enables horizontal scaling of the logs processing when required.


I'm not seeing these typos being resolved in the latest diff.

zmoog · 2023-03-23T14:03:37Z

I'm not seeing these typos being resolved in the latest diff.

Fixed. Thank you for double-checking.

Co-authored-by: Damien Mathieu <[email protected]>

zmoog · 2023-03-24T12:23:56Z

@dmathieu, do you think this needs more work or we're good to merge?

dmathieu

elasticmachine · 2023-03-24T13:31:24Z

Package azure - 1.5.13 containing this change is available at https://epr.elastic.co/search?package=azure

zmoog self-assigned this Mar 21, 2023

zmoog added Team:Cloud-Monitoring Label for the Cloud Monitoring team docs labels Mar 21, 2023

zmoog changed the title ~~[Azure Logs] Clarify Storage Account container role in the integration~~ [Azure Logs] Clarify the Storage Account container role in the integration Mar 21, 2023

zmoog force-pushed the zmoog/improve-azure-logs-documentation-on-storage-account branch from a37d603 to 93de680 Compare March 21, 2023 12:19

zmoog marked this pull request as ready for review March 22, 2023 19:30

zmoog requested a review from a team as a code owner March 22, 2023 19:30

zmoog requested review from alstolten and colleenmcginnis March 22, 2023 19:31

endorama approved these changes Mar 23, 2023

View reviewed changes

alstolten approved these changes Mar 23, 2023

View reviewed changes

dmathieu reviewed Mar 23, 2023

View reviewed changes

alstolten approved these changes Mar 23, 2023

View reviewed changes

zmoog and others added 5 commits March 23, 2023 12:36

Clarify that the SA container is required

f8bab44

Update changelog

e166ea9

Apply suggestions from code review

bcbf721

Co-authored-by: Damien Mathieu <[email protected]>

zmoog force-pushed the zmoog/improve-azure-logs-documentation-on-storage-account branch from d8d8e02 to d7cfe27 Compare March 23, 2023 11:36

zmoog requested a review from a team as a code owner March 23, 2023 11:36

dmathieu reviewed Mar 23, 2023

View reviewed changes

Revise and clean up data streams docs

631e99f

zmoog and others added 2 commits March 24, 2023 12:29

Update packages/azure/_dev/build/docs/README.md

3e47d20

Co-authored-by: Damien Mathieu <[email protected]>

Fix typo

ee7789d

zmoog force-pushed the zmoog/improve-azure-logs-documentation-on-storage-account branch from 698eb96 to ee7789d Compare March 24, 2023 11:29

dmathieu approved these changes Mar 24, 2023

View reviewed changes

zmoog merged commit 39c90bc into main Mar 24, 2023

zmoog deleted the zmoog/improve-azure-logs-documentation-on-storage-account branch March 24, 2023 13:11

[Azure Logs] Clarify the Storage Account container role in the integration #5604

[Azure Logs] Clarify the Storage Account container role in the integration #5604

Uh oh!

Conversation

zmoog commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Checklist

Screenshots

Uh oh!

elasticmachine commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

zmoog commented Mar 21, 2023

Uh oh!

zmoog commented Mar 21, 2023

Uh oh!

alstolten commented Mar 23, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmathieu Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

dmathieu Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alstolten Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmoog commented Mar 23, 2023

Uh oh!

zmoog commented Mar 23, 2023

Uh oh!

Uh oh!

dmathieu Mar 23, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog commented Mar 23, 2023

Uh oh!

zmoog commented Mar 24, 2023

Uh oh!

dmathieu left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 24, 2023

Uh oh!

Uh oh!

zmoog commented Mar 21, 2023 •

edited

Loading

elasticmachine commented Mar 21, 2023 •

edited

Loading

elasticmachine commented Mar 21, 2023 •

edited

Loading

zmoog Mar 23, 2023 •

edited

Loading