[Nvidia/GPU] Introduce Nvidia GPU Integration

[strawgate]

Proposed commit message

Introduce NVIDIA GPU Monitoring Integration

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

How to test this PR locally

Deploy NVIDIA DGCM on a device with an NVIDIA GPU to get a prometheus metrics endpoint that you can provide to the integration.

If you have docker this just requires:

docker run -d --gpus all --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
curl localhost:9400/metrics

Configure the integration to point at the host running the container and GPU http://nvidiahost:9400/metrics

Some metrics are not enabled by default with the container, enabling all metrics requires some extra steps.

Related issues

Fixes #11930

Screenshots

WIP:

[strawgate]

Todo: Add k8s container, pod, and namespace info from labels

Should also include labels / mapping for kubernetes container, pod, and namespace

container="dcgmproftester11",namespace="default",pod="dcgmproftester"

container => kubernetes.container.name
namespace => kubernetes.namespace
pod => kubernetes.pod.name

- name: kubernetes
  type: group
  fields:
    - name: pod.name
      type: keyword
      description: >
        Kubernetes pod name
    - name: container.name
      type: keyword
      description: >
        Kubernetes container name
    - name: namespace
      type: keyword
      description: >
        Kubernetes namespace

- rename:
    field: prometheus.labels.container
    target_field: kubernetes.container.name
    ignore_missing: true
- rename:
    field: prometheus.labels.namespace
    target_field: kubernetes.namespace
    ignore_missing: true
- rename:
    field: prometheus.labels.pod
    target_field: kubernetes.pod.name
    ignore_missing: true

Perhaps with corresponding dashboard elements

[ishleenk17]

Thanks for the contribution.
Shared inital set of comments from the 1st review.

[agithomas]

While going through this issue, there is a mention about

• high resource utilisation
• deprecation of a few metrics including DCGM_FI_DEV_GPU_UTIL.

Is this scenario observed while integration testing? If there are metrics that are deprecated or result in high resource intensive, it would be best to not consider this metric for creating dashboard visualisation.

[strawgate]

Will remove this metric and consider replacing with the referenced ones

[agithomas]

Added @daniela-elastic as the reviewer for the dashboard.

[daniela-elastic]

Looks good and great to see starting to provide support for GPUs. Minor comments (important if we want to move this integration to GA) - it would be great to have some totals at the top of the dashboard - eg total energy consumption, total number of errors, etc. Also, it would be great to have the dashboard follow best practices for integration dashboards described here.

[agithomas]

From the comments, I understand that there is certain improvements we must make on the dashboard.

The PR has been in the hold state for some time.

We can take up the dashboard improvements as part of the new release of the package.

PR Apporved.

[agithomas]

@strawgate , you have all the approvals for this PR. Do you want to consider merging? Do let us know if you need any help.

[strawgate]

Enabling the expensive metrics is a client side change and so us showing them isn't a problem afaik.

I'm good with merging and opening an issue with the current feedback

will bump min kibana ver to try to fix build

[agithomas]

It appears that the build.yml, reference, is missing from this package causing the error.

Also, a few fields mappings are also missing, leading to the following errors

test case failed: one or more errors found in document: [0] field "service.address" is undefined
[1] field "service.type" is undefined
[2] field "ecs.version" is undefined
[3] field "prometheus.node.name" is undefined
[4] field "prometheus.up.value" is undefined

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
92.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 54f2b09

History

• 💚 Build #24794 succeeded 24ebd64
• 💔 Build #24791 failed b37d119
• 💔 Build #24790 failed 7adc794
• 💔 Build #24271 failed f5d45f3
• 💔 Build #22530 failed 11d5263
• 💔 Build #22526 failed 58dae3b

[elastic-vault-github-plugin-prod]

Package nvidia_gpu - 0.1.0 containing this change is available at https://epr.elastic.co/package/nvidia_gpu/0.1.0/