如何监控NVIDIA GPU服务器？

最新推荐文章于 2025-07-31 09:51:44 发布

原创最新推荐文章于 2025-07-31 09:51:44 发布 · 1.7k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#docker #GPU #NVIDIA #prometheus #监控

监控系统的应用与开发专栏收录该内容

51 篇文章

订阅专栏

本文介绍如何使用NVIDIA的dcgm-exporter工具监控GPU状态。通过docker安装并运行该工具，可获取包括频率、温度、功耗等在内的多种GPU指标。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

可使用NVIDIA官方提供的dcgm-exporter监控GPU，项目地址：https://github.com/NVIDIA/dcgm-exporter，推荐使用docker安装：

docker run -d --gpus all --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.0.4-3.0.0-ubuntu20.04

需要使用高版本docker，低版本docker无法使用--gpus参数。

如上述镜像无法拉取，可使用我存在阿里云的镜像：

docker run -d --gpus all --rm -p 9400:9400 registry.cn-hangzhou.aliyuncs.com/zhangrongjie/dcgm-exporter:3.0.4-3.0.0-ubuntu20.04

指标地址为：http://localhost:9400/metrics

Grafana dashboard：https://grafana.com/grafana/dashboards/12239-nvidia-dcgm-exporter-dashboard/

指标释义如下：

# HELP DCGM_FI_DEV_SM_CLOCK SM clock frequency (in MHz).
# TYPE DCGM_FI_DEV_SM_CLOCK gauge

# HELP DCGM_FI_DEV_MEM_CLOCK Memory clock frequency (in MHz).
# TYPE DCGM_FI_DEV_MEM_CLOCK gauge

# HELP DCGM_FI_DEV_GPU_TEMP GPU temperature (in C).
# TYPE DCGM_FI_DEV_GPU_TEMP gauge

# HELP DCGM_FI_DEV_POWER_USAGE Power draw (in W).
# TYPE DCGM_FI_DEV_POWER_USAGE gauge

# HELP DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION Total energy consumption since boot (in mJ).
# TYPE DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION counter

# HELP DCGM_FI_DEV_PCIE_REPLAY_COUNTER Total number of PCIe retries.
# TYPE DCGM_FI_DEV_PCIE_REPLAY_COUNTER counter

# HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).
# TYPE DCGM_FI_DEV_GPU_UTIL gauge

# HELP DCGM_FI_DEV_MEM_COPY_UTIL Memory utilization (in %).
# TYPE DCGM_FI_DEV_MEM_COPY_UTIL gauge

# HELP DCGM_FI_DEV_ENC_UTIL Encoder utilization (in %).
# TYPE DCGM_FI_DEV_ENC_UTIL gauge

# HELP DCGM_FI_DEV_DEC_UTIL Decoder utilization (in %).
# TYPE DCGM_FI_DEV_DEC_UTIL gauge

# HELP DCGM_FI_DEV_XID_ERRORS Value of the last XID error encountered.
# TYPE DCGM_FI_DEV_XID_ERRORS gauge

# HELP DCGM_FI_DEV_FB_FREE Framebuffer memory free (in MiB).
# TYPE DCGM_FI_DEV_FB_FREE gauge

# HELP DCGM_FI_DEV_FB_USED Framebuffer memory used (in MiB).
# TYPE DCGM_FI_DEV_FB_USED gauge

# HELP DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL Total number of NVLink bandwidth counters for all lanes
# TYPE DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL counter

# HELP DCGM_FI_DEV_VGPU_LICENSE_STATUS vGPU License status
# TYPE DCGM_FI_DEV_VGPU_LICENSE_STATUS gauge

指标释义可参看：https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/dcgm-exporter.html