​OpenObserve + Logstash + Filebeat 轻量化监控Nginx

📦 整体架构

Nginx -->|JSON 日志| Filebeat -->|5044 端口| Logstash
Logstash -->|HTTP 输出| OpenObserve

🛠️ 部署步骤

1. ​创建目录结构​

mkdir -p ob-stack/{filebeat,logstash/pipeline,nginx/conf}
cd ob-stack

2. ​**编写 docker-compose.yml**​

version: '3.8'
services:
  # OpenObserve 服务（对接对象存储）
  openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    ports:
      - "5080:5080"
    environment:
      ZO_ROOT_USER_EMAIL: "admin@example.com"
      ZO_ROOT_USER_PASSWORD: "Complexpass#123"
    networks:
      - obs-net

  # Logstash 服务
  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.4
    ports:
      - "5044:5044"
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    environment:
      LS_JAVA_OPTS: "-Xmx2g -Xms2g"
    networks:
      - obs-net

  # Filebeat 服务
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.13.4
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - ./nginx/logs:/var/log/nginx
    networks:
      - obs-net
    depends_on:
      - nginx
      - logstash

  # Nginx 服务（日志源）
  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./nginx/logs:/var/log/nginx
      - ./nginx/conf/log-json.conf:/etc/nginx/conf.d/log-json.conf
    networks:
      - obs-net

networks:
  obs-net:
    driver: bridge

⚙️ 关键配置详解

​1. Nginx JSON 日志配置​

nginx/conf/log-json.conf：

log_format json_combined escape=json
  '{'
    '"timestamp":"$time_iso8601",'
    '"client_ip":"$remote_addr",'
    '"method":"$request_method",'
    '"path":"$uri",'
    '"status":$status,'
    '"response_size":$body_bytes_sent,'
    '"user_agent":"$http_user_agent"'
  '}';

access_log /var/log/nginx/access.log json_combined;

​2. Filebeat 配置​

filebeat/filebeat.yml：

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true  # 直接解析 JSON 字段[6,8](@ref)
  json.overwrite_keys: true

output.logstash:
  hosts: ["logstash:5044"]  # 发送到 Logstash 容器

​3. Logstash 管道配置​

logstash/pipeline/logstash.conf：

input {
  beats { port => 5044 }
}

filter {
  # 时间戳标准化（对齐 OpenObserve）
  date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
  }
  # 删除冗余字段
  mutate {
    remove_field => ["log", "input", "ecs"]
  }
}

output {
  http {
    url => "http://8.154.24.125:5080/api/default/default/_json"
    http_method => "post"
    format => "json_batch"
    headers => {
      "Authorization" => "Basic YWRtaW5AZXhhbXBsZS5jb206Q29tcGxleHBhc3MjMTIz"
      "Content-Type" => "application/json"
    }
  }
}

💾 ​性能对比​：列式存储 + 压缩技术使 OpenObserve 存储成本比 ES ​降低 140倍，查询速度快 3-5 倍4。

✅ ​启动与验证​

# 生成 Base64 凭证（用于 Logstash 认证）
echo -n "admin@example.com:Complexpass#123" | base64 > .env
export BASE64_CREDS=$(cat .env)

# 启动所有服务
docker-compose up -d

# 生成测试日志
curl http://localhost

# 登录 OpenObserve 查询
浏览器访问 http://localhost:5080 → 输入账号密码 → 执行查询：

SELECT * FROM "default" WHERE status = 200

通过此方案，您已构建了一个存储成本极低、查询高效、支持 PB 级日志的云原生观测平台，替代传统 ELK 方案可节省 90% 以上成本4。