Skip to content

access_loggers: OpenTelemetry custom_tags do not use configured formatters #45453

@kamalmarhubi

Description

@kamalmarhubi

The OpenTelemetry access logger custom_tags don't currently use formatter extensions configured through OpenTelemetryAccessLogConfig.formatters. This means custom_tags.value cannot use formatter extensions configured on the OTel access logger. This will matter more if / when it becomes possible to configure CelExpressionConfig via formatters as suggested in #45420 and #45447.

OpenTelemetryAccessLogConfig has a formatters field for formatter plugins that can be called from the access log configuration:

// Specifies a collection of Formatter plugins that can be called from the access log configuration.
// See the formatters extensions documentation for details.
// [#extension-category: envoy.formatter]
repeated config.core.v3.TypedExtensionConfig formatters = 7;

Those configured formatters appear to be parsed and used for body and attributes, but not for custom_tags. Since custom_tags.value uses substitution formatter syntax, I would expect it to be able to use the same configured formatter extensions as the rest of the OpenTelemetry access log config.

Repro steps:
Set up a config with a non-default envoy.formatter like envoy.formatter.file_content or envoy.formatter.generic_secret added on the OTel access log config.

I would expect that to work the same way configured formatters work in OTel body and attributes.

Admin and Stats Output:
N/A

Config:
A couple of configs using FILE_CONTENT in different places:

attributes-ok.yaml with FILE_CONTENT in attributes
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                direct_response:
                  status: 200
                  body:
                    inline_string: "ok\n"
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          access_log:
          - name: envoy.access_loggers.open_telemetry
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.open_telemetry.v3.OpenTelemetryAccessLogConfig
              log_name: repro
              buffer_size_bytes: 1
              http_service:
                http_uri:
                  uri: http://otel.example.local/v1/logs
                  cluster: otel
                  timeout: 1s
              attributes:
                values:
                - key: debian_version
                  value:
                    string_value: "%FILE_CONTENT(/etc/debian_version)%"
              formatters:
              - name: envoy.formatter.file_content
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.formatter.file_content.v3.FileContent
  clusters:
  - name: otel
    type: STATIC
    connect_timeout: 1s
    load_assignment:
      cluster_name: otel
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 4318
custom-tags-fail.yaml with FILE_CONTENT in custom_tags; fails to validate, see logs
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                direct_response:
                  status: 200
                  body:
                    inline_string: "ok\n"
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          access_log:
          - name: envoy.access_loggers.open_telemetry
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.open_telemetry.v3.OpenTelemetryAccessLogConfig
              log_name: repro
              buffer_size_bytes: 1
              http_service:
                http_uri:
                  uri: http://otel.example.local/v1/logs
                  cluster: otel
                  timeout: 1s
              custom_tags:
              - tag: debian_version
                value: "%FILE_CONTENT(/etc/debian_version)%"
              formatters:
              - name: envoy.formatter.file_content
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.formatter.file_content.v3.FileContent
  clusters:
  - name: otel
    type: STATIC
    connect_timeout: 1s
    load_assignment:
      cluster_name: otel
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 4318

Logs:

with custom-tags-fail.yaml:

❯ docker run --rm \
          -v /tmp/envoy-otel-custom-tags-repro:/etc/envoy:ro \
          envoyproxy/envoy:v1.38.0 \
          --mode validate \
          -c /etc/envoy/custom-tags-fail.yaml
[2026-06-05 02:04:53.607][1][info][main] [source/server/server.cc:961] runtime: {}
[2026-06-05 02:04:53.608][1][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration
[2026-06-05 02:04:53.608][1][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)
[2026-06-05 02:04:53.608][1][info][config] [source/server/configuration_impl.cc:138] loading 1 cluster(s)
[2026-06-05 02:04:53.610][1][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)
[2026-06-05 02:04:53.636][1][critical][main] [source/server/config_validation/server.cc:76] error initializing configuration '/etc/envoy/custom-tags-fail.yaml': Not supported field in StreamInfo: FILE_CONTENT

compared to with attributes-ok.yaml:

❯ docker run --rm \
          -v /tmp/envoy-otel-custom-tags-repro:/etc/envoy:ro \
          envoyproxy/envoy:v1.38.0 \
          --mode validate \
          -c /etc/envoy/attributes-ok.yaml
[2026-06-05 02:05:22.752][1][info][main] [source/server/server.cc:961] runtime: {}
[2026-06-05 02:05:22.754][1][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration
[2026-06-05 02:05:22.754][1][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)
[2026-06-05 02:05:22.754][1][info][config] [source/server/configuration_impl.cc:138] loading 1 cluster(s)
[2026-06-05 02:05:22.755][1][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)
[2026-06-05 02:05:22.758][1][info][config] [source/server/configuration_impl.cc:164] loading stats configuration
configuration '/etc/envoy/attributes-ok.yaml' OK

Call Stack:
N/A

Related issues

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions