Option to log all request and response headers on the access log

[Brunomachadob]

Title: Option to log all request and response headers on the access log

Description:

Hey all, I would like to run something by you that could be beneficial (although it also comes with some cons).

Right now, we are forced to explicitly set all information/headers we want on the access logs, but it could be beneficial to also dump all headers (request and responses), due to:

1. Logging weird/unexpected headers might be useful to detect threats and their patterns.
   1.1. One good example is when the Log4J RCE exploit came around, nobody was specifically looking at a header called spring.cloud.function.routing-expression that was used in the exploit.
2. When we decide to change the configuration with new headers (sometimes due to 1), we must change a bunch of configs and deploy a bunch of Envoys all over again.

Here's a configuration example:

access_log:
  - name: "envoy.access_loggers.file"
    typed_config:
      '@type': "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
      path: "/dev/stdout"
      log_format:
        json_format:
          timestamp: "%START_TIME%"
          schema: "v1.0.0"
          source: "%DOWNSTREAM_REMOTE_ADDRESS%"
          status: "%RESPONSE_CODE%"
          method: "%REQ(:METHOD)%"
          duration: "%DURATION%"
          response_flags: "%RESPONSE_FLAGS%"
          user_agent: "%REQ(USER-AGENT)%"
          upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
          downstream:
            tls:
              version: "%DOWNSTREAM_TLS_VERSION%"
          http:
            request:
              method: "%REQ(:METHOD)%"
              bytes: "%BYTES_RECEIVED%"
              headers: "%REQ_ALL_HEADERS%" # non-existent

I understand this could bring two problems:

1. Performance issues if there's too many headers?
   1.1. Could we somehow limit the amount of headers or value sizes?
2. Logging of sensitive information
   2.1. Not exactly the same as discussed here, but this seems like your current opinion on the matter: debug: redact sensitive info from debug logs #27579 (comment)
   2.2. In our case we may treat (the known ones, maybe even unknown ones depending on their value pattern?) in our logging pipeline

Known workaround

After some research, I noticed the following workaround where all the information dynamically processed through a Lua script, added to the dynamic metadata, which is then used on the access log config:
https://themartian.hashnode.dev/http-request-body-logging-with-istio-and-envoy
Not sure how comfortable I am with this though.

What are your opinions on this topic?

Thanks!

[phlax]

it would definitely need some kind of configurable obfuscation, which might mean its better to roll your own if you really want this

cc @mattklein123 @yanavlasov as responders on related ticket

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.