Multitenancy does not work with non-GRPC ingestion

[mnadeem]

Describe the bug
failed to extract org id when auth_enabled: true

To Reproduce
Steps to reproduce the behavior:

tempo-local.yaml

auth_enabled: true

server:
  http_listen_port: 3100

distributor:
  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/master/receiver
        thrift_http:                   #
        grpc:                          # for a production deployment you should only enable the receivers you need!
        thrift_binary:
        thrift_compact:
    zipkin:
    otlp:
      protocols:
        http:
        grpc:
    opencensus:

ingester:
  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
  #max_block_bytes: 1_000_000           # cut the head block when it hits this size or ...
  traces_per_block: 1_000_000
  max_block_duration: 5m               #   this much time passes

compactor:
  compaction:
    compaction_window: 1h              # blocks in this time window will be compacted together
    max_compaction_objects: 1000000    # maximum size of compacted blocks
    block_retention: 1h
    compacted_block_retention: 10m

storage:
  trace:
    backend: local                     # backend configuration to use
    wal:
      path: E:\practices\docker\tempo\tempo\wal             # where to store the the wal locally
      bloom_filter_false_positive: .05 # bloom filter false positive rate.  lower values create larger filters but fewer false positives
      index_downsample: 10             # number of traces per index record
    local:
      path: E:\practices\docker\tempo\blocks
    pool:
      max_workers: 100                 # the worker pool mainly drives querying, but is also used for polling the blocklist
      queue_depth: 10000

docker run -d --rm -p 6831:6831/udp -p 9411:9411 -p 3100:3100  --name tempo -v E:\practices\docker\tempo\tempo-local.yaml:/etc/tempo-local.yaml --network docker-tempo  grafana/tempo:0.5.0 --config.file=/etc/tempo-local.yaml

docker run -d --rm -p 16686:16686 -v E:\practices\docker\tempo\tempo-query.yaml:/etc/tempo-query.yaml  --network docker-tempo  grafana/tempo-query:0.5.0  --grpc-storage-plugin.configuration-file=/etc/tempo-query.yaml

curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -H 'X-Scope-OrgID: demo' -d '[{
 "id": "1234",
 "traceId": "0123456789abcdef",
 "timestamp": 1608239395286533,
 "duration": 100000,
 "name": "span from bash!",
 "tags": {
    "http.method": "GET",
    "http.path": "/api"
  },
  "localEndpoint": {
    "serviceName": "shell script"
  }
}]'

level=error ts=2021-01-31T07:16:33.6168738Z caller=log.go:27 msg="failed to extract org id" err="no org id"

Expected behavior

Environment:

• Infrastructure: [ Kubernetes, laptop]
• Deployment tool: [manual and jenkins]

Additional Context

[mnadeem]

Same issue with grafana/tempo:latest

[joe-elliott]

"auth_enabled" is a bit misleading, but it is consistent with Cortex and Loki. What this does is simply checks all incoming requests for a tenant id in the X-Scope-OrgID header. Unless you are doing a multitenant Tempo install just set auth_enabled to false.

[mnadeem]

I have enabled multitent in tempo.yaml and started tempo backend

auth_enabled: true

Curl and post man fails even though I am passing the X-Scope-OrgID header

curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -H 'X-Scope-OrgID: demo' 

And the message is

level=error ts=2021-01-31T07:16:33.6168738Z caller=log.go:27 msg="failed to extract org id" err="no org id"

Note: I do want to enable multi tenant

Same behavior in openshift

[annanay25]

This smells like a bug on the Zipkin receiver, where we are not passing our authentication middleware to the zipkin server, but I will have to investigate further to confirm.

[annanay25]

Also for our documentation, multitenancy_enabled is probably a better name for that config label, rather than auth_enabled.

[annanay25]

Update after some preliminary digging -

• Middleware authentication is not passed down to any receiver from the vendored collector
• As of now, we are able to extract tenant information from grpc metadata only, because it translates header info into ctx
• This commit shows what we need at every receiver to make multitenancy work (make sure to view the collapsed vendor changes)
• It may be possible to wrap the Zipkin HTTP server and Jaeger HTTP Server with the middleware auth that we require but it requires better understanding. It is also less obvious how it would work with the other protocols

[joe-elliott]

Although we can try, this unfortunately this might not be fixable the way we reuse the OTEL collector receivers. I've added an example in this PR that shows how you can use multitenancy in Tempo using the OTEL collector:

#497

Note the addition of an X-Scope-OrgID in the OTEL collector. This means that all traces passing through this collector will belong to the named tenant. You can then configure the collector to accept Zipkin.

    headers:
      x-scope-orgid: authed-tenant

The tempo datasource in Grafana also has headers added to indicate the tenant:

  jsonData:
    httpHeaderName1: 'Authorization'
  secureJsonData:
    httpHeaderValue1: 'Bearer authed-tenant'

and finally an extra parameter passed to tempo-query:

    command:
    - --query.bearer-token-propagation=true  # required to pass auth to the grpc plugin

Using the bearer token is unfortunate hack in series of unfortunate hacks. @annanay25 is currently working on dropping the Jaeger-query requirement which will at least cleanup this piece. @annanay25 please remember to clean up this example when you get rid of Jaeger query.

@mnadeem Thank you for reporting this issue. I am going to change the title to reflect that the issue is that multitenancy is not supported on non-GRPC ingestion. I consider everything in this post and the linked PR a workaround. So we will leave this issue open.

[annanay25]

@annanay25 please remember to clean up this example when you get rid of Jaeger query.

👍 I will make sure. This work will start in coordination with the Grafana project team, probably in the next couple of weeks.

[batwork7]

@annanay25 @mnadeem , I found this article to be helpful, but I am also running into the same issue even with auth.enabled set to false. In my case it is single-tenant. I do not want to enable multi tenancy.

Environment : Stand alone Tempo deployment with storage as local
Multi Tenancy : No
auth-enabled : false
Intent : to send Jaeger traces to Tempo for the purposes of tracing from traces to Loki logs.
Tempo Version : 0.7 the latest

[batwork7]

Any help in resolving this issue would be grealy appreciated.

[jjneely]

I believe I'm experiencing this with Tempo 1.2.1 as well. Any updates here?