Add IMDSv2 support to `repository-s3` plugin

[DaveCTurner]

Today we support IMDSv2 in the discovery-ec2 plugin (see #84410) but repository-s3 still only supports IMDSv1. Should we add IMDSv2 support to repository-s3?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ATecha]

I'm running into this at work, where we're required to use IMDSv2.
I'm glad I found the recent post by strophy; I've been pulling my hair out on this as well.
I'd also like this info be added to the documentation (that only v1 is currently supported), to save others from the frustration, at least while v2 support is being added.

[benriou]

We are also interested in IMDSv2 support for the repository-s3 plugin.

[ParinithaNagaraja]

Any updates on when repository-s3 will support IMDSv2 ? Is there a timeline for this?.
Does anyone know of a technical representative who we can contact?

[fabianocosta]

We also need it here. Any updates?

[alanwds]

I spend some time trying to understand the reason why backup (and any other operations related to S3) fail in one of our clusters. Unfortunately, we are using a old version of ES (6.3) and repository-s3 plugin (6.3.2). The response is quit confuse:

{
  "error": {
    "root_cause": [
      {
        "type": "amazon_service_exception",
        "reason": "Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"
      }
    ],
    "type": "amazon_service_exception",
    "reason": "Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)"
  },
  "status": 500
}

The log is also limited

[WARN ][r.suppressed             ] path: /_cat/snapshots/my_repo, params: {s=id, v=, repository=my_repo}
com.amazonaws.AmazonServiceException: Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null)
	at com.amazonaws.internal.EC2CredentialsUtils.handleErrorResponse(EC2CredentialsUtils.java:156) ~[?:?]

Basically, before try to access the S3 bucket, elasticsearch instance try to get their our profile using metadata API. The response is an 401, since IMDSv2 required a valid token before access the API data. This 401 make the elasticsearch operation fail and return the errors listed above. The only solution for make it work is disable IMDSv2 requirement for now.

curl -i 169.254.169.254/latest/meta-data/iam/security-credentials/
HTTP/1.1 401 Unauthorized

I hope that this can be useful for anyone facing this "issue".

[jonkowas]

I (@DaveCTurner) have hidden this comment because it (a) relates to an extremely old version of ES and (b) suggests some very risky actions to modify the Elasticsearch installation. We strongly recommend not running versions of ES which have passed EOL, and definitely discourage the other risky actions mentioned here.

Click to expand original dangerous comment

Hi there,

After some detailed debugging into repository-s3 it seems it uses S3Service from AWS for authentication and it seems there are no elastic search customizations there. In our case AWS SDK (aws-java-sdk-core) was old (1.11.294) and it was NOT supporting IMDSv2.
Minimal versions of all SDKs supporting IMDSv2 can be found here:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#use-a-supported-sdk-version-for-imdsv2
So once we updated the version of AWS SDK to a newer one the support for IMDSv2 comes out of the box!
In fact recent versions of ElasticSearch use recent versions of AWS SDK and therefore the IMDSv2 should be supported without any extra effort from Elastic Search people.
More on that for older version of ElasticSearch is still should be possible to use newer version of AWS SDK because it is backward compatible (in most cases!)

I hope this helps!

Regards,
JV

[robsears]

I (@DaveCTurner) have hidden this comment because it (a) relates to an extremely old version of ES and (b) suggests some very risky actions to modify the Elasticsearch installation. We strongly recommend not running versions of ES which have passed EOL, and definitely discourage the other risky actions mentioned here.

Click to expand original dangerous comment

@alanwds and @jonkowas comments helped unblock me with this issue, and I wanted to add some additional comments for anyone else stumbling over this thread.

It turns out the earliest version of Elasticsearch to have IMDSv2 support is 7.7.0, which was released May 2020. If you have an older version (from 5.x through 7.6.2), you'll need to update the AWS SDK version.

It was initially unclear to me how to go about doing this, but it turns out to be more straightforward than these things usually are.

In the plugins/repository-s3 directory, you'll see several JARs, but of particular interest are: aws-java-sdk-core-<VERSION>.jar, aws-java-sdk-kms-<VERSION>.jar, aws-java-sdk-s3-<VERSION>.jar. Remove these, and replace with some version equal to or greater than 1.11.678, then restart. Repeat for all master-eligible nodes in your cluster. That's it.

Tested this solution with 5.6.16, 6.8.23 and 7.2.0 running on instances that require IMDSv2. For my test, I used the SDK version that was introduced in 7.7.0:

aws-java-sdk-core-1.11.749.jar
aws-java-sdk-kms-1.11.749.jar
aws-java-sdk-s3-1.11.749.jar

I started by confirming that I'd get the 401 response while creating repos. After all master-eligible nodes had those JARs replaced with 1.11.749 and restarted, the Unauthorized (Service: null; Status Code: 401; Error Code: null; Request ID: null) message went away. I could create repos and snapshot/restore as expected.

I bet one could also successfully use the latest version of the SDK. Hope that helps!

[k4kratik]

any updates on this ?