Elasticsearch 8.11.1 RPM restarts process during update

[holgr]

Elasticsearch Version

8.11.1

Installed Plugins

No response

Java Version

bundled

OS Version

Rocky 9.2 5.14.0-70.26.1.el9_0.x86_64

Problem Description

While updating several nodes from ES 8.11.0 to 8.11.1 I noticed that the nodes tried to restart the elasticsearch process. In previous versions this was not the case and required a manual step. Additionally the restart process failed on all nodes I updated. A manual systemctl restart elasticsearch worked however and the nodes are back up.

Steps to Reproduce

Install the elasticsearch-8.11.1 RPM from https://artifacts.elastic.co/packages/8.x/yum/

Logs (if relevant)

Nov 13 21:21:09 hostname systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 13 21:21:09 hostname systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Nov 13 21:21:09 hostname systemd[1]: elasticsearch.service: Consumed 21min 32.788s CPU time.

[ceeeekay]

I've also run into this issue with the .deb installer. Normally I would preinstall the upgrade and then restart the nodes in an orderly fashion, however this upgrade caused my entire cluster to restart at the same time on 8.11.1, and required manual intervention.

[holgr]

Yep, that's my exact workflow as well. Glad I started on a cluster where it didn't really matter.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[mark-vieira]

I was unable to replicate this when upgrading 8.10.0 to 8.11.1. The only scenario in which the service should be restarted on upgrade is if RESTART_ON_UPGRADE=true in /etc/sysconfig/elasticsearch and by default that is commented out. Was this perhaps the case? Can you provide which version you upgraded from?

See: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#rpm-configuring

[holgr]

The ones described above went from 8.11.0 to 8.11.1. I've since upgraded one additional cluster. It came from 8.10.4 (I think? – it was on the last 8.10.x version) and didn't have this issue.

RESTART_ON_UPGRADE=true is commented out on all of these nodes.

[ceeeekay]

I have the same, but in /etc/default/elasticsearch in my case. Also upgrading 8.11.0 to 8.11.1.

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#ES_RESTART_ON_UPGRADE=true

[mark-vieira]

I wasn't able to reproduce this going from 8.11.0 to 8.11.1 either. There's nothing in our packaging scripts that would call systemctl unless RESTART_ON_UPGRADE is set. Something must have triggered the restart though. Can you share any of the journalctl log entries immediately prior to the start failure. I'm wondering if there's anything that might indicate what triggered the restart.

[ceeeekay]

Here's what I have: https://gist.github.com/ceeeekay/8e407092ef24ac89dec897c1c1748e1a

Seems to be looking for x-pack-core-8.11.0.jar and not finding it.

[mark-vieira]

This sound suspiciously like the server actually errored during the upgrade and systemd attempted to restart it. Looks like we tried to load a class from a jar that no longer exist because it was replaced during the upgrade. This is likely a real problem.

@rjernst Thoughts here? This was likely always a problem. We don't really support in-place distribution upgrades and we might even document this somewhere but this is exactly what a Linux package upgrade is doing. I'm wondering if doing a package upgrade on a running node should be supported, or if we should stop, upgrade, then restart to avoid this scenario.

Second, I'm wondering if there have been some changes that make this more likely. Are we loading service providers more often? Or not caching results of these when we probably should? I would expect issues with trying to load classes from jars on disk to be rare for nodes that have been running for any length of time. It does indeed look like there were some recent changes in 8.11 in the specific stacktrace linked above.