Restoring a v${N-2} index should apply necessary blocks automatically

[DaveCTurner]

Elasticsearch 9.x admits indices created in 7.x as long as the indices were marked as readonly in version ≥8.18.0, as indicated by the private index.verified_read_only setting. However if a user has a snapshot from an older (8.x or 7.x) cluster then today we cannot restore any 7.x indices therein. Instead, the user must restore the snapshot into an 8.last cluster, mark the indices as readonly, take another snapshot of the now-verified-read-only indices, and then restore that snapshot into the 9.x cluster.

Is this process necessary on a snapshot restore or could we treat an old index being restored from a snapshot as verified-read-only and apply the needed settings automatically?

Relates https://discuss.elastic.co/t/378853/

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

Speaking with the team, this is a little trickier than just™ applying the index.verified_read_only setting during restore: the snapshot may have been taken while actively indexing and in that case the restored shards may have gaps in their operation sequence (i.e. they have GCP ≠ MSN) that a normal restore would fix up, but the normal process won't work on these older indices. Instead we need to do some alternative fix-up process to fill in any gaps.

It should be easy enough to reproduce the problem with a test case that derives from AbstractIndexCompatibilityTestCase.

I am however relabelling this as :Distributed Indexing/Recovery since it seems that there is nontrivial recovery work left to do (and the associated ES-11248 is labelled similarly).

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)