Restore snapshot with include_global_state=true is not reverting some custom metadata

[idegtiarenko]

Elasticsearch version (bin/elasticsearch --version): 7.16

Description of the problem including expected versus actual behavior:

According to the code:

elasticsearch/server/src/main/java/org/elasticsearch/snapshots/RestoreService.java

Line 1203 in 0f10825

final Metadata.Builder mdBuilder = Metadata.builder(currentState.metadata());

and

elasticsearch/server/src/main/java/org/elasticsearch/snapshots/RestoreService.java

Lines 1410 to 1422 in 0f10825

	if (metadata.customs() != null) {
	for (ObjectObjectCursor<String, Metadata.Custom> cursor : metadata.customs()) {
	if (RepositoriesMetadata.TYPE.equals(cursor.key) == false
	&& DataStreamMetadata.TYPE.equals(cursor.key) == false
	&& cursor.value instanceof Metadata.NonRestorableCustom == false) {
	// TODO: Check request.skipOperatorOnly for Autoscaling policies (NonRestorableCustom)
	// Don't restore repositories while we are working with them
	// TODO: Should we restore them at the end?
	// Also, don't restore data streams here, we already added them to the metadata builder above
	mdBuilder.putCustom(cursor.key, cursor.value);
	}
	}
	}

when restoring global state from a snapshot we keep existing metadata as a baseline. This means some custom metadata items are going to be preserved as is if they are not present in a global state from the snapshot (if it was taken before the those were configured or even existed).

Steps to reproduce:

• start a new cluster
• configure a snapshot repository
• create a snapshot (with include_global_state=true)
• configure any custom metadata that could be restored from a snapshot (for example auto-follow pattern, but not data repository/stream/autoscaling)
  restore from the snapshot (with include_global_state=true)

Expected result:

• new custom metadata is set to null

Actual result:

• new custom metadata is kept as is

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)