Avoiding watcher validation errors when a data stream points to more than one index #85507

masseyke · 2022-03-30T20:02:08Z

We are seeing this error in 8.0.1 if a node is restarted after the .watcher-history-16 data stream is pointing to more than one index.

[instance-0000000000] error validating to start watcher
java.lang.IllegalStateException: Alias [.watcher-history-16] points to more than one index
	at org.elasticsearch.xpack.watcher.watch.WatchStoreUtils.getConcreteIndex(WatchStoreUtils.java:32) ~[x-pack-watcher-8.1.0.jar:8.1.0]
	at org.elasticsearch.xpack.watcher.history.HistoryStore.validate(HistoryStore.java:79) ~[x-pack-watcher-8.1.0.jar:8.1.0]
	at org.elasticsearch.xpack.watcher.WatcherService.validate(WatcherService.java:158) [x-pack-watcher-8.1.0.jar:8.1.0]
	at org.elasticsearch.xpack.watcher.WatcherLifeCycleService.clusterChanged(WatcherLifeCycleService.java:162) [x-pack-watcher-8.1.0.jar:8.1.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:564) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:550) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:510) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:428) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260) [elasticsearch-8.1.0.jar:8.1.0]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223) [elasticsearch-8.1.0.jar:8.1.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

This change makes it ok to use an alias with a write index or a data stream (which always has a write index).
Closes #85508

…than one index

elasticmachine · 2022-03-30T20:02:11Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2022-03-30T20:08:04Z

Hi @masseyke, I've created a changelog YAML for you.

martijnvg

Looks good! I left a question and a comments about the tests.

martijnvg · 2022-03-31T13:21:47Z

x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/watch/WatchStoreUtils.java

        }

-        if (indexAbstraction.getType() != IndexAbstraction.Type.CONCRETE_INDEX && indexAbstraction.getIndices().size() > 1) {
+        if ((indexAbstraction.getType() != IndexAbstraction.Type.CONCRETE_INDEX


This if statement does look a bit complicated.
Can we instead have two simpler if statements for the valid cases?
One if statement for when indexAbstraction is a data stream and one when indexAbstraction is an alias.
In all other cases we would fail.

I think we also support if it's just a concrete index, right? I was not sure if there were other types I didn't know about so I left it mostly as it was. If there are definitely just those 3 though (alias, data stream, concrete index) I can definitely simplify it.

I think we also support if it's just a concrete index, right?

I thought it was either an alias or data stream. But I might be wrong here.

In at least one place we pass .watches to this method, which I believe is a concrete index (although I could definitely be wrong about that).

I changed it to explicitly look for aliases here, which does simplify things a little.

Thanks, it does.

...plugin/watcher/src/test/java/org/elasticsearch/xpack/watcher/watch/WatchStoreUtilsTests.java

…github.com:masseyke/elasticsearch into fix/watcher-validation-fails-with-multiple-indices

masseyke · 2022-03-31T19:56:01Z

@elasticmachine update branch

…-indices

martijnvg

👍 LGTM (pending successful build)

Failure occurred due to error message change:

java.lang.AssertionError: |  
-- | --
  | Expected: is "Alias [.triggered_watches] points to more than one index" |  
  | but: was "Alias [.triggered_watches] points to 2 indices, and does not have a designated write index" |  
  | at __randomizedtesting.SeedInfo.seed([8F6C099FB918BA97:988C8EAA4DCC424]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.watcher.execution.TriggeredWatchStoreTests.testLoadingFailsWithTwoAliases(TriggeredWatchStoreTests.java:391)

(and I think bwc failures should be fixed by merging in master)

…github.com:masseyke/elasticsearch into fix/watcher-validation-fails-with-multiple-indices

masseyke · 2022-04-01T14:47:07Z

@elasticmachine update branch

…-indices

…than one index (elastic#85507) This commit fixes a validation bug that prevented watcher from starting if the .watcher-history-16 data stream is pointing to more than one index. Before 8.0, .watcher-history-16 data had been an alias and had never pointed to more than one index, so this had not been a problem.

elasticsearchmachine · 2022-04-01T15:49:23Z

💚 Backport successful

Status	Branch	Result
✅	8.2

…than one index (#85507) (#85631) This commit fixes a validation bug that prevented watcher from starting if the .watcher-history-16 data stream is pointing to more than one index. Before 8.0, .watcher-history-16 data had been an alias and had never pointed to more than one index, so this had not been a problem.

…than one index (elastic#85507) This commit fixes a validation bug that prevented watcher from starting if the .watcher-history-16 data stream is pointing to more than one index. Before 8.0, .watcher-history-16 data had been an alias and had never pointed to more than one index, so this had not been a problem.

…than one index (#85507) (#85755) * Avoiding watcher validation errors when a data stream points to more than one index (#85507) This commit fixes a validation bug that prevented watcher from starting if the .watcher-history-16 data stream is pointing to more than one index. Before 8.0, .watcher-history-16 data had been an alias and had never pointed to more than one index, so this had not been a problem. * Fixing test for backport

Avoiding watcher validation errors when a data stream points to more …

1254c23

…than one index

masseyke added :Data Management/Watcher auto-backport-and-merge v8.3.0 v8.2.1 labels Mar 30, 2022

elasticmachine added the Team:Data Management Meta label for data/management team label Mar 30, 2022

Adding unit tests

c59aac6

masseyke added the >bug label Mar 30, 2022

Update docs/changelog/85507.yaml

5239579

masseyke requested a review from martijnvg March 30, 2022 20:42

martijnvg reviewed Mar 31, 2022

View reviewed changes

masseyke added 3 commits March 31, 2022 14:21

code review feedback

c5b060c

code review feedback

67d4494

Merge branch 'fix/watcher-validation-fails-with-multiple-indices' of …

5cfcb46

…github.com:masseyke/elasticsearch into fix/watcher-validation-fails-with-multiple-indices

masseyke requested a review from martijnvg March 31, 2022 19:35

Merge branch 'master' into fix/watcher-validation-fails-with-multiple…

9d29b5b

…-indices

martijnvg approved these changes Apr 1, 2022

View reviewed changes

masseyke added 2 commits April 1, 2022 09:46

fixing unit test

0501da9

Merge branch 'fix/watcher-validation-fails-with-multiple-indices' of …

1a9322a

…github.com:masseyke/elasticsearch into fix/watcher-validation-fails-with-multiple-indices

Merge branch 'master' into fix/watcher-validation-fails-with-multiple…

ba3cd45

…-indices

masseyke merged commit 40fc3a5 into elastic:master Apr 1, 2022

masseyke deleted the fix/watcher-validation-fails-with-multiple-indices branch April 1, 2022 15:48

masseyke mentioned this pull request Apr 1, 2022

[8.2] Avoiding watcher validation errors when a data stream points to more than one index (#85507) #85631

Merged

williamrandolph mentioned this pull request Apr 1, 2022

Log at WARN level for Watcher cluster state validation errors #85632

Merged

jakelandis added v8.2.0 backport pending v8.1.3 and removed v8.2.1 labels Apr 7, 2022

masseyke removed the backport pending label Apr 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Avoiding watcher validation errors when a data stream points to more than one index #85507

Avoiding watcher validation errors when a data stream points to more than one index #85507

Uh oh!

masseyke commented Mar 30, 2022 •

edited

Loading

Uh oh!

elasticmachine commented Mar 30, 2022

Uh oh!

elasticsearchmachine commented Mar 30, 2022

Uh oh!

martijnvg left a comment

Uh oh!

martijnvg Mar 31, 2022

Uh oh!

masseyke Mar 31, 2022

Uh oh!

martijnvg Mar 31, 2022

Uh oh!

masseyke Mar 31, 2022

Uh oh!

masseyke Mar 31, 2022

Uh oh!

martijnvg Apr 1, 2022

Uh oh!

Uh oh!

Uh oh!

masseyke commented Mar 31, 2022

Uh oh!

martijnvg left a comment •

edited

Loading

Uh oh!

masseyke commented Apr 1, 2022

Uh oh!

elasticsearchmachine commented Apr 1, 2022

Uh oh!

Uh oh!

Avoiding watcher validation errors when a data stream points to more than one index #85507

Avoiding watcher validation errors when a data stream points to more than one index #85507

Uh oh!

Conversation

masseyke commented Mar 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 30, 2022

Uh oh!

elasticsearchmachine commented Mar 30, 2022

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 31, 2022

Choose a reason for hiding this comment

Uh oh!

masseyke Mar 31, 2022

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 31, 2022

Choose a reason for hiding this comment

Uh oh!

masseyke Mar 31, 2022

Choose a reason for hiding this comment

Uh oh!

masseyke Mar 31, 2022

Choose a reason for hiding this comment

Uh oh!

martijnvg Apr 1, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

masseyke commented Mar 31, 2022

Uh oh!

martijnvg left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masseyke commented Apr 1, 2022

Uh oh!

elasticsearchmachine commented Apr 1, 2022

💚 Backport successful

Uh oh!

Uh oh!

masseyke commented Mar 30, 2022 •

edited

Loading

martijnvg left a comment •

edited

Loading