.watches index mapping explosion when creating complex watches or frequently changing them

[lukewhiting]

Description

The .watches system index can exceed the index.mapping.total_fields.limit (default: 1000) when users have many watches with complex metadata fields or when watches are frequently created, updated, and deleted over time. This prevents users from creating or updating new watches.

Problem

Users are unable to save new watchers when the .watches system index reaches the field limit. The error message presented is:

Limit of total fields [1000] has been exceeded

Key issues:

• Users cannot increase the field limit on .watches index does to hard coded restriction on updating index settings for system indices
• The index.mapping.total_fields.ignore_dynamic_beyond_limit setting cannot be used as all watcher fields are considered important
• Mapping explosion appears to occur from accumulated metadata fields over time, especially when watches are frequently modified

Reproduction

1. Create/update/delete many watches over time with varying metadata fields
2. The .watches index accumulates field mappings that eventually exceed 1000 fields
3. Attempt to create or update a watcher
4. Observe error: "Limit of total fields [1000] has been exceeded"

Impact

• Users cannot create new watches
• Users cannot update existing watches

Expected Behavior

The .watches index should not have a mapping explosion over time

Current Workaround

1. Export all watches via API
2. Stop watcher service
3. Delete .watches index to clean up mappings
4. Re-add watches (automatically recreates .watches index)
5. Restart watcher service

This workaround requires downtime of watcher and is not sustainable long-term as the field's will just grow again over time.

Proposed Solution

Update the mappings of the .watches system index to prevent mapping explosion in the same way that was previously done for .watcher-history-* indices. The mappings should be more restrictive to prevent dynamic field creation from metadata or other fields that can vary significantly between watches.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)