HDDS-11513. All deletion configurations should be configurable without restart

[sarvekshayr]

What changes were proposed in this pull request?

The following deletion related configurations are now dynamically reconfigurable without requiring a restart.
This allows for easy tuning of deletion in response to logs and metrics.

OM:

• ozone.directory.deleting.service.interval
• ozone.thread.number.dir.deletion

DATANODE:

• ozone.block.deleting.service.interval
• ozone.block.deleting.service.timeout

What is the link to the Apache JIRA

HDDS-11513

How was this patch tested?

OM:

• List all OM reconfigurable properties

bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 properties
OM: Node [om:9862] Reconfigurable properties:
ozone.administrators
ozone.directory.deleting.service.interval
ozone.key.deleting.limit.per.task
ozone.om.server.list.max.size
ozone.om.volume.listall.allowed
ozone.readonly.administrators
ozone.thread.number.dir.deletion

• Reconfigured ozone.directory.deleting.service.interval successfully

bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 05:51:56 UTC 2025 and finished at Tue Mar 04 05:51:56 UTC 2025.
SUCCESS: Changed property ozone.directory.deleting.service.interval
        From: "1m"
        To: "2m"

• Reconfigured ozone.thread.number.dir.deletion successfully

bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 06:41:19 UTC 2025 and finished at Tue Mar 04 06:41:19 UTC 2025.
SUCCESS: Changed property ozone.thread.number.dir.deletion
        From: ""
        To: "20"

• Error case when ozone.thread.number.dir.deletion value is negative.

bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 start     
OM: Started reconfiguration task on node [om:9862].
bash-5.1$ ozone admin reconfig --service=OM --address=om:9862 status
OM: Reconfiguring status for node [om:9862]: started at Tue Mar 04 07:07:08 UTC 2025 and finished at Tue Mar 04 07:07:08 UTC 2025.
FAILED: Change property ozone.thread.number.dir.deletion
        From: ""
        To: "-20"
        Error: ozone.thread.number.dir.deletion cannot be negative..

DATANODE:

• List all DN reconfigurable properties

bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 properties
DN: Node [datanode:19864] Reconfigurable properties:
hdds.datanode.block.delete.threads.max
hdds.datanode.block.deleting.limit.per.interval
hdds.datanode.replication.streams.limit
ozone.block.deleting.service.interval
ozone.block.deleting.service.timeout
ozone.block.deleting.service.workers

• Reconfigured ozone.block.deleting.service.interval and ozone.block.deleting.service.timeout successfully

bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 start     
DN: Started reconfiguration task on node [datanode:19864].
bash-5.1$ ozone admin reconfig --service=DATANODE --address=datanode:19864 status
DN: Reconfiguring status for node [datanode:19864]: started at Tue Mar 04 05:50:29 UTC 2025 and finished at Tue Mar 04 05:50:29 UTC 2025.
SUCCESS: Changed property ozone.block.deleting.service.interval
        From: "1m"
        To: "2m"
SUCCESS: Changed property ozone.block.deleting.service.timeout
        From: "300000ms"
        To: "350000ms"

[adoroszlai]

Thanks @sarvekshayr for the patch.

Making properties reconfigurable is a good first step, but the services do not pick up the updated configurations. dirDeletingServiceInterval etc. are only used by the tests.

[adoroszlai]

These should not store the raw String config.

[aryangupta1998]

@adoroszlai, the dirDeletingServiceInterval(ozone.directory.deleting.service.interval) and blockDeletingServiceInterval(ozone.block.deleting.service.interval) are used by the background services to determine the intervals between their execution. These intervals, apart from being used in tests, are configured for the background services that delete directories and blocks. The background service uses scheduleWithFixedDelay to schedule the tasks with the specified intervals.
public void start() { exec.scheduleWithFixedDelay(service, 0, interval, unit); }

[adoroszlai]

BlockDeletingService is created with interval and timeout taken from config:

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java

Lines 245 to 260 in 10945c8

	Duration blockDeletingSvcInterval = dnConf.getBlockDeletionInterval();
	long blockDeletingServiceTimeout = config
	.getTimeDuration(OZONE_BLOCK_DELETING_SERVICE_TIMEOUT,
	OZONE_BLOCK_DELETING_SERVICE_TIMEOUT_DEFAULT,
	TimeUnit.MILLISECONDS);
	int blockDeletingServiceWorkerSize = config
	.getInt(OZONE_BLOCK_DELETING_SERVICE_WORKERS,
	OZONE_BLOCK_DELETING_SERVICE_WORKERS_DEFAULT);
	blockDeletingService =
	new BlockDeletingService(this, blockDeletingSvcInterval.toMillis(),
	blockDeletingServiceTimeout, TimeUnit.MILLISECONDS,
	blockDeletingServiceWorkerSize, config,
	datanodeDetails.threadNamePrefix(),
	context.getParent().getReconfigurationHandler());

They are stored in the base BackgroundService class:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java

Lines 58 to 65 in 10945c8

	public BackgroundService(String serviceName, long interval,
	TimeUnit unit, int threadPoolSize, long serviceTimeout,
	String threadNamePrefix) {
	this.interval = interval;
	this.unit = unit;
	this.serviceName = serviceName;
	this.serviceTimeoutInNanos = TimeDuration.valueOf(serviceTimeout, unit)
	.toLong(TimeUnit.NANOSECONDS);

which schedules the task when started:

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java

Lines 108 to 109 in 10945c8

	public void start() {
	exec.scheduleWithFixedDelay(service, 0, interval, unit);

Reconfiguring does not change values in BackgroundService (they are final), and even updating interval wouldn't change the task already scheduled with fixed delay (it needs to be rescheduled).

[aryangupta1998]

Thanks for the clarification, @adoroszlai. I checked the code, for these configs to be updated, we need to restart the OM. The "ozone om" command starts the key manager, which then initializes the block deleting and directory deleting service with their intervals. So, simply updating these properties without starting the OM again won't be effective here.

[adoroszlai]

simply updating these properties without starting the OM again won't be effective here

That's my point. OM can be tweaked to pick up the config without restart, but the current patch is not enough for that yet.