Skip to content

HDDS-13026. KeyDeletingService should also delete RenameEntries #8447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
May 28, 2025

Conversation

swamirishi
Copy link
Contributor

What changes were proposed in this pull request?

Currently the entry from rename entries are never removed from snapshot renamed table which can lead to a bloat in the table size. Key deleting service should use ReclaimableRenameEntry to remove entries which don't even point to any object in the previous snapshot. This can be done as part of the snapshot deep cleaning and AOS garbage collection service

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13026

How was this patch tested?

Additional Unit tests

swamirishi added 14 commits May 11, 2025 20:12
… each and every ratis request

Change-Id: I5c29c572df9d2240b1d58fbc88826eb5ed8ad881
…shotInfo

Change-Id: I16d2881af973799773335343debd856cb763f72b
Change-Id: I67d685aff12daaf98cf6a8eb5f6a5750c4cf6394
Change-Id: I4215cacc0f0486a7aa46b60971149483028bed38
Change-Id: I7a05f30d0a58538708e5acdbce36d836b6a5542e
Change-Id: I5a9e9dd3cc74e185b1f0745af7f2fee1026d125a

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDeletingService.java
Change-Id: I86c0adae571debf1e35e86660beb8c6a62d8b6fc
Change-Id: I403f238e8fd1178540da893f73a42aa2223572ad
Change-Id: Id3202dbe7a5a71c43526e80af34b18dd7b2ed66e
Change-Id: I17c1771c7630292fded0eebfbe6ee14f97798506
Change-Id: I03e67781351ed3a18666b32a12109141a5a4a34d
Change-Id: I1712f2dd4138a1f679eb4e4676b3d5ac41c80583
Change-Id: Id4a53d3eb8f23031ef517b3793dc73b067b11d62
Change-Id: I99896a697c523d6c174a06e66935a4be3b2bd541
@swamirishi swamirishi requested review from smengcl and jojochuang May 14, 2025 09:50
@swamirishi swamirishi added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label May 14, 2025
@swamirishi swamirishi requested a review from sadanand48 May 15, 2025 14:43
Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huge code piece. need time to review.

@@ -140,6 +140,7 @@ private void updateSnapshotInfoAndCache(SnapshotInfo snapInfo, OmMetadataManager
// current snapshot is deleted. We can potentially
// reclaim more keys in the next snapshot.
snapInfo.setDeepClean(false);
snapInfo.setDeepCleanedDeletedDir(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is part of #8451?

Comment on lines 195 to 199
/**
*
* @param currentSnapshotInfo if null, deleted directories in AOS should be processed.
* @param keyManager KeyManager of the underlying store.
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/**
 * Scans the OM database for keys or rename entries that are eligible for deletion
 * in the given store (either the active store or a specific snapshot).
 * 
 * It collects a batch of eligible keys and/or renamed entries up to the specified
 * remainNum limit, submits deletion requests to SCM, and, on successful confirmation,
 * removes the entries from the metadata store. The method also updates snapshot
 * properties related to reclaimed space and deep cleaning flags as required.
 *
 * @param currentSnapshotInfo If non-null, processes deleted keys for a specific snapshot; 
 *                            if null, processes for the active object store.
 * @param keyManager          The KeyManager instance for accessing keys in the underlying store.
 * @param remainNum           The maximum number of entries to process in this invocation.
 * @return The number of entries remaining to be processed after this method runs.
 * @throws IOException If any error occurs during OM DB or SCM operations.
 */

* @param currentSnapshotInfo if null, deleted directories in AOS should be processed.
* @param keyManager KeyManager of the underlying store.
*/
private int processDeletedKeysForStore(SnapshotInfo currentSnapshotInfo, KeyManager keyManager,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method appears to be extracted out of the below call(). But why is it much longer. Can it be broken up into smaller pieces.

@jojochuang
Copy link
Contributor

also is there a test to verify that rename entries are removed after key deletion?

Change-Id: I76d096c4176bbf5508b6f75160c3524c1a04c5f0

# Conflicts:
#	hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLock.java
#	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDirectoryCleaningService.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java
Change-Id: Ibfcfe5d817ac6ac83f76abe3af08b5794bbb4b3e
Change-Id: I8137e7f06014951719dbf8736124f261193f2143

# Conflicts:
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyDeletingService.java
#	hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyDeletingService.java
Change-Id: I7af63cb80766fc83ec75ef9f7ad0ca8d299caa7a
Change-Id: Ib4c0199f50e5efb8f40b4e5c897d4b33acdf9d62
Change-Id: I67ca2fea367627a7a0ad97312ae450c74fabe95b
@swamirishi
Copy link
Contributor Author

huge code piece. need time to review.

@jojochuang this pr size should have dropped significantly given that #8450 is merged

Change-Id: Ie9efcc5f15cee9d58cfcea7e373882c792cdd96b
Change-Id: I0f7cafce55e87884a0eb9fde3c5e15004b4cdb71
@swamirishi
Copy link
Contributor Author

also is there a test to verify that rename entries are removed after key deletion?

Done adding a test case.

Copy link
Contributor

@jojochuang jojochuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

Optional, but we could consider adding a metric for removed renamed table entries. metadataManager.countRowsInTable(snapshotRenamedTable) should only be used in tests not used in production metrics.

Change-Id: I046604c6d5e7c049c99f6e65010a1a40dfbda965
Change-Id: Ifd2c97c3d932fb1335eed97b4535c0b6256add4b
@swamirishi swamirishi marked this pull request as ready for review May 27, 2025 17:43
@smengcl smengcl changed the title HDDS-13026. KeyDeleting service should also delete RenameEntries HDDS-13026. KeyDeletingService should also delete RenameEntries May 27, 2025
Comment on lines 281 to 282
public static String addRenamedEntryToTable(long trxnLogIndex, String volumeName, String bucketName, String key,
OMMetadataManager omMetadataManager) throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: alignment is off

Suggested change
public static String addRenamedEntryToTable(long trxnLogIndex, String volumeName, String bucketName, String key,
OMMetadataManager omMetadataManager) throws Exception {
public static String addRenamedEntryToTable(long trxnLogIndex, String volumeName, String bucketName, String key,
OMMetadataManager omMetadataManager) throws Exception {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Change-Id: I16f7492a085ae56d6ce6402ee5fa2e102636d4bf
@swamirishi
Copy link
Contributor Author

Thank you for reviewing the patch @jojochuang & @smengcl

@swamirishi swamirishi merged commit fd3d70c into apache:master May 28, 2025
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
snapshot https://issues.apache.org/jira/browse/HDDS-6517
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants