Skip to content

HDDS-10374. Make container scanner generate merkle trees during the scan #7490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 63 commits into
base: HDDS-10239-container-reconciliation
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
a8b8dbc
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
errose28 Nov 22, 2024
999a913
Add code to build and write the tree from the data scanners
errose28 Nov 22, 2024
b0d1ba9
Update todo in acceptance test
errose28 Nov 25, 2024
382bce2
Add unit tests for tree generation by scanners based on container state
errose28 Nov 25, 2024
28b1889
Add initial (failing) unit test for KeyValueContaienrCheck
errose28 Nov 26, 2024
dc182e8
Update container data checksum when building the tree
errose28 Nov 26, 2024
a3401a9
Fix handling of fully truncated block of 0 size
errose28 Jan 7, 2025
a25d44d
Add unit tests for new addBlock method in tree
errose28 Jan 7, 2025
7550a3c
Test that SCM gets a checksum with the container report
errose28 Jan 7, 2025
847f8d8
Add (failing) tests that SCM sees updated checksums
errose28 Jan 7, 2025
452c294
Update acceptance test
errose28 Jan 8, 2025
dc45eca
Add javadoc for tree generation from metadata
errose28 Jan 8, 2025
1cb291f
Data integration tests passing
errose28 Jan 8, 2025
d6b21d2
Don't generate tree from metadata for unhealthy container
errose28 Jan 9, 2025
2a2dbbd
Checkstyle
errose28 Jan 9, 2025
c9a077c
Marking container unhealthy should not write a merkle tree (test fix)
errose28 Jan 9, 2025
0bbbdc5
Checkstyle
errose28 Jan 9, 2025
7b971a9
Address review comments
errose28 Jan 13, 2025
15d6848
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
errose28 Apr 11, 2025
0989881
Initial use of on demand scan in TestKeyValueHandler
errose28 Apr 11, 2025
834be96
Make on-demand scanner a normal instance
errose28 Apr 15, 2025
e73757e
Register on-demand scan callback in ContainerSet
errose28 Apr 15, 2025
f0d8efe
Migrate scanContainer usage in prod code
errose28 Apr 15, 2025
4cb054c
Switch terminology from error to scan. Add existence checks
errose28 Apr 15, 2025
8abedb6
Update tests
errose28 Apr 15, 2025
577a075
Add unit test for ContainerSet
errose28 Apr 16, 2025
4c8d843
Checkstyle
errose28 Apr 16, 2025
0bd4127
Improve comments and test
errose28 Apr 16, 2025
61fae12
Merge branch 'non-static-on-demand-scan' into HDDS-10374-scanner-buil…
errose28 Apr 16, 2025
61f30f3
WIP migrate reconciliation unit tests
errose28 Apr 17, 2025
192eb7b
Most tests passing
errose28 Apr 23, 2025
0cf79f6
Improve logging in test and prod code
errose28 Apr 28, 2025
8b30f54
Fix tree tracking during reconcile process
errose28 Apr 28, 2025
9c74f4b
Use mixin to standardize scanner operations, log checksum changes in …
errose28 Apr 29, 2025
d550669
Logging improvements
errose28 Apr 29, 2025
97e02ea
Add checksum validation, generate readable data
errose28 Apr 30, 2025
22b41b8
Use tree writer between peer updates. All tests pass
errose28 May 5, 2025
f49a9dd
Wait for on-demand scans to complete in test
errose28 May 5, 2025
f5d4dbf
Improve char data generation, reset scan metrics
errose28 May 5, 2025
1140c90
Update test name
errose28 May 5, 2025
e0aa7cb
Checkstyle
errose28 May 5, 2025
62d7794
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
errose28 May 6, 2025
9c3b87c
Merge branch 'reconcile-unit-test-framework' into HDDS-10374-scanner-…
errose28 May 6, 2025
9322b4a
Fix TODOs dependent on this patch
errose28 May 13, 2025
9b75957
Rename container scan helper
errose28 May 13, 2025
f615275
Add comment on failure type
errose28 May 13, 2025
dadc829
Fix checkstyle unique to this PR
errose28 May 13, 2025
076a82e
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
errose28 May 14, 2025
cc55527
Fix sending ICR when only checksum changes (pending test)
errose28 May 14, 2025
35879b4
Updates after reviewing diff
errose28 May 14, 2025
1ab8c14
Add unit test for KeyValueHandler#updateContainerChecksum
errose28 May 14, 2025
6c8be07
Improve and update scanner integration tests
errose28 May 14, 2025
60a1a6e
Add unit tests that checksum update failure does not stop container s…
errose28 May 14, 2025
d035c17
Checkstyle
errose28 May 14, 2025
53336ae
Fix scan gap for unit test
errose28 May 15, 2025
56e7ed4
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
errose28 May 16, 2025
2504638
Fix metadata scan test
errose28 May 16, 2025
4be9992
Update based on review
errose28 May 19, 2025
c0b89dd
pmd
errose28 May 19, 2025
e24a24e
Update ContainerData checksum info after reconcile with each peer
errose28 May 22, 2025
dc27f74
Support bypassing scan gap (tests are failing)
errose28 May 22, 2025
e2974b4
Checkstyle
errose28 May 27, 2025
34b4b9a
Fix scan gap bug. All tests expected to pass
errose28 May 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Rename container scan helper
  • Loading branch information
errose28 committed May 13, 2025
commit 9b75957eb1e6173e97793396faf3971efb16f3cc
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,10 @@

import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.time.Instant;
import java.util.Iterator;
import java.util.Optional;

import org.apache.hadoop.hdfs.util.Canceler;
import org.apache.hadoop.hdfs.util.DataTransferThrottler;
import org.apache.hadoop.ozone.container.checksum.ContainerChecksumTreeManager;
import org.apache.hadoop.ozone.container.common.impl.ContainerData;
import org.apache.hadoop.ozone.container.common.interfaces.Container;
import org.apache.hadoop.ozone.container.common.volume.HddsVolume;
import org.slf4j.Logger;
Expand All @@ -48,7 +45,7 @@ public class BackgroundContainerDataScanner extends
private final Canceler canceler;
private static final String NAME_FORMAT = "ContainerDataScanner(%s)";
private final ContainerDataScannerMetrics metrics;
private final ContainerScannerMixin scannerMixin;
private final ContainerScanHelper scannerMixin;

public BackgroundContainerDataScanner(ContainerScannerConfiguration conf,
ContainerController controller,
Expand All @@ -60,7 +57,7 @@ public BackgroundContainerDataScanner(ContainerScannerConfiguration conf,
canceler = new Canceler();
this.metrics = ContainerDataScannerMetrics.create(volume.toString());
this.metrics.setStorageDirectory(volume.toString());
this.scannerMixin = new ContainerScannerMixin(LOG, controller, metrics, conf);
this.scannerMixin = new ContainerScanHelper(LOG, controller, metrics, conf);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ public class BackgroundContainerMetadataScanner extends
LoggerFactory.getLogger(BackgroundContainerMetadataScanner.class);
private final ContainerMetadataScannerMetrics metrics;
private final ContainerController controller;
private final ContainerScannerMixin scannerMixin;
private final ContainerScanHelper scannerMixin;

public BackgroundContainerMetadataScanner(ContainerScannerConfiguration conf,
ContainerController controller) {
super("ContainerMetadataScanner", conf.getMetadataScanInterval());
this.controller = controller;
this.metrics = ContainerMetadataScannerMetrics.create();
this.scannerMixin = new ContainerScannerMixin(LOG, controller, metrics, conf);
this.scannerMixin = new ContainerScanHelper(LOG, controller, metrics, conf);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,33 +17,29 @@

package org.apache.hadoop.ozone.container.ozoneimpl;

import static org.apache.hadoop.hdds.HddsUtils.checksumToString;

import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
import org.apache.hadoop.hdfs.util.Canceler;
import org.apache.hadoop.hdfs.util.DataTransferThrottler;
import org.apache.hadoop.ozone.container.checksum.ContainerChecksumTreeManager;
import org.apache.hadoop.ozone.container.common.impl.ContainerData;
import org.apache.hadoop.ozone.container.common.interfaces.Container;
import org.apache.hadoop.ozone.container.common.interfaces.ScanResult;
import org.apache.hadoop.ozone.container.common.utils.ContainerLogger;
import org.apache.hadoop.ozone.container.common.volume.HddsVolume;
import org.slf4j.Logger;

/**
* Mixin to handle common data and metadata scan operations among background and on-demand scanners.
*/
public class ContainerScannerMixin {
public class ContainerScanHelper {
private final Logger log;
private final ContainerController controller;
private final AbstractContainerScannerMetrics metrics;
private final long minScanGap;

public ContainerScannerMixin(Logger log, ContainerController controller,
AbstractContainerScannerMetrics metrics, ContainerScannerConfiguration conf) {
public ContainerScanHelper(Logger log, ContainerController controller,
AbstractContainerScannerMetrics metrics, ContainerScannerConfiguration conf) {
this.log = log;
this.controller = controller;
this.metrics = metrics;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import java.util.concurrent.TimeUnit;
import org.apache.hadoop.hdfs.util.Canceler;
import org.apache.hadoop.hdfs.util.DataTransferThrottler;
import org.apache.hadoop.ozone.container.checksum.ContainerChecksumTreeManager;
import org.apache.hadoop.ozone.container.common.interfaces.Container;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -44,7 +43,7 @@ public final class OnDemandContainerDataScanner {
private final ConcurrentHashMap
.KeySetView<Long, Boolean> containerRescheduleCheckSet;
private final OnDemandScannerMetrics metrics;
private final ContainerScannerMixin scannerMixin;
private final ContainerScanHelper scannerMixin;

public OnDemandContainerDataScanner(
ContainerScannerConfiguration conf, ContainerController controller) {
Expand All @@ -54,7 +53,7 @@ public OnDemandContainerDataScanner(
metrics = OnDemandScannerMetrics.create();
scanExecutor = Executors.newSingleThreadExecutor();
containerRescheduleCheckSet = ConcurrentHashMap.newKeySet();
this.scannerMixin = new ContainerScannerMixin(LOG, controller, metrics, conf);
this.scannerMixin = new ContainerScanHelper(LOG, controller, metrics, conf);
}

public Optional<Future<?>> scanContainer(Container<?> container) {
Expand Down