HDDS-12078. Improve container reconciliation CLIs #7944

errose28 · 2025-02-22T00:08:11Z

What changes were proposed in this pull request?

Summary

Currently ozone admin container reconcile is used to trigger reconciliation for a specific container, but the resulting checksums are only shown in ozone admin container info --json which can be verbose and usually requires jq filtering to get the desired information.

This PR adds new functionality to the ozone admin container reconcile command:

Multiple container IDs can be passed: ozone admin container reconcile 1 2 3 4
IDs can be read from stdin, similar to ozone admin container info: cat <id-file> | ozone admin container reconcile -
Either of the above options work with a --status flag that prints replica and checksum information instead of sending reconcile commands: ozone admin container reconcile --status 1 2 3 4
- In the status output, a replicasMatch field is set to true or false on the client side, which can be used to quickly filter out containers with mismatched replicas.

No proto changes are made. The same ScmClient#getContainer and ScmClient#getContainerReplicas methods used by ozone admin container info are re-used for ozone admin container reconcile --status. This command just provides a more filtered view of the results specific to reconciliation and adds the replicasMatch field.

Examples

Querying status of one container:

$ ozone admin container reconcile --status 1               
[ {
  "containerID" : 1,
  "state" : "CLOSING",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Querying the status of multiple containers (json list output):

sh-4.4$ ozone admin container reconcile --status 1  3             
[ {
  "containerID" : 1,
  "state" : "CLOSED",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  } ]
}, {
  "containerID" : 3,
  "state" : "OPEN",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Given a list of container IDs in a file, output the IDs of all the ones with mismatched replicas:

$ cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = false) | .containerID'
1
3

Same as above, but now send the IDs of mismatched containers back in for reconciliation (TODO clean up duplicate output):

cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = true) | .containerID' | ozone admin container reconcile -
Reconciliation has been triggered for container 1
Use "ozone admin container reconcile --status 1" to see the checksums of each container replica
Reconciliation has been triggered for container 3
Use "ozone admin container reconcile --status 3" to see the checksums of each container replica

What is the link to the Apache JIRA

HDDS-12078

How was this patch tested?

Tested manually in docker
Unit tests (WIP)
Update existing acceptance tests (WIP)

…prove-cli * HDDS-10239-container-reconciliation: (113 commits) HDDS-12361. Mark testGetSnapshotDiffReportJob as flaky HDDS-12375. Random object created and used only once (apache#7933) HDDS-12335. Fix ozone admin namespace summary to give complete output (apache#7908) HDDS-12364. Require Override annotation for overridden methods (apache#7923) HDDS-11530. Support listMultipartUploads max uploads and markers (apache#7817) HDDS-11867. Remove code paths for non-Ratis SCM. (apache#7911) HDDS-12363. Add import options to IntelliJ IDEA style settings (apache#7921) HDDS-12215. Mark testContainerStateMachineRestartWithDNChangePipeline as flaky HDDS-12331. BlockOutputStream.failedServers is not thread-safe (apache#7885) HDDS-12188. Move server-only upgrade classes from hdds-common to hdds-server-framework (apache#7903) HDDS-12362. Remove temporary checkstyle suppression file (apache#7920) HDDS-12343. Fix spotbugs warnings in Recon (apache#7902) HDDS-12286. Fix license headers and imports for ozone-tools (apache#7919) HDDS-12284. Fix license headers and imports for ozone-s3-secret-store (apache#7917) HDDS-12275. Fix license headers and imports for ozone-integration-test (apache#7904) HDDS-12164. Rename and deprecate DFSConfigKeysLegacy config keys (apache#7803) HDDS-12283. Fix license headers and imports for ozone-recon-codegen (apache#7916) HDDS-12330. Convert Volume, Bucket and Key count to use comma separated numbers (apache#7881) HDDS-12281. Fix license headers and imports for ozone-filesystem-hadoop3 (apache#7913) HDDS-12285. Fix license headers and imports for ozone-s3gateway (apache#7918) ... Conflicts: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaInfo.java hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/server/JsonUtils.java hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java

kerneltime

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

kerneltime · 2025-02-25T02:07:18Z

...p-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java

+  @CommandLine.Parameters(description = "One or more container IDs separated by spaces. " +
+      "To read from stdin, specify '-' and supply the container IDs " +
+      "separated by newlines.",


Not for this PR, but we can look into brace expansion so that the CLI can specify a range {1...1000}. This can ease the scanning for status via CLI across many containers.

It's not something Ozone CLI needs to implement. Shells like bash and zsh do that already with any command:

$ echo {1..5} 1 2 3 4 5

Yeah having a "recipes" section or similar in our CLI docs would help to call out tricks like this one and some of the jq examples given in the PR description. Probably too verbose to put in our one-ish line CLI help messages and we don't have man pages, so I think website docs would be the place to add such details.

kerneltime · 2025-02-25T03:28:40Z

...p-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java

+        // processing the remaining containers.
+        int invalidCount = 0;
+        while (idIterator.hasNext()) {
+          long containerID = Long.parseLong(idIterator.next());


Might be useful to catch NumberFormatException and print a more friendly error message.

Good catch. Forgot this wasn't being handled as a long automatically by picocli. Will add a test for this.

kerneltime · 2025-02-25T03:30:30Z

...p-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java

+    System.out.println("Use \"ozone admin container reconcile --status " + containerID + "\" to see the checksums of " +
+        "each container replica");


We don't have a good way to indicate when reconciliation is completed... Maybe we add a reconciliation count as a metric to the container info?

Agreed, some sort of feedback that it finished would be good. I was planning to revisit this with the SCM integration. For example, SCM may end up tracking the known in-flight reconciliations and could report the results.

Are you suggesting adding a jmx metric or just a counter in the code? I'm not sure exactly how this work, but I imagine the counter would get persisted on the DN side and SCM would see the updates. Users would then look for the count to increase when they send in a command.

adoroszlai · 2025-02-25T16:53:59Z

...p-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java

+      // Read from stdin.
+      idIterator = new Scanner(System.in, StandardCharsets.UTF_8.name());


We can get rid of code duplicated from container InfoSubcommand after #7970.

Nice. Do you want us to review and merge that one first?

errose28 · 2025-02-26T03:19:58Z

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

Yes I will add robot tests as well. The current tests are unit tests without a mini ozone cluster. The scm client is mocked to return various combinations of replicas that would otherwise be hard to create in a mini ozone or docker cluster.

errose28 added 11 commits February 7, 2025 19:23

Add initial running status command

ef07f83

Add inital untested stdin/container list support

c430fdc

Minor fixes and handle failures triggering reconcile for list

7e2b27a

Initial test cases

9f037c7

WIP test updates

b0d5a91

Basic test passes

eb9c92a

Set up tests to read from args and stdin and stub out all cases

7a0fefe

Checkstyle

fe9dd21

Fix findbugs and rat

9ba8efd

Remove unused visible for testing

02b7dd4

github-actions bot added the container-reconciliation label Feb 22, 2025

errose28 changed the title ~~HDDS-12078 improve cli~~ HDDS-12078. Improve container reconciliation CLIs Feb 22, 2025

errose28 requested review from aswinshakil and kerneltime February 22, 2025 00:10

kerneltime reviewed Feb 25, 2025

View reviewed changes

adoroszlai reviewed Feb 25, 2025

View reviewed changes

errose28 mentioned this pull request Mar 11, 2025

HDDS-12553. ozone admin container list should output real JSON array #8050

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-12078. Improve container reconciliation CLIs #7944

HDDS-12078. Improve container reconciliation CLIs #7944

Uh oh!

errose28 commented Feb 22, 2025 •

edited

Loading

Uh oh!

kerneltime left a comment

Uh oh!

kerneltime Feb 25, 2025

Uh oh!

adoroszlai Feb 25, 2025

Uh oh!

errose28 Feb 26, 2025

Uh oh!

kerneltime Feb 25, 2025

Uh oh!

errose28 Feb 26, 2025

Uh oh!

kerneltime Feb 25, 2025

Uh oh!

errose28 Feb 26, 2025

Uh oh!

adoroszlai Feb 25, 2025

Uh oh!

errose28 Feb 26, 2025

Uh oh!

errose28 commented Feb 26, 2025

Uh oh!

Uh oh!

		System.out.println("Use \"ozone admin container reconcile --status " + containerID + "\" to see the checksums of " +
		"each container replica");

		// Read from stdin.
		idIterator = new Scanner(System.in, StandardCharsets.UTF_8.name());

HDDS-12078. Improve container reconciliation CLIs #7944

Are you sure you want to change the base?

HDDS-12078. Improve container reconciliation CLIs #7944

Uh oh!

Conversation

errose28 commented Feb 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Summary

Examples

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

kerneltime left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

errose28 commented Feb 26, 2025

Uh oh!

Uh oh!

errose28 commented Feb 22, 2025 •

edited

Loading