Skip to content

HDDS-12078. Improve container reconciliation CLIs #7944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: HDDS-10239-container-reconciliation
Choose a base branch
from

Conversation

errose28
Copy link
Contributor

@errose28 errose28 commented Feb 22, 2025

What changes were proposed in this pull request?

Summary

Currently ozone admin container reconcile is used to trigger reconciliation for a specific container, but the resulting checksums are only shown in ozone admin container info --json which can be verbose and usually requires jq filtering to get the desired information.

This PR adds new functionality to the ozone admin container reconcile command:

  • Multiple container IDs can be passed: ozone admin container reconcile 1 2 3 4
  • IDs can be read from stdin, similar to ozone admin container info: cat <id-file> | ozone admin container reconcile -
  • Either of the above options work with a --status flag that prints replica and checksum information instead of sending reconcile commands: ozone admin container reconcile --status 1 2 3 4
    • In the status output, a replicasMatch field is set to true or false on the client side, which can be used to quickly filter out containers with mismatched replicas.

No proto changes are made. The same ScmClient#getContainer and ScmClient#getContainerReplicas methods used by ozone admin container info are re-used for ozone admin container reconcile --status. This command just provides a more filtered view of the results specific to reconciliation and adds the replicasMatch field.

Examples

Querying status of one container:

$ ozone admin container reconcile --status 1               
[ {
  "containerID" : 1,
  "state" : "CLOSING",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Querying the status of multiple containers (json list output):

sh-4.4$ ozone admin container reconcile --status 1  3             
[ {
  "containerID" : 1,
  "state" : "CLOSED",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "CLOSED",
    "dataChecksum" : "0"
  } ]
}, {
  "containerID" : 3,
  "state" : "OPEN",
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "minimumNodes" : 1,
    "replicationType" : "RATIS"
  },
  "replicasMatch" : true,
  "replicas" : [ {
    "datanode" : {
      "hostname" : "ha-dn5-1.ha_net",
      "uuid" : "927a2620-feb9-43f0-bf0c-3be63eae8f46"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn3-1.ha_net",
      "uuid" : "e39fb4dc-c676-4ace-a3f0-d52c8a3a21c8"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  }, {
    "datanode" : {
      "hostname" : "ha-dn4-1.ha_net",
      "uuid" : "cf484945-b60f-4906-9c4c-024ba429783a"
    },
    "state" : "OPEN",
    "dataChecksum" : "0"
  } ]
} ]

Given a list of container IDs in a file, output the IDs of all the ones with mismatched replicas:

$ cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = false) | .containerID'
1
3

Same as above, but now send the IDs of mismatched containers back in for reconciliation (TODO clean up duplicate output):

cat ids.txt | ozone admin container reconcile --status - | jq '.[] | select(.replicasMatch = true) | .containerID' | ozone admin container reconcile -
Reconciliation has been triggered for container 1
Use "ozone admin container reconcile --status 1" to see the checksums of each container replica
Reconciliation has been triggered for container 3
Use "ozone admin container reconcile --status 3" to see the checksums of each container replica

What is the link to the Apache JIRA

HDDS-12078

How was this patch tested?

  • Tested manually in docker
  • Unit tests (WIP)
  • Update existing acceptance tests (WIP)

…prove-cli

* HDDS-10239-container-reconciliation: (113 commits)
  HDDS-12361. Mark testGetSnapshotDiffReportJob as flaky
  HDDS-12375. Random object created and used only once (apache#7933)
  HDDS-12335. Fix ozone admin namespace summary to give complete output (apache#7908)
  HDDS-12364. Require Override annotation for overridden methods (apache#7923)
  HDDS-11530. Support listMultipartUploads max uploads and markers (apache#7817)
  HDDS-11867. Remove code paths for non-Ratis SCM. (apache#7911)
  HDDS-12363. Add import options to IntelliJ IDEA style settings (apache#7921)
  HDDS-12215. Mark testContainerStateMachineRestartWithDNChangePipeline as flaky
  HDDS-12331. BlockOutputStream.failedServers is not thread-safe (apache#7885)
  HDDS-12188. Move server-only upgrade classes from hdds-common to hdds-server-framework (apache#7903)
  HDDS-12362. Remove temporary checkstyle suppression file (apache#7920)
  HDDS-12343. Fix spotbugs warnings in Recon (apache#7902)
  HDDS-12286. Fix license headers and imports for ozone-tools (apache#7919)
  HDDS-12284. Fix license headers and imports for ozone-s3-secret-store (apache#7917)
  HDDS-12275. Fix license headers and imports for ozone-integration-test (apache#7904)
  HDDS-12164. Rename and deprecate DFSConfigKeysLegacy config keys (apache#7803)
  HDDS-12283. Fix license headers and imports for ozone-recon-codegen (apache#7916)
  HDDS-12330. Convert Volume, Bucket and Key count to use comma separated numbers (apache#7881)
  HDDS-12281. Fix license headers and imports for ozone-filesystem-hadoop3 (apache#7913)
  HDDS-12285. Fix license headers and imports for ozone-s3gateway (apache#7918)
  ...

Conflicts:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaInfo.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/server/JsonUtils.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/ReconcileSubcommand.java
@errose28 errose28 changed the title HDDS-12078 improve cli HDDS-12078. Improve container reconciliation CLIs Feb 22, 2025
Copy link
Contributor

@kerneltime kerneltime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

Comment on lines +51 to +53
@CommandLine.Parameters(description = "One or more container IDs separated by spaces. " +
"To read from stdin, specify '-' and supply the container IDs " +
"separated by newlines.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR, but we can look into brace expansion so that the CLI can specify a range {1...1000}. This can ease the scanning for status via CLI across many containers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not something Ozone CLI needs to implement. Shells like bash and zsh do that already with any command:

$ echo {1..5}
1 2 3 4 5

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah having a "recipes" section or similar in our CLI docs would help to call out tricks like this one and some of the jq examples given in the PR description. Probably too verbose to put in our one-ish line CLI help messages and we don't have man pages, so I think website docs would be the place to add such details.

// processing the remaining containers.
int invalidCount = 0;
while (idIterator.hasNext()) {
long containerID = Long.parseLong(idIterator.next());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to catch NumberFormatException and print a more friendly error message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Forgot this wasn't being handled as a long automatically by picocli. Will add a test for this.

Comment on lines +127 to +128
System.out.println("Use \"ozone admin container reconcile --status " + containerID + "\" to see the checksums of " +
"each container replica");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have a good way to indicate when reconciliation is completed... Maybe we add a reconciliation count as a metric to the container info?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, some sort of feedback that it finished would be good. I was planning to revisit this with the SCM integration. For example, SCM may end up tracking the known in-flight reconciliations and could report the results.

Are you suggesting adding a jmx metric or just a counter in the code? I'm not sure exactly how this work, but I imagine the counter would get persisted on the DN side and SCM would see the updates. Users would then look for the count to increase when they send in a command.

Comment on lines +69 to +70
// Read from stdin.
idIterator = new Scanner(System.in, StandardCharsets.UTF_8.name());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can get rid of code duplicated from container InfoSubcommand after #7970.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Do you want us to review and merge that one first?

@errose28
Copy link
Contributor Author

I prefer robot tests over integration tests. Overall seems fine, will go over a few more times.

Yes I will add robot tests as well. The current tests are unit tests without a mini ozone cluster. The scm client is mocked to return various combinations of replicas that would otherwise be hard to create in a mini ozone or docker cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants