Force Replication #7791

mosleyit · 2025-02-02T00:37:14Z

mosleyit
Feb 2, 2025

I have a HA Ozone Cluster and something has happened causing some datanodes to not have all the data.

Is there a way to force ozone to replicate or to check everything and repair?

errose28 · 2025-02-03T21:57:38Z

errose28
Feb 3, 2025
Collaborator

Can you elaborate on the issue a more? Are you seeing under-replicated containers? Ozone (SCM specifically) should be replicating those automatically if there are enough datanodes with space available. Some commands you can use to check:

ozone admin datanode list will show the datanodes currently up and running in the cluster
ozone admin container report will provide counts and examples of specific containers that may be in undesirable states like over/under replicated.
ozone admin container info <id> [--json] will give specific info on the replicas of a given container you may have seen referenced from the container report command above.

3 replies

mosleyit Feb 4, 2025
Author

@errose28 Effectively what we found is that through our stress testing of ozone that something happened causing us to only have two copies of the data on 2 of the three nodes. Through stress and failure testing it seems that it lost track of under replicated files.

We are configured with RATIS with a replication factor of three. We found this issue when taking down a datanode and then ensuring that we could still read all the files.

After doing this we found that some files faced errors when downloading. Inspecting further we found that the data only existed on two nodes yest did not show anything as under replicated.

When inspecting the container we received this error:

ozone@nas-atx:/opt/ozone/bin$ ./ozone debug container inspect
{
  "containerID": 1,
  "schemaVersion": "3",
  "containerState": "CLOSED",
  "currentDatanodeID": "e5d028ce-f2af-468a-b7e3-f41da849f7fa",
  "originDatanodeID": "e5d028ce-f2af-468a-b7e3-f41da849f7fa",
  "dBMetadata": {
    "#BLOCKCOUNT": 1502,
    "#BYTESUSED": 2903970822,
    "#PENDINGDELETEBLOCKCOUNT": null,
    "#delTX": null,
    "#BCSID": 95820
  },
  "aggregates": {
    "blockCount": 1502,
    "usedBytes": 2903970822,
    "pendingDeleteBlocks": 0,
    "pendingDeleteBytes": 0
  },
  "chunksDirectory": {
    "path": "/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/1/chunks",
    "present": true,
    "fileCount": 0
  },
  "correct": true,
  "errors": []
}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007990d2ffd552, pid=3847257, tid=3847320
#
# JRE version: OpenJDK Runtime Environment (11.0.25+9) (build 11.0.25+9-post-Ubuntu-1ubuntu124.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.25+9-post-Ubuntu-1ubuntu124.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [librocksdbjni16566112592457274601.so+0x5fd552]  rocksdb::ColumnFamilyOptions::Dump(rocksdb::Logger*) const+0x382
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /opt/ozone/bin/core.3847257)
#
# An error report file with more information is saved as:
# /opt/ozone/bin/hs_err_pid3847257.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

We also went further once we found a file that was missing and inspected the locations for chunks. We found:

ozone@nas-atx:/opt/ozone/bin$ ./ozone debug chunkinfo s3v/wordcount/2019/04/26/in-s-5082742415-20190426-221527-1556316927.47.wav
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.gson.internal.reflect.ReflectionHelper (file:/opt/ozone/share/ozone/lib/gson-2.9.0.jar) to field java.lang.ThreadLocal.threadLocalHashCode
WARNING: Please consider reporting this to the maintainers of com.google.gson.internal.reflect.ReflectionHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
{
  "KeyLocations": [
    [
      {
        "Datanode-HostName": "nas.sat.somewhere.co",
        "Datanode-IP": "192.168.11.31",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      },
      {
        "Datanode-HostName": "nas.atx.somewhere.co",
        "Datanode-IP": "192.168.11.30",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      },
      {
        "Datanode-HostName": "nas.dfw.somewhere.co",
        "Datanode-IP": "192.168.68.15",
        "Container-ID": 20,
        "Block-ID": 115816896921600335,
        "Locations": {
          "files": [
            "/data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block"
          ],
          "pipelineID": "f063c805-34b8-4852-810d-108bd84c6f23"
        }
      }
    ]
  ]
}
ozone@nas-atx:/opt/ozone/bin$

And found that chunks were missing:

jeffmosley@nas-dfw:~$ ls -l /data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
ls: cannot access '/data/ozone/dn/nvme5n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block': No such file or directory

ozone@nas-sat:/opt/ozone/bin$ ls -l /data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
-rw-r--r-- 1 ozone ozone 118124 Jan 27 00:29 /data/ozone/dn/nvme4n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block

ozone@nas-atx:/opt/ozone/bin$ ls -l /data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block
ls: cannot access '/data/ozone/dn/nvme1n1/hdds/CID-d05e3d06-41d7-4b0a-b39d-eb8852de4e23/current/containerDir0/20/chunks/115816896921600335.block': No such file or directory

Once we realized that this was the case and recon was not seeing this. We started going down the path of forcing replication. We have found that there is no way to force replication that we can find.

We attempted to use the freon tools but it seems they are meant for non production use.

Reviewing further we found that if we decommission a node it will check the replication status of everything. We attempted this and it just does nothing. Our belief is that the reason is it needs three active nodes. Since our testing cluster is only 3 when we take down the datanode it does not have another node to maintain a quorum and fails.

Our concern is that none of this is being reported by recon on container info. We also believe that this happened as we showed 1000s of under replicated files at one point in time (we did not log them but should have) and assumed that they just magically replicated but at the same time we were stress testing taking nodes out of service bringing them back etc.

In a perfect world it would be nice to have a utility that can iterate over everything and ensure replication is consistent.

Any feedback is MUCH appreciated and we are setting up a second cluster in order to do further testing.

We are using 1.4.1 binary for reference.

Current ozone-site.xml for reference:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <!-- Basic Configuration -->
    <property>
        <name>ozone.enabled</name>
        <value>true</value>
    </property>
    
    <!-- SCM HA Configuration -->
    <property>
        <name>ozone.scm.service.ids</name>
        <value>scm-ha</value>
    </property>
    
    <property>
        <name>ozone.scm.nodes.scm-ha</name>
        <value>scm1,scm2,scm3</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm1</name>
        <value>scm.atx.somewhere.co</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm2</name>
        <value>scm.sat.somewhere.co</value>
    </property>
    
    <property>
        <name>ozone.scm.address.scm-ha.scm3</name>
        <value>scm.dfw.somewhere.co</value>
    </property>
    
    <!-- Port Configuration -->
    <property>
        <name>ozone.scm.client.port</name>
        <value>9860</value>
    </property>
    
    <property>
        <name>ozone.scm.block.client.port</name>
        <value>9863</value>
    </property>
    
    <property>
        <name>ozone.scm.datanode.port</name>
        <value>9861</value>
    </property>
    
    <!-- Storage Directories -->
    <property>
        <name>ozone.metadata.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.metadata.dirs</name>
        <value>/data/ozone/scm/meta</value>
    </property>
    <property>
        <name>ozone.scm.db.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.scm.db.dirs</name>
        <value>/data/ozone/scm/db</value>
    </property>
    <property>
        <name>ozone.scm.ha.ratis.storage.dir</name>
        <value>/data/ozone/scm/ratis</value>
    </property>

    <!-- Security Configuration -->
    <property>
        <name>ozone.security.enabled</name>
        <value>false</value>
    </property>
    
    
    <!-- Metrics Configuration -->
    <property>
        <name>ozone.metrics.enabled</name>
        <value>true</value>
    </property>
    
    <property>
        <name>ozone.metrics.prometheus.endpoint</name>
        <value>prometheus.somewhere.co</value>
    </property>
    <!-- OM HA Configuration --> 
    <property>
        <name>ozone.scm.ratis.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>ozone.om.ratis.enable</name>
        <value>true</value>
    </property>
    <property> 
        <name>ozone.om.service.ids</name> 
        <value>om-ha</value> 
    </property> 
    <property> 
        <name>ozone.om.nodes.om-ha</name> 
        <value>om1,om2,om3</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om1</name> 
        <value>om.atx.somewhere.co</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om2</name> 
        <value>om.sat.somewhere.co</value> 
    </property> 
    <property> 
        <name>ozone.om.address.om-ha.om3</name> 
        <value>om.dfw.somewhere.co</value> 
    </property> 
    <property>
        <name>ozone.om.ratis.storage.dir</name>
        <value>/data/ozone/om/ratis</value>
    </property>
    <property>
        <name>ozone.om.ratis.snapshot.dir</name>
        <value>/data/ozone/om/snapshot</value>
    </property>
    <!-- Port Configuration --> 
    <property> 
        <name>ozone.om.rpc-port</name> 
        <value>9862</value> 
    </property> 
    <property> 
        <name>ozone.om.http-port</name> 
        <value>9874</value> 
    </property> 
    <property> 
        <name>ozone.om.https-port</name> 
        <value>9875</value> 
    </property> 
    <property> 
        <name>ozone.om.ha-port</name> 
        <value>9876</value> 
    </property> 
    <!-- Storage Directories --> 
    <property>
        <name>ozone.om.db.dirs.permissions</name>
        <value>750</value>
    </property> 
    <property> 
        <name>ozone.om.db.dirs</name> 
        <value>/data/ozone/om/db</value> 
    </property> 
    <!-- DataNode Configuration --> 
    <property> 
        <name>hdds.datanode.dir</name> 
        <value>/data/ozone/dn/nvme0n1,/data/ozone/dn/nvme1n1,/data/ozone/dn/nvme2n1,/data/ozone/dn/nvme3n1,/data/ozone/dn/nvme4n1,/data/ozone/dn/nvme5n1,/data/ozone/dn/nvme6n1,/data/ozone/dn/nvme7n1,/data/ozone/dn/nvme8n1,/data/ozone/dn/nvme10n1</value>  
    </property> 
    <property> 
        <name>hdds.container.ratis.datanode.storage.dir</name> 
        <value>/data/ozone/dn/ratis</value> 
    </property> 
    <property> 
        <name>ozone.scm.datanode.id.dir</name> 
        <value>/data/ozone/dn/id</value> 
    </property>
    <property> 
        <name>hdds.datanode.container.db.dir</name> 
        <value>/data/ozone/dn/db</value> 
    </property> 
    <property> 
        <name>hdds.datanode.http-address</name> 
        <value>0.0.0.0:9882</value> 
    </property> 
    <property> 
        <name>hdds.datanode.https-address</name> 
        <value>0.0.0.0:9883</value> 
    </property> 
    <!-- Storage Configuration --> 
    <property> 
        <name>hdds.datanode.storage.utilization.critical.threshold</name> 
        <value>0.95</value> 
    </property> 
    <property> 
        <name>hdds.datanode.storage.utilization.warning.threshold</name> 
        <value>0.85</value> 
    </property> 
    <property> 
        <name>ozone.scm.names</name> 
        <value>scm.atx.somewhere.co,scm.sat.somewhere.co,scm.dfw.somewhere.co</value> 
    </property>
        <!-- Recon Configuration -->
    <property>
        <name>ozone.recon.http-address</name>
        <value>0.0.0.0:9888</value>
    </property>
    
    <property>
        <name>ozone.recon.https-address</name>
        <value>0.0.0.0:9889</value>
    </property>    
    <property>
        <name>ozone.recon.db.dir</name>
        <value>/data/ozone/recon/db</value>
    </property>
    <property>
        <name>ozone.recon.db.dirs.permissions</name>
        <value>750</value>
    </property>
    <property>
        <name>ozone.recon.om.db.dir</name>
        <value>/data/ozone/recon/omdb</value>
    </property>
    <property>
        <name>ozone.recon.prometheus.http.endpoint</name>
        <value>http://prometheus.somewhere.co:9090</value>
    </property>
    <property>
        <name>ozone.recon.address</name>
        <value>recon.atx.somewhere.co:9891</value>
    </property>
    <property>
        <name>hdds.container.ratis.enabled</name>
        <value>true</value>
    </property>
</configuration>

NOTE: edited 15 times for formating!
NOTE: and i cannot spell formatting.

ivandika3 Mar 7, 2025
Collaborator

@mosleyit Currently the community is working on https://issues.apache.org/jira/browse/HDDS-12425 to handle issues where data are not replicated to all 3 nodes due to pipeline failure / container close.

Currently during normal writes (using Ratis, not EC), the client will wait until the Raft logs about the WriteChunk and PutBlock are committed in all 3 datanodes (using watch request with ALL_COMMITTED). If the Raft logs are not committed in all 3 datanodes by a certain timeout (default 30s), the client will just need to see the same Raft logs in 2 datanodes (using watch request with MAJORITY_COMMITTED) and exclude the slow datanode for the next block allocation and writes.

However, "committed" doesn't mean that the data has been applied (i.e persisted to disk) yet, it just means that the Datanode "promise" that they will eventually apply the operations in the Raft log (WriteChunk / PutBlock).

The issue the community encountered recently is that some failure conditions, Ozone will try to close the pipeline as soon as possible. However, currently this is implemented by removing the Raft group immediately, regardless whether the "committed" logs have been applied. This can cause issues where the data are not actually replicated to all datanodes (as you've seen in your stress test) or in most cases, some block only contains stale data (BCSID is lower than the other replicas).

One idea is https://issues.apache.org/jira/browse/HDDS-12247 where the client will wait until the block is applied before allocating and writing the next block. This should technically means that the data will be persisted in all 3 DNs once the client block writes return. However, this might cause some increase in latency since some datanodes might be slow to apply the logs. Another idea is apache/ratis#1218 where the Ratis group remove need to wait until all committed logs have been applied, but there are also some edge cases where this doesn't apply.

https://issues.apache.org/jira/browse/HDDS-12425 also will improve the ReplicationManager on how to replicate the replicas so that the up-to-date data are replicated after the container has been CLOSED / QUASI_CLOSED.

siddhantsangwan Mar 12, 2025
Collaborator

@mosleyit can you check what ozone admin container info <container-id> --json shows for the problematic container?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Force Replication #7791

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Force Replication #7791

Uh oh!

mosleyit Feb 2, 2025

Replies: 1 comment · 3 replies

Uh oh!

errose28 Feb 3, 2025 Collaborator

Uh oh!

Uh oh!

mosleyit Feb 4, 2025 Author

Uh oh!

ivandika3 Mar 7, 2025 Collaborator

Uh oh!

siddhantsangwan Mar 12, 2025 Collaborator

mosleyit
Feb 2, 2025

Replies: 1 comment 3 replies

errose28
Feb 3, 2025
Collaborator

mosleyit Feb 4, 2025
Author

ivandika3 Mar 7, 2025
Collaborator

siddhantsangwan Mar 12, 2025
Collaborator