hadoop3高可用集群搭建异常(Can‘t format the storage directory because the current directory is not empty)

最新推荐文章于 2024-12-24 19:31:36 发布

qq_21480329

最新推荐文章于 2024-12-24 19:31:36 发布

阅读量6.7k

点赞数

CC 4.0 BY-SA版权

文章标签： hdfs hadoop

本文链接：https://blog.csdn.net/qq_21480329/article/details/111991547

在配置Hadoop HA集群时，由于在node01、node02、node03上启动JournalNode服务并尝试格式化NameNode时遇到异常。错误信息显示JournalNode目录处于不一致状态，因目录不为空无法格式化。问题根源在于hdfs-site.xml配置文件中dfs.namenode.shared.edits.dir配置错误，误将node02的地址重复写入。修正配置为正确包含node01、node02、node03的地址后，停止并重新启动JournalNode服务，再执行NameNode格式化操作，问题得到解决，成功完成格式化。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

集群环境

4台虚拟机: centos7

hadoop3.3.0

jdk-8u131

zookeeper: 3.6.2

1.hdfs --daemon start journalnode

node01, node02,node03执行正确。为什么node01, node02,node03是因为把这三个节点作为journalnode。

2.node01执行hdfs namenode -format异常如下：

org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:305)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.format(QuorumJournalManager.java:264)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.formatNonFileJournals(FSEditLog.java:429)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:189)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1271)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1821)
2020-12-29 15:07:02,479 ERROR namenode.NameNode: Failed to start namenode.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

        at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:305)
        at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.format(QuorumJournalManager.java:264)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.formatNonFileJournals(FSEditLog.java:429)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:189)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1271)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1821)
2020-12-29 15:07:02,487 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not format one or more JournalNodes. 1 exceptions thrown:
192.168.2.104:8485: Directory /var/sxt/hadoop/ha/journalnode/mycluster is in an inconsistent state: Can't format the storage directory because the current directory is not empty.
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.checkEmptyCurrent(Storage.java:622)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:706)
        at org.apache.hadoop.hdfs.qjournal.server.JNStorage.format(JNStorage.java:219)
        at org.apache.hadoop.hdfs.qjournal.server.Journal.format(Journal.java:257)
        at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.format(JournalNodeRpcServer.java:183)
        at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.format(QJournalProtocolServerSideTranslatorPB.java:150)
        at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:31922)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:532)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1020)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:948)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2952)

2020-12-29 15:07:02,492 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node01/192.168.2.104
************************************************************/
[root@node01 hadoop]#

解决方案，经过排查发现，hdfs-site.xml配置错误导致，错误配置如下：

dfs.namenode.shared.edits.dir qjournal://node01:8485; node02:8485; node02:8485/mycluster

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
</property>

修改配置后，在node01, node02,node03执行hdfs --daemon stop journalnode停止journalnode

在这三个节点再次执行hdfs --daemon start journalnode，在执行hdfs namenode -format不在报错，成功格式化。