hive动态分区过多异常

最新推荐文章于 2023-06-21 23:50:36 发布

原创

最新推荐文章于 2023-06-21 23:50:36 发布 · 2.2k 阅读

1 ·

CC 4.0 BY-SA版权

在执行Hive的动态插入分区SQL时遇到错误，日志显示无法创建新的block并频繁地放弃和排除datanode。问题根源是datanode的xceiverCount超过了最大限制8192。解决方案是减少分区数量或调整datanode的`dfs.datanode.max.xcievers`配置以增加线程数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

hive 执行动态插入分区sql 报如下错误：

2019-05-28 16:04:40,381 WARN [Thread-1018] org.apache.hadoop.hdfs.DFSClient: Abandoning BP-618233350--1497874683119:blk_1607145278_533526274 2019-05-28 16:04:40,382 WARN [Thread-1018] org.apache.hadoop.hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.131.5:1004,DS-6bc116bc-58f1-43df-a4b6-5ef97e486c5b,DISK] 2019-05-28 16:04:40,397 WARN [Thread-1018] org.apache.hadoop.hdfs.DFSClient: Abandoning BP-618233350--1497874683119:blk_1607145379_533526375 2019-05-28 16:04:40,399 WARN [Thread-1018] org.apache.hadoop.hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.131.49:1004,DS-9c965e9b-e86c-4cf2-b0b6-e5a622cc92f2,DISK] 2019-05-28 16:04:40,399 WARN [Thread-1052] org.apache.hadoop.hdfs.DFSClient: Abandoning BP-618233350--1497874683119:blk_1607145375_533526371 2019-05-28 16:04:40,400 WARN [Thread-1018] org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception java.io.IOException: Unable to create new block. at org.apache.h