Hadoop 中的Short-circuit local reads是什么？

最新推荐文章于 2023-09-01 16:37:35 发布

原创最新推荐文章于 2023-09-01 16:37:35 发布 · 864 阅读

0 ·

CC 4.0 BY-SA版权

喜欢文章？请私信联系作者。

Hadoop 专栏收录该内容

52 篇文章

订阅专栏

本文介绍Hadoop中短路读取(short-circuit local reads)的概念，它允许客户端直接从本地磁盘读取数据块，跳过网络传输，提高如HBase等应用的性能。通过设置dfs.client.read.shortcircuit为true并配置dfs.domain.socket.path，可以启用此功能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

`Hadoop`中的`Short-circuit local reads`是什么？

在client 和 datanode之间的传输数据
When reading a file from HDFS, the client contacts the datanode and the data is sent to the client via a TCP connection.
什么是short-circuit local reads
If the block being read is on the same node as the client, then it is more efficient for the client to bypass the network and read the block data directly from the disk. This is termed a short-circuit local read, and can make applications like HBase perform better.
如何开启short-circuit local read
You can enable short-circuit local reads by setting dfs.client.read.shortcircuit to true. Short-circuit local reads are implemented using Unix domain sockets, which use a local path for client-datanode communication. The path is set using the property dfs.domain.socket.path, and must be a path that only the datanode user (typically hdfs) or root can create, such as /var/run/hadoop-hdfs/dn_socket.