Flink 1.12.2 源码浅析 : 网络参数大全

最新推荐文章于 2025-05-22 10:50:29 发布

张伯毅

最新推荐文章于 2025-05-22 10:50:29 发布

阅读量687

点赞数

CC 4.0 BY-SA版权

分类专栏： Flink 1.12.2 源码

本文链接：https://blog.csdn.net/zhanglong_4444/article/details/117216732

Flink 1.12.2 源码专栏收录该内容

33 篇文章

订阅专栏

本文档详细介绍了Flink 1.12.2版本中关于网络通讯的配置参数，包括taskmanager的数据端口、SSL、压缩设置、网络队列管理和内存分配等，帮助开发者理解和优化任务间数据交换性能。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Flink1.12.2 网络通讯的配置参数&默认值都是在NettyShuffleEnvironmentOptions类里面.
内容如下(已忽略过期的参数.) :

Network General Options

名称	描述	默认值
taskmanager.data.port	taskmanager通讯端口	0
taskmanager.data.bind-port	任务管理器为进行数据交换而侦听的本地网络端口。	与`taskmanager.data.port` 互斥
taskmanager.data.ssl.enabled	是否开启ssl	true [实际使用false]
taskmanager.network.blocking-shuffle.compression.enabled	blocking-shuffle开启压缩[试验用,尚不稳定]	false
taskmanager.network.compression.codec	shuffle data压缩方式	LZ4
taskmanager.network.detailed-metrics	用于启用/禁用有关入站/出站网络队列长度的更详细度量。	false
taskmanager.network.memory.buffers-per-channel	Number of network buffers to use for each outgoing/incoming channel (subpartition/inputchannel).	2
taskmanager.network.memory.floating-buffers-per-gate	Number of extra network buffers to use for each outgoing/incoming gate (result partition/input gate). In credit-based flow control mode, this indicates how many floating credits are shared among all the input channels. The floating buffers are distributed based on backlog (real-time output buffers in the subpartition) feedback, and can help relieve back-pressure caused by unbalanced data distribution among the subpartitions. This value should be increased in case of higher round trip times between nodes and/or larger number of machines in the cluster.	8
taskmanager.network.sort-shuffle.min-buffers	Minimum number of network buffers required per sort-merge blocking result partition.	64
taskmanager.network.sort-shuffle.min-parallelism	Parallelism threshold to switch between sort-merge based blocking shuffle and the default hash-based blocking shuffle.	Integer.MAX_VALUE
taskmanager.network.memory.max-buffers-per-channel	Number of max buffers can be used for each output subparition.	10
taskmanager.network.memory.exclusive-buffers-request-timeout-ms	The timeout for requesting exclusive buffers for each channel.	30000L
taskmanager.network.blocking-shuffle.type	The blocking shuffle type, either mmap or file. The auto means selecting the property type automaticallbased on system memory architecture (64 bit for mmap and 32 bit for file). Note that the memory usage of mmap is not accounteby configured memory limits, but some resource frameworks like yarn would track this memory usage and kill the container oncmemory exceeding some threshold. Also note that this option is experimental and might be changed future);	file

Netty Options

名称	描述	默认值
taskmanager.network.netty.num-arenas	The number of Netty arenas.	-1
taskmanager.network.netty.server.numThreads	The number of Netty server threads.	-1
taskmanager.network.netty.client.numThreads	The number of Netty client threads.	-1
taskmanager.network.netty.server.backlog	The netty server connection backlog.	default: 0 => Netty’s default
taskmanager.network.netty.client.connectTimeoutSec	The Netty client connection timeout.	default: 120s = 2min
taskmanager.network.retries	The number of retry attempts for network communication. Currently it’s only used for establishing input/output channel connections	0
taskmanager.network.netty.sendReceiveBufferSize [!!!]	The Netty send and receive buffer size. This defaults to the system buffer size (cat `/proc/sys/net/ipv4/tcp_[rw]mem`) and is 4 MiB in modern Linux.	default: 0 => Netty’s default
taskmanager.network.netty.transport	The Netty transport type, either “nio” or “epoll”. The “auto” means selecting the property mode automatically based on the platform. Note that the “epoll” mode can get better performance, less GC and have more advanced features which are only available on modern Linux.
auto

Partition Request Options

名称	描述	默认值
taskmanager.network.request-backoff.initial	Minimum backoff in milliseconds for partition requests of input channels.	100
taskmanager.network.request-backoff.max	Maximum backoff for partition requests of input channels	10000