生产BUG集-CSDN博客

磁盘达到阈值导致ES无法删除数据

method [POST], host [http://xx.xxx.xxx.xxx:9200], URI [/security_event/_delete_by_query?slices=1&requests_per_second=-1&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&refresh=true&conflicts=proceed&wait_for_completion=true&timeout=1m], status line [HTTP/1.1 403 Forbidden]\n{\"took\":19,\"timed_out\":false,\"total\":8,\"deleted\":0,\"batches\":1,\"version_conflicts\":0,\"noops\":0,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1.0,\"throttled_until_millis\":0,\"failures\":[{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"ecb098ef-2e3a-4c7a-a282-4484cabb362f\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"30c1da71-25c5-4e1f-a58f-95f6f5abfc52\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"687e14ba-0bc8-466a-83b1-294a3f9b2422\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"e0f25c75-cce3-4c44-9691-d4c79ecb72e1\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"dbfabe19-3318-420c-a7f0-eb74eab25d43\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"222fcf5e-8b72-4bd8-b3c8-a8c615db2ae9\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"690f57a2-1306-4749-8dfc-a1f0c5d926e5\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"280ff8cc-f43a-40ee-9ae3-9af48960832b\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}]}

排查思路。之前有一个清理数据的接口。用来清理大屏的历史数据。运维人员根据往常一样执行接口。但是报错了反馈到我这里。

一开始看到报错有很多403。联想到之前这个环境做了安全检测。关闭了很多端口。之前es没有认证以为被扫到了，询问运维人员得知es在内部网络并没有开放出去。于是重新审视问题报错。看到

index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

索引是只读模式不允许删除，但是并没有人去主动操作es。深入查询后发现es会检测磁盘使用率达到85%(默认)会将索引模型调整。检查磁盘后确实使用率达到了95%

解决方案：

临时解除只读限制

curl -X PUT "localhost:9200/security_event/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '

永久配置磁盘水印

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } } '

后续交给运维人员清理磁盘数据。

Kafka 多监听/网络隔离转发

问题现象

其他业务系统需要将其他网络的kafka发送消息，网络链路是通的，但是收发有问题。

环境

B网络其他业务系统需要向A网络Kafka发送消息，于是开放B网络访问对A网络宿主机的50501端口的网络策略，A网络宿主机将50501转发到Kafka虚拟机(docker)的19092端口

kafka使用docker方式，命令如下

忽略...指令404。有空再补充(单网卡监听)

---------------------------------------------------------------------------

最后查找一番找到了一种解决方案，kafka采用多端口监听方式，命令如下:

docker run -d --restart always -p 9092:9092 -p 19092:19092 \
-v /data/sto/kafka:/kafka \
-e KAFKA_BROKER_ID=1 -e KAFKA_MESSAGE_MAX_BYTES="1000000000" \
-e KAFKA_SOCKET_REQUEST_MAX_BYTES="104857600" \
-e KAFKA_AUTO_CREATE_TOPICS_ENABLE="true" \
-e KAFKA_ZOOKEEPER_CONNECT="192.168.3.127:2181" \
-e KAFKA_LISTENERS="INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:19092" \
-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT \
-e KAFKA_ADVERTISED_LISTENERS="INSIDE://172.172.10.56:9092,OUTSIDE://160.0.27.65:50501" \
-e KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE -e TZ=Asia/Shanghai \
--name kafka 镜像名称:版本

KAFKA_LISTENERS：用来监听端口
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP：不同协议的分类
KAFKA_ADVERTISED_LISTENERS：告知客户端后续收发的节点地址

这样的话连接9092端口kafka会发送172.172.10.56:9092节点作为收发节点地址，连接19092发送160.0.27.65:50501作为后续收发的节点地址