Logstash配置语法

最新推荐文章于 2025-06-14 14:54:04 发布

原创

最新推荐文章于 2025-06-14 14:54:04 发布 · 4.3k 阅读

7 ·

CC 4.0 BY-SA版权

文章标签：

#logstash

本文详细介绍了Logstash的配置语法，包括输入插件如File、Stdin、Syslog和TCP，编码插件如plain、json、json_lines，以及过滤器插件Grok、Date、Mutate和GeoIP。Logstash通过输入、解码、过滤、编码和输出，实现了日志数据的处理和流转。示例中展示了如何配置和使用各种插件，以实现日志数据的结构化和分析。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

文章目录

Logstash基本语法组成

logstash之所以功能强大和流行，还与其丰富的过滤器插件是分不开的，过滤器提供的并不单单是过滤的功能，还可以对进入过滤器的原始数据进行复杂的逻辑处理，甚至添加独特的事件到后续流程中。

Logstash配置文件有如下三部分组成，其中input、output部分是必须配置，filter部分是可选配置，而filter就是过滤器插件，可以在这部分实现各种日志过滤功能。

input {
   
   
    #输入插件
}
filter {
   
   
    #过滤匹配插件
}
output {
   
   
    #输出插件
}

Logstash输入插件(Input)

input插件介绍、下载及配置文件写法，官方文档：https://www.elastic.co/guide/en/logstash/current/input-plugins.html

1、读取文件(File)

logstash使用一个名为filewatch的ruby gem库来监听文件变化，并通过一个叫.sincedb的数据库文件来记录被监听的日志文件的读取进度（时间戳），这个sincedb数据文件的默认路径在 <path.data>/plugins/inputs/file下面，文件名类似于 .sincedb_452905a167cf4509fd08acb964fdb20c，而<path.data>表示logstash插件存储目录，默认是LOGSTASH_HOME/data。

[root@logstashserver ~]# ll /usr/local/logstash/data/plugins/inputs/file/.sincedb_452905a167cf4509fd08acb964fdb20c
-rw-r--r--. 1 root root 60 Aug 10 11:04 /usr/local/logstash/data/plugins/inputs/file/.sincedb_452905a167cf4509fd08acb964fdb20c

看下面一个事件配置文件：

[root@logstashserver config]# vim logstash1.conf
input {
   
   
    file {
   
   		# 指定输入源为文件
        path => ["/var/log/secure"]	# 文件路径，指定多个使用逗号分隔；["file1","file2","file3"]
        type => "system"				# 标记事件类型，会原样输出
        start_position => "beginning"	# 按时间戳记录的时间开始读取该文件，如果没有时间戳则从文件的开头读取
    }
}
output {
   
   
    stdout{
   
   		# 输出源为标准输出
        codec=>rubydebug  # 编码格式   
    }
}

这个配置是监听并接收本机的/var/log/messages文件内容，start_position表示按时间戳记录的地方开始读取，如果没有时间戳则从头开始读取，有点类似cat命令，默认情况下，logstash会从文件的结束位置开始读取数据，也就是说logstash进程会以类似tail -f命令的形式逐行获取数据。type用来标记事件类型，通常会在输入区域通过type标记事件类型。

执行

# 因为指定从文件的开头开始读取，所以输出信息比较多
[root@logstashserver config]# ../bin/logstash -f logstash1.conf 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2021-08-11T10:59:03,540][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-11T10:59:03,565][INFO ][logstash.runner          ] Starting Logstash {
   
   "logstash.version"=>"6.5.4"}
[2021-08-11T10:59:07,203][INFO ][logstash.pipeline        ] Starting pipeline {
   
   :pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-08-11T10:59:07,512][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {
   
   :sincedb_path=>"/usr/local/logstash/data/plugins/inputs/file/.sincedb_730aea1d074d4636ec2eacfacc10f882", :path=>["/var/log/secure"]}
[2021-08-11T10:59:07,574][INFO ][logstash.pipeline        ] Pipeline started successfully {
   
   :pipeline_id=>"main", :thread=>"#<Thread:0x3918232d run>"}
[2021-08-11T10:59:07,683][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-08-11T10:59:07,711][INFO ][logstash.agent           ] Pipelines running {
   
   :count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-08-11T10:59:08,253][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {
   
   :port=>9600}
{
   
   
    "@timestamp" => 2021-08-11T02:59:08.488Z,
          "host" => "logstashserver",
      "@version" => "1",
          "type" => "system",
       "message" => "Aug  8 10:08:32 nginx login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)",
          "path" => "/var/log/secure"
}
{
   
   
    "@timestamp" => 2021-08-11T02:59:08.517Z,
          "host" => "logstashserver",
      "@version" => "1",
          "type" => "system",
       "message" => "Aug  8 10:08:32 nginx login: ROOT LOGIN ON tty1",
          "path" => "/var/log/secure"
}
{
   
   
    "@timestamp" => 2021-08-11T02:59:08.518Z,	# 时间戳
          "host" => "logstashserver",	# 日志来自哪个服务器
      "@version" => "1",		# 版本
          "type" => "system",	# 配置文件中标记的事件类型
       "message" => "Aug  8 10:09:09 nginx polkitd[665]: Registered Authentication Agent for unix-process:1175:93557 (system bus name :1.23 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)",		# 日志内容
          "path" => "/var/log/secure"	# 文件源
}
……省略……

2、标准输入(Stdin)

stdin是从标准输入获取信息，关于stdin的使用，前面已经做过了一些简单的介绍，这里再看一个稍微复杂一点的例子，下面是一个关于stdin的事件配置文件：

[root@logstashserver config]# vim logstash2.conf
input{
   
   
    stdin{
   
   
        add_field=>{
   
   "key"=>"iivey"}		# 自定义添加字段（一对键值对）
        tags=>["add1"]		# 标签
        type=>"test1"		# 事件类型
    }
}
output {
   
   
    stdout{
   
   
        codec=>rubydebug    
    }
}

# 执行
[root@logstashserver config]# ../bin/logstash -f logstash2.conf 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2021-08-11T11:06:09,529][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-11T11:06:09,555][INFO ][logstash.runner          ] Starting Logstash {
   
   "logstash.version"=>"6.5.4"}
[2021-08-11T11:06:13,561][INFO ][logstash.pipeline        ] Starting pipeline {
   
   :pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-08-11T11:06:13,697][INFO ][logstash.pipeline        ] Pipeline started successfully {
   
   :pipeline_id=>"main", :thread=>"#<Thread:0x49013ec0 run>"}
The stdin plugin is now waiting for input:
[2021-08-11T11:06:13,768][INFO ][logstash.agent           ] Pipelines running {
   
   :count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-08-11T11:06:14,012][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {
   
   :port=>9600}
hello world			# 标准输入
{
   
   
    "@timestamp" => 2021-08-11T03:06:22.801Z,
           "key" => "iivey",
      "@version" => "1",
       "message" => "hello world",
          "host" => "logstashserver",
          "tags" => [
        [0] "add1"
    ],
          "type" => "test1"
}

3、读取 Syslog日志

如何将rsyslog收集到的日志信息发送到logstash中，这里以centos7.5为例，需要做如下两个步骤的操作：

首先，在需要收集日志的服务器上找到rsyslog的配置文件/etc/rsyslog.conf,添加如下内容：

[root@filebeatserver ~]# ifconfig ens32 | awk 'NR==2 {print $2}'
192.168.126.90
[root@filebeatserver ~]# vim /etc/rsyslog.conf
*.* @@192.168.126.94:6666
# 其中，192.168.126.94是logstash服务器的地址。6666端口必须是logstash启动时所监听的一个端口。

接着，重启rsyslog服务：

root@filebeatserver ~]# systemctl  restart rsyslog

然后，在logstash服务器上创建一个事件配置文件，内容如下：

[root@logstashserver config]# vim logstash3.conf
input {
   
   
  syslog {
   
   
    port => "6666"		# 此端口用于接收rsyslog传递的日志数据
  }
}

output {
   
   
    stdout{
   
   
        codec=>rubydebug    
    }
}

# 启动logstash
[root@logstashserver config]# ../bin/logstash -f logstash3.conf 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2021-08-11T11:20:45,094][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-08-11T11:20:45,113][INFO ][logstash.runner          ] Starting Logstash {
   
   "logstash.version"=>"6.5.4"}
[2021-08-11T11:20:48,854][INFO ][logstash.pipeline        ] Starting pipeline {
   
   :pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2021-08-11T11:20:49,376][INFO ][logstash.pipeline        ] Pipeline started successfully {
   
   :pipeline_id=>"main", :thread=>"#<Thread:0x17a8ed98 run>"}
[2021-08-11T11:20:49,438][INFO ][logstash.inputs.syslog   ] Starting syslog tcp listener {
   
   :address=>"0.0.0.0:6666"}	# 启动基于TCP的6666端口
[2021-08-11T11:20:49,448][INFO ][logstash.agent           ] Pipelines running {
   
   :count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2021-08-11T11:20:49,467][INFO ][logstash.inputs.syslog   ] Starting syslog udp listener {
   
   :address=>"0.0.0.0:6666"}	# 启动基于UDP的6666端口
[2021-08-11T11:20:49,738][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {
   
   :port=>9600}
# 此时在rsyslog服务器上新开一个xshell会话窗口，相当于模拟一些日志信息，此时logstash接收到后输出内容如下：
[2021-08-11T11:21:24,752][INFO ][logstash.inputs.syslog   ] new connection {
   
   :client=>"192.168.126.90:55320"}
{
   
   
          "facility" => 10,
          "@version" => "1",
          "severity" => 6,
          "priority" => 86,
               "pid" => "1098",
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "Accepted password for root from 192.168.126.1 port 60771 ssh2\n",
    "facility_label" => "security/authorization",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "sshd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 5,
          "@version" => "1",
          "severity" => 6,
          "priority" => 46,
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "action 'action 7' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]\n",
    "facility_label" => "syslogd",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "rsyslogd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 5,
          "@version" => "1",
          "severity" => 6,
          "priority" => 46,
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "action 'action 7' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]\n",
    "facility_label" => "syslogd",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "rsyslogd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 4,
          "@version" => "1",
          "severity" => 6,
          "priority" => 38,
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "New session 2 of user root.\n",
    "facility_label" => "security/authorization",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "systemd-logind",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 3,
          "@version" => "1",
          "severity" => 6,
          "priority" => 30,
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "Started Session 2 of user root.\n",
    "facility_label" => "system",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "systemd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 10,
          "@version" => "1",
          "severity" => 6,
          "priority" => 86,
               "pid" => "1098",
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n",
    "facility_label" => "security/authorization",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "sshd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
{
   
   
          "facility" => 3,
          "@version" => "1",
          "severity" => 6,
          "priority" => 30,
        "@timestamp" => 2021-08-11T03:21:24.000Z,
           "message" => "Starting Session 2 of user root.\n",
    "facility_label" => "system",
         "timestamp" => "Aug 11 11:21:24",
              "host" => "192.168.126.90",
           "program" => "systemd",
    "severity_label" => "Informational",
         "logsource" => "filebeatserver"
}
# 将每条日志信息格式化为许多字段，这有助于日志的分析和过滤等

4、读取TCP网络数据

下面的事件配置文件就是通过"LogStash::Inputs::TCP"和"LogStash::Filters::Grok"配合实现syslog功能的例子，这里使用了logstash的TCP/UDP插件读取网络数据：

[root@logstashserver config]# vim logstash4.conf
input {
   
   
  tcp {
   
   
    port => "6666"
  }
}

filter {
   
   
  grok {
   
   
    match => {
   
    "message" => "%{SYSLOGLINE}" }
  }
}

output {
   
   
    stdout{
   
   
        codec=>rubydebug
    }
}
# 其中，6666端口是logstash启动的tcp监听的端口。注意这里用到了日志过滤"LogStash::Filters::Grok"功能

[root@logstashserver config]# ../bin/logstash -f logstash4.conf 
Sending Logstash logs to /usr/local/logstash/logs which is now configured via log4j2.properties
[2021-08-11T11:55:22,992]