logstash解析naxsi日志的问题

目前在用naxsi防火墙,使用elk来做一个日志分析,遇到问题如下:

naxsi作为waf会产生error日志,目前我打开了NAXSI_EXLOG日志选项,因为这个选项可以看到具体的请求内容。

对于同一个请求,naxsi会产生2行或者3行的日志,格式如下:

2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_EXLOG: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&id=1009&zone=BODY&var_name=passwd&content=admin'%20or%20'1'='1'%20xxxxxxxxxx, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"
2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_EXLOG: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&id=1013&zone=BODY&var_name=passwd&content=admin'%20or%20'1'='1'%20xxxxxxxxxx, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"
2017/10/23 17:45:36 [error] 744#0: *19 NAXSI_FMT: ip=192.168.141.232&server=192.168.182.141&uri=/sqli-labs/Less-11/&learning=0&vers=0.55.3&total_processed=4&total_blocked=4&block=1&cscore0=$SQL&score0=22&cscore1=$XSS&score1=40&zone0=BODY&id0=1009&var_name0=passwd&zone1=BODY&id1=1013&var_name1=passwd, client: 192.168.141.232, server: _, request: "POST /sqli-labs/Less-11/ HTTP/1.1", host: "192.168.182.141:8000", referrer: "1.1.1.1"

这是同一个请求产生的结果,因为每次请求都会有一个id值在里面,这个是19:

问题:如何取出NAXSI_EXLOG里面的content,跟NAXSI_FMT里面的结果合并到一起?

我写的logstash和正则如下:

DA1 \d{4}/\d{2}/\d{2}
TM1 \d{2}:\d{2}:\d{2}
LEVEL (\w+)
NUM1 \d+(?:#0: \*)
NUM2 \d+
EXLOG NAXSI_EXLOG
FMT NAXSI_FMT
ID1 (\d+)
ZONE \w+
VAR1  (.*)
CONTENT (.*)
T3 \w+
T4 HTTP/1\.1", host: "(.*)", referrer: "
HOST (.*)

NAXSI %{DA1:date1}\s%{TM1:time}\s\[%{LEVEL:level}\]\s%{NUM1:num1}%{NUM2:num2}\s(?<logtype>NAXSI_EXLOG):\s\w+=%{HOST:client_ip}&server=%{HOST:hostname}&uri=%{PROG:filepath}&id=%{ID1:id}&zone=%{ZONE:zone}&var_name=%{VAR1:var}&content=%{CONTENT:content},\sclient\:\s%{HOST:ip3},\sserver:\s(.*)\srequest:\s"%{T3:method}\s%{HOST:uri}\sHTTP/1\.1",\shost:\s"%{HOST:host22}"

NAXSI2 %{DA1:date1}\s%{TM1:time}\s\[%{LEVEL:level}\]\s%{NUM1:num1}%{NUM2:num2}\s(?<logtype>NAXSI_EXLOG):\s\w+=%{HOST:client_ip}&server=%{HOST:hostname}&uri=%{PROG:filepath}&id=%{ID1:id}&zone=%{ZONE:zone}&var_name=%{VAR1:var}&content=%{CONTENT:content},\sclient\:\s%{HOST:ip3},\sserver:\s(.*)\srequest:\s"%{T3:method}\s%{HOST:uri}\sHTTP/1\.1",\shost:\s"%{HOST:host22}",\sreferrer:\s"(?<referrer>(.*))

FMT %{DA1:date1}\s%{TM1:time}\s\[%{LEVEL:level}\]\s%{NUM1:num1}%{NUM2:num2}\s(?<logtype>NAXSI_FMT):\sip=%{HOST:ip}&server=%{HOST:server}&uri=%{UNIXPATH:uri}&learning=%{HOST:learing}&vers=%{HOST:vers}&total_processed=%{HOST:toal_processed}&total_blocked=%{HOST:blocked}&block=%{HOST:block}&cscore0=%{HOST:attack}&score0=%{HOST:score0}&cscore1=%{HOST:xss}&score1=%{HOST:score}&zone0=%{WORD:args}&id0=%{NUMBER:id}&var_name0=%{HOST:varname},\sclient:\s%{HOST:ip3},\sserver:\s(.*)\srequest:\s"%{T3:method}\s%{HOST:uri}\sHTTP/1\.1",\shost:\s"%{HOST:host22}

logstash.conf:

input {
 file {
       path => "/usr/local/nginx/logs/naxsi.err"
       type => "naxsi-error"
       start_position => "beginning"
   }
   }
   filter {
    if [type] == "naxsi-error" {
    grok {
        patterns_dir => "/opt/logstash-5.5.1/pattern"
        match => [ "message" , "%{NAXSI2}",
               "message" , "%{NAXSI}",
               "message" , "%{FMT}"
            ]

    }
    # aggregate {
    #   task_id => "%{num2}"
    #       code => "map['sql_duration'] = 0"
    #   end_of_task => true
    #   }

}  }
output {
  if [type] == "naxsi-error" {
    elasticsearch {
       hosts => ["localhost"]
       index => "nxapi"
           document_id => "%{num2}"
        }
     }
}
发布于:2019-11-04 16:52:49