ELK之Logstash安装与配置及使用

1、Logstash介绍

Logstash 是开源的服务器端数据处理管道，能够同时 从多个来源采集数据、转换数据，然后将数据发送到您最喜欢的 “存储库” 中。（我们的存储库当然是 Elasticsearch。）

2、安装jdk

# yum -y install java-1.8.0  
# java -version  
java version "1.8.0_51"  
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)  
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)  

3、安装logstash

# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.tar.gz  
# tar zxf logstash-6.0.0.tar.gz -C /Data/apps/  

配置logstash的环境变量

# echo "export PATH=\$PATH:/Data/apps/logstash-6.0.0/bin" > /etc/profile.d/logstash.sh  
# . /etc/profile  

4、查看帮助

# logstash --help

5、logstash常用参数

-e :指定logstash的配置信息，可以用于快速测试;
-f :指定logstash的配置文件；可以用于生产环境;

6、启动logstash

6.1 通过-e参数指定logstash的配置信息，用于快速测试，直接输出到屏幕。–quiet：日志输出安静模式

$ logstash -e "input {stdin{}} output {stdout{}}" --quiet

6.2

$ logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

7、logstash以配置文件方式启动

$ vim logstash.conf  
input { stdin {} }  
output {  
   stdout { codec=> rubydebug }  
}  
$ logstash -f logstash.conf --quie  
yes ,i can  
{  
      "@version" => "1",  
          "host" => "wechat1-dev.bj1.xxxx.net",  
    "@timestamp" => 2017-11-25T10:28:38.763Z,  
       "message" => "yes ,i can"  
}  

8、更多样例

请参考官方文档样例：https://www.elastic.co/guide/en/logstash/current/config-examples.html

8.1 样例 elasticsearch

input { stdin { } }  
output {  
  elasticsearch { hosts => ["localhost:9200"] }  
  stdout { codec => rubydebug }  
}  

8.2 样例 access_log

input {  
  file {  
    path => "/tmp/access_log"  
    start_position => "beginning"  
  }  
}  


filter {  
  if [path] =~ "access" {  
    mutate { replace => { "type" => "apache_access" } }  
    grok {  
      match => { "message" => "%{COMBINEDAPACHELOG}" }  
    }  
  }  
  date {  
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]  
  }  
}  


output {  
  elasticsearch {  
    hosts => ["localhost:9200"]  
  }  
  stdout { codec => rubydebug }  
}  

8.3 写入redis

input { stdin { } }  
output {  
    stdout { codec => rubydebug }  
    redis {  
        host => '192.168.1.104'  
        data_type => 'list'  
        key => 'logstash:redis'  
    }  
}