閱讀271 返回首頁    go 阿裏雲 go 技術社區[雲棲]

Centos7安裝配置ELK（Elasticsearch + Logstash + Kibana）分析Nginx日誌簡單單點配置

ELK的架構原理：

logstash收集nginx日誌，並對日誌進行過濾拆分，並將處理後的結構化數據輸出給elastcsearch，es對日誌進行存儲和索引構建，kibana提供圖形界麵及對es 查詢api進行了封裝，提供友好的查詢和統計頁麵。
在生產環境中，logstash作為agent安裝部署在任何想要收集日誌的主機上，為了緩解多個agent對ES的輸出壓力，需要定義一個broker（redis）對日誌進行輸入緩衝，然後定義一個logstash server對broker中的日誌統一讀取並輸出給ES集群。broker常常使用redis，為了broker的高可用，還可以對redis做集群部署。
單點安裝測試隻部署一個es，一個logstash agent，一個kibana，一個nginx。

安裝測試流程：

1.安裝nginx-1.12.0



#安裝gcc等編譯工具

sudo yum groupinstall -y '開發工具'

#安裝nginx需要的pcre ，zlib開發庫

yum install -y pcre-devel zlib-devel  

#創建nginx的安裝目錄

mkdir nginx

#配置編譯安裝nginx
tar zxf nginx-1.12.0.tar.gz

cd nginx-1.12.0

./configure --prefix=/home/hoewon/nginx 

make 

make install

#簡單配置nginx

user root

#運行
sudo nginx

2.安裝logstash

#
tar zxf logstash-5.5.2.tar.gz

#對grok-pattern做連接

ln -s $LOGSTASH_HOME/ vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.1.1/patterns/grok-patterns/grok-patterns grok-patterns

#在grok-patterns追加nginx日誌的模式，因為對http_x_forwarded_for 的匹配不好使，所以zhushidiaole

NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}
# %{NOTSPACE:http_x_forwarded_for}


#編輯logstash啟動腳本
vim simple.conf

input {
        file{
                path => ["/home/hoewon/nginx/logs/access.log"]
                type => "nginxlog"
                start_position => "beginning"
        }

}

filter{
        grok{
                match => {

                        "message" => "%{NGINXACCESS}"
                }
        }
}

output{
        stdout{
                codec => rubydebug
        }

}

#檢查運行配置文件
 bin/logstash -t -f simple.conf

#運行logstash，並測試輸出
 bin/logstash  -f simple.conf

輸出如下

{
        "request" => "/favicon.ico",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.3368.400 QQBrowser/9.6.11974.400\"",
           "verb" => "GET",
        "message" => "192.168.247.1 - - [08/Sep/2017:15:25:46 +0800] \"GET /favicon.ico HTTP/1.1\" 403 571 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.3368.400 QQBrowser/9.6.11974.400\"",
           "type" => "nginxlog",
    "remote_user" => "-",
           "path" => "/home/hoewon/nginx/logs/access.log",
       "referrer" => "\"-\"",
     "@timestamp" => 2017-09-08T08:04:19.534Z,
       "response" => "403",
          "bytes" => "571",
       "clientip" => "192.168.247.1",
       "@version" => "1",
           "host" => "kube01",
    "httpversion" => "1.1",
      "timestamp" => "08/Sep/2017:15:25:46 +0800"
}

測試輸入輸出無誤的話，更改output插件為elsaticsearch

input {
        file{
                path => ["/home/hoewon/nginx/logs/access.log"]
                type => "nginxlog"
                start_position => "beginning"
        }

}

filter{
        grok{
                match => {

                        "message" => "%{NGINXACCESS}"
                }
        }
}

output{

        elasticsearch{
                hosts => ["192.168.247.142:9200"]
                index => "nginxlog"

        }
}

3.安裝Elasticsearch



# 
tar zxf elasticsearch-5.5.2.tar.gz

#
sudo vim /etc/security/limits.conf

#<domain>        <type>  <item>  <value>
hoewon           soft    nofile     65536
hoewon           hard    nofile     65536
hoewon           soft    nproc      2048
hoewon           hard    nproc      2048

#modify the vm.max_map_count

sudo vim /etc/sysctl.conf

  vm.max_map_count=262144

#
sysctl -p

#vim $ES_HOME/conf/elasticsearch.conf
network.host:   192.168.247.142 （or 0.0.0.0）
http.port: port
#如果是集群修改如下配置，集群是通過cluster.name自動在9300端口上尋找節點信息的
node.name: nodename
cluster.name: clustername

#
$ES_HOME/bin/elasticsearch

4.安裝kibana


#
tar zxf kibana-5.5.2-linux-x86_64.tar.gz

# 
vim $KIBANA_HOME/conf/kibana.yml

server.host: "192.168.247.142"
elasticsearch.url: "https://192.168.247.142:9200"

#
$KIBANA_HOME/bin/kibana

測試：
訪問nginx所在主機80端口。logstash會自動收集日誌，並輸出給es，登錄kibana所在主機:5601/，配置好es index的pattern，然後在discover中就可以查到文檔信息。如下：
MF_QULS0RTQI8CNHHPAPHZT

最後更新：2017-09-11 12:02:41

  上一篇：go  vue中的組件
  下一篇：go  Android 進程回收之LowMemoryKiller原理篇