前段時間用ossec收集了一些系統(tǒng)的日志(syslog、secure、maillog等)，看了下elk這個架構(gòu)，發(fā)現(xiàn)很適合ossec，也很好玩。

創(chuàng)新互聯(lián)公司是一家專業(yè)提供云城企業(yè)網(wǎng)站建設(shè),專注與網(wǎng)站設(shè)計、成都做網(wǎng)站、H5高端網(wǎng)站建設(shè)、小程序制作等業(yè)務(wù)。10年已為云城眾多企業(yè)、政府機(jī)構(gòu)等服務(wù)。創(chuàng)新互聯(lián)專業(yè)網(wǎng)站設(shè)計公司優(yōu)惠進(jìn)行中。

一、介紹：

elk官網(wǎng) https://www.elastic.co/downloads

elk由elasticsearch、logstash和kiabana三個開源工具組成。

二、ossec+redis+elk架構(gòu)圖：

1、每個應(yīng)用的功能：

ossec：事件源、alert源

redis：用于處理隊列，防止數(shù)據(jù)丟失。緩沖數(shù)據(jù)。

logstash: 它用來對日志進(jìn)行收集、分割、集中日志平臺

elasticsearch: 開源分布式搜索引擎,提供搜索功能，并用來存儲最終的數(shù)據(jù)

kibana: web頁面展示，支持各種查詢、統(tǒng)計和展示

2、工作流程：

(1)、ossec client通過1514端口把日志發(fā)送給ossec server（存儲在/var/logs/ossec/alerts/alerts.log），logstash-shipper把ossec server的所有日志分割，并將分割后的日志內(nèi)容發(fā)給redis。

(2)、redis作為ossec server和logstash indexer之間的緩沖區(qū)，用來提升系統(tǒng)性能與可靠性,當(dāng)logstash提取數(shù)據(jù)失敗時,數(shù)據(jù)保存在redis中,不至于丟失。

(3)、logstash indexer提取redis的日志，將日志收集在一起(負(fù)責(zé)匯總數(shù)據(jù))。

(4)、logstash indexer再把數(shù)據(jù)交給elasticsearch，elasticsearch存儲最終的數(shù)據(jù)，并提供搜索功能。

(5)、最后通過kibana提供日志分析的web界面。

三、安裝elk：

1、elk包

elk更新很快，版本眾多，如果選擇版本不一致，可能沒辦法使用。

如果安裝最新版本elk，logstash3.x配置要更改，如果使用logstash2.52的配置，會報錯。

elk有3種安裝方式，我這里選擇tar.gz包來安裝。

logstash-1.5.2.tar.gz

elasticsearch-1.6.0.tar.gz

kibana-4.1.1-linux-x64.tar.gz

redis-3.0.6.tar.gz

2、服務(wù)器IP

ossec client：192.168.153.187

ossec server：192.168.153.172（安裝ossec server和logstash，把這臺服務(wù)器看成是logstash的client（即logstash-shipper）

elk+redis：192.168.153.200（這個logstash是server,即indexer）

3、安裝過程

(1)、192.168.153.187

安裝 ossec client，安裝見之前的博客

(2)、192.168.153.172

安裝 ossec server，安裝見之前的博客

安裝logstash

logstash依賴jdk的，安裝jdk

[root@elk-redis ~]# yum install java-1.8.0-openjdk

[root@elk-redis ~]# java -version

openjdk version "1.8.0_91"

[root@ossec-server ~]# wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

[root@ossec-server ~]# tar -xf logstash-1.5.2.tar.gz -C /usr/local/

后臺運(yùn)行logstash

[root@ossec-server ~]# /usr/local/logstash-1.5.2/bin/logstash -f /usr/local/logstash-1.5.2/logstash-200.conf &

Logstash startup completed

{

          "@timestamp" => "2016-05-19T02:03:22.746Z",

            "@version" => "1",

         "ossec_group" => "pam,syslog,",

        "reporting_ip" => "192.168.153.187",

    "reporting_source" => "/var/log/secure",

         "rule_number" => "5502",

            "severity" => 3,

           "signature" => "Login session closed.",

            "@message" => "May 19 10:03:57 localhost sshd[4623]: pam_unix(sshd:session): session closed for user root",

    "@fields.hostname" => "agent15",

     "@fields.product" => "ossec",

         "raw_message" => "** Alert 1463623401.3764: - pam,syslog,\n2016 May 19 10:03:21 (agent15) 192.168.153.187-

>/var/log/secure\nRule: 5502 (level 3) -> 'Login session closed.'\nMay 19 10:03:57 localhost sshd[4623]: pam_unix

(sshd:session): session closed for user root",

        "ossec_server" => "ossec-server"

}

{

          "@timestamp" => "2016-05-19T02:03:58.846Z",

            "@version" => "1",

         "ossec_group" => "syslog,sshd,authentication_success,",

    "reporting_source" => "192.168.153.172",

         "rule_number" => "5715",

            "severity" => 3,

           "signature" => "SSHD authentication success.",

              "src_ip" => "192.168.153.1",

                "acct" => "root",

            "@message" => "May 19 10:03:57 ossec-server sshd[22805]: Accepted password for root from 192.168.153.1 port 31490 

ssh3",

    "@fields.hostname" => "ossec-server",

     "@fields.product" => "ossec",

         "raw_message" => "** Alert 1463623437.4008: - syslog,sshd,authentication_success,\n2016 May 19 10:03:57 ossec-server-

>192.168.153.172\nRule: 5715 (level 3) -> 'SSHD authentication success.'\nSrc IP: 192.168.153.1\nUser: root\nMay 19 10:03:57 

ossec-server sshd[22805]: Accepted password for root from 192.168.153.1 port 31490 ssh3",

        "ossec_server" => "ossec-server"

(3)、192.168.153.200

a、安裝elasticsearch

elasticsearch是依賴jdk的，所以先安裝jdk

[root@elk-redis ~]# yum install java-1.8.0-openjdk

[root@elk-redis ~]# java -version

openjdk version "1.8.0_91"

[root@elk-redis ~]# tar -xf elasticsearch-1.6.0.tar.gz -C /usr/local/

后臺啟動Elasticsearch

[root@elk-redis ~]# /usr/local/elasticsearch-1.6.0/bin/elasticsearch -d

訪問192.168.153.200:9200端口，200表明es啟動成功

[root@elk-redis ~]# curl http://192.168.153.200:9200

{

  "status" : 200,

  "name" : "elasticsearch-node01",

  "cluster_name" : "elasticsearch",

  "version" : {

    "number" : "1.6.0",

    "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0",

    "build_timestamp" : "2015-06-09T13:36:34Z",

    "build_snapshot" : false,

    "lucene_version" : "4.10.4"

  },

  "tagline" : "You Know, for Search"

}

b、安裝redis 3.0.6

[root@elk-redis ~]#  tar zxvf redis-3.0.6.tar.gz

[root@elk-redis ~]#  cd redis-3.0.6

[root@elk-redis ~]#  make PREFIX=/usr/local/redis install

//這里糾結(jié)一下, redis如果不指定prefix路徑,那么默認(rèn)會在你這個解壓的文件夾中編譯生成bin文件

[root@elk-redis ~]# ln -sv /usr/local/redis/bin/redis-server /usr/bin/redis-server

[root@elk-redis ~]# ln -sv /usr/local/redis/bin/redis-cli /usr/bin/redis-cli

[root@elk-redis ~]# cp tmp/redis-3.0.6/utils/redis_init_script /etc/rc.d/init.d/redis

配置redis

[root@elk-redis ~]# vi /etc/rc.d/init.d/redis.conf

//然后在第二行插入chkconfig配置,然后修改EXEC和CLI,我的這個文件前幾行是這樣的

#!/bin/sh

# chkconfig: 2345 90 10

# Simple Redis init.d script conceived to work on Linux systems

# as it does use of the /proc filesystem.

 

REDISPORT=6379

EXEC=/usr/local/redis/bin/redis-server

CLIEXEC=/usr/local/redis/bin/redis-cli

 

PIDFILE=/var/run/redis_${REDISPORT}.pid

CONF="/etc/redis/${REDISPORT}.conf"

 

 

[root@elk-redis ~]# mkdir /etc/redis/

//這個目錄用于放我們的配置文件

[root@elk-redis ~]# mkdir /var/rdb/

//這個目錄存放redis的數(shù)據(jù)庫文件

redis源碼包中自帶redis.conf,但這個只是模版,具體配置根據(jù)自己的環(huán)境設(shè)置

[root@elk-redis ~]# vi /etc/redis/redis.conf

啟動redis

[root@elk-redis ~]# /etc/init.d/redis start

Starting Redis server...

1447:M 18 May 17:03:50.342 * Increased maximum number of open files to 10032 (it was originally set to 1024).

                _._                                                  

           _.-``__ ''-._                                             

      _.-``    `.  `_.  ''-._           Redis 3.0.6 (00000000/0) 64 bit

  .-`` .-```.  ```\/    _.,_ ''-._                                   

 (    '      ,       .-`  | `,    )     Running in standalone mode

 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379

 |    `-._   `._    /     _.-'    |     PID: 1447

  `-._    `-._  `-./  _.-'    _.-'                                   

 |`-._`-._    `-.__.-'    _.-'_.-'|                                  

 |    `-._`-._        _.-'_.-'    |           http://redis.io        

  `-._    `-._`-.__.-'_.-'    _.-'                                   

 |`-._`-._    `-.__.-'    _.-'_.-'|                                  

 |    `-._`-._        _.-'_.-'    |                                  

  `-._    `-._`-.__.-'_.-'    _.-'                                   

      `-._    `-.__.-'    _.-'                                       

          `-._        _.-'                                           

              `-.__.-'                                               

1447:M 18 May 17:03:50.345 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is 

set to the lower value of 128.

1447:M 18 May 17:03:50.346 # Server started, Redis version 3.0.6

1447:M 18 May 17:03:50.346 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix 

this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl 

vm.overcommit_memory=1' for this to take effect.

1447:M 18 May 17:03:50.346 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create 

latency and memory usage issues with Redis. To fix this issue run the command 'echo never > 

/sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a 

reboot. Redis must be restarted after THP is disabled.

1447:M 18 May 17:03:50.357 * DB loaded from disk: 0.011 seconds

1447:M 18 May 17:03:50.357 * The server is now ready to accept connections on port 6379

1447:M 18 May 17:21:03.197 * 1 changes in 900 seconds. Saving...

1447:M 18 May 17:21:03.198 * Background saving started by pid 1466

1466:C 18 May 17:21:03.202 * DB saved on disk

1466:C 18 May 17:21:03.202 * RDB: 0 MB of memory used by copy-on-write

1447:M 18 May 17:21:03.299 * Background saving terminated with success

1447:M 18 May 17:26:04.090 * 10 changes in 300 seconds. Saving...

1447:M 18 May 17:26:04.090 * Background saving started by pid 1468

1468:C 18 May 17:26:04.104 * DB saved on disk

[root@elk-redis]# redis-cli 

127.0.0.1:6379> MONITOR

OK

1463623574.234636 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623575.258853 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623575.453969 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:03:58.848Z\",\"@version

\":\"1\",\"ossec_group\":\"pam,syslog,authentication_success,\",\"reporting_source\":\"192.168.153.172\",\"rule_number\":

\"5501\",\"severity\":3,\"signature\":\"Login session opened.\",\"@message\":\"May 19 10:03:57 ossec-server sshd[22805]: 

pam_unix(sshd:session): session opened for user root by (uid=0)\",\"@fields.hostname\":\"ossec-server\",\"@fields.product\":

\"ossec\",\"raw_message\":\"** Alert 1463623437.4316: - pam,syslog,authentication_success,\\n2016 May 19 10:03:57 ossec-

server->192.168.153.172\\nRule: 5501 (level 3) -> 'Login session opened.'\\nMay 19 10:03:57 ossec-server sshd[22805]: pam_unix

(sshd:session): session opened for user root by (uid=0)\",\"ossec_server\":\"ossec-server\"}"

1463623575.456066 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623576.477031 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.018922 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.534860 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:05:17.007Z\",\"@version

\":\"1\",\"ossec_group\":\"pam,syslog,\",\"reporting_source\":\"192.168.153.172\",\"rule_number\":\"5502\",\"severity\":3,

\"signature\":\"Login session closed.\",\"@message\":\"May 19 10:05:16 ossec-server sshd[22805]: pam_unix(sshd:session): 

session closed for user root\",\"@fields.hostname\":\"ossec-server\",\"@fields.product\":\"ossec\",\"raw_message\":\"** Alert 

1463623516.4585: - pam,syslog,\\n2016 May 19 10:05:16 ossec-server->192.168.153.172\\nRule: 5502 (level 3) -> 'Login session 

closed.'\\nMay 19 10:05:16 ossec-server sshd[22805]: pam_unix(sshd:session): session closed for user root\",\"ossec_server\":

\"ossec-server\"}"

1463623601.542622 [0 127.0.0.1:48009] "blpop" "logstash:redis" "0" "1"

1463623601.562655 [0 192.168.153.172:36662] "rpush" "logstash:redis" "{\"@timestamp\":\"2016-05-19T02:05:43.092Z\",\"@version

\":\"1\",\"ossec_group\":\"syslog,sshd,authentication_success,\",\"reporting_ip\":\"192.168.153.187\",\"reporting_source\":

\"/var/log/secure\",\"rule_number\":\"5715\",\"severity\":3,\"signature\":\"SSHD authentication success.\",\"src_ip\":

\"192.168.153.1\",\"acct\":\"root\",\"@message\":\"May 19 10:06:18 localhost sshd[4834]: Accepted password for root from 

192.168.153.1 port 31537 ssh3\",\"@fields.hostname\":\"agent15\",\"@fields.product\":\"ossec\",\"raw_message\":\"** Alert 

1463623542.4820: - syslog,sshd,authentication_success,\\n2016 May 19 10:05:42 (agent15) 192.168.153.187->/var/log/secure\

\nRule: 5715 (level 3) -> 'SSHD authentication success.'\\nSrc IP: 192.168.153.1\\nUser: root\\nMay 19 10:06:18 localhost sshd

[4834]: Accepted password for root from 192.168.153.1 port 31537 ssh3\",\"ossec_server\":\"ossec-server\"}"

c、redis設(shè)置密碼訪問

[root@elk-redis ~]# vi /etc/redis/redis.conf  #此文件默認(rèn)在根目錄下。

# requirepass foobared去掉注釋，foobared改為自己的密碼，我在這里改為

requirepass xxxxxxxx

重啟服務(wù) 

[root@elk-redis ~]# /etc/init.d/redis restart

測試連接：./redis-cli -h 192.168.153.200 -p 6379 

輸入命令 會提示(error) NOAUTH Authentication required. 這是屬于正?，F(xiàn)象。

我們輸入 auth  xxxxxxxx  #你剛才設(shè)置的密碼 

d、安裝logstash

[root@elk-redis ~]# wget https://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz

[root@elk-redis ~]# tar -xf logstash-1.5.2.tar.gz -C /usr/local/

logstash配置文件

[root@elk-redis ~]# cat /usr/local/logstash-1.5.2/logstash-ossec.conf

input {

    redis 

    {

    host => "127.0.0.1"

    data_type =>"list"

    port => "6379"

    key => "logstash:redis"

    type => "ossec"

    }

}

output {

stdout { codec => rubydebug }

 if [type] == "ossec" {

   elasticsearch {

     host => "127.0.0.1"

     port => "9300"

     #cluster => "ossec"

     index => "logstash-ossec-%{+YYYY.MM.dd}"

     document_type => "ossec"

     template_name => "template-ossec"

     template => "/usr/local/share/logstash/elasticsearch_template.json"

     template_overwrite => true

        }

   }

}

后臺運(yùn)行l(wèi)ogstash

[root@elk-redis ~]# /usr/local/logstash-1.5.2/bin/logstash -f /usr/local/logstash-1.5.2/logstash-ossec.conf &

{

          "@timestamp" => "2016-05-19T02:05:43.103Z",

            "@version" => "1",

         "ossec_group" => "pam,syslog,authentication_success,",

        "reporting_ip" => "192.168.153.187",

    "reporting_source" => "/var/log/secure",

         "rule_number" => "5501",

            "severity" => 3,

           "signature" => "Login session opened.",

            "@message" => "May 19 10:06:18 localhost sshd[4834]: pam_unix(sshd:session): session opened for user root by 

(uid=0)",

    "@fields.hostname" => "agent15",

     "@fields.product" => "ossec",

         "raw_message" => "** Alert 1463623542.5137: - pam,syslog,authentication_success,\n2016 May 19 10:05:42 (agent15) 

192.168.153.187->/var/log/secure\nRule: 5501 (level 3) -> 'Login session opened.'\nMay 19 10:06:18 localhost sshd[4834]: 

pam_unix(sshd:session): session opened for user root by (uid=0)",

        "ossec_server" => "ossec-server",

                "type" => "ossec"

e、安裝kibana

[root@elk-redis ~]# tar -xf kibana-4.1.1-linux-x64.tar.gz -C /usr/local/

[root@elk-redis ~]# nohup /usr/local/kibana-4.1.1-linux-x64/bin/kibana &

(4)、訪問kibana

http://192.168.153.200:5601

elk安裝參考文章

http://baidu.blog.51cto.com/71938/1676798

新聞標(biāo)題：Centos6.4搭建ELK（1）
轉(zhuǎn)載注明：http://aaarwkj.com/article36/jjpgsg.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供網(wǎng)站制作、響應(yīng)式網(wǎng)站、面包屑導(dǎo)航、網(wǎng)站改版、關(guān)鍵詞優(yōu)化、網(wǎng)站建設(shè)

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)