培訓-ELK日志監(jiān)控報警實戰(zhàn)課件

ELK日志監(jiān)控報警實戰(zhàn)磁云科技張人杰2018.10.1612ELK日志監(jiān)控報警實戰(zhàn)磁云科技張人杰12什么是ELKE:

ElasticSearchL:

LogstashK:

Kibana211/22/2022什么是ELKE:ElasticSearch211/22/2運行效果當服務器宕機時,立即發(fā)送郵件通知311/22/2022運行效果當服務器宕機時,立即發(fā)送郵件通知311/22/202環(huán)境搭建(一)

ElasticSearch安裝1、安裝elasticsearch的yum源的密鑰rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch2、配置elasticsearch的yum源vim/etc/yum.repos.d/elasticsearch.repo[elasticsearch-6.x]name=Elasticsearchrepositoryfor6.xpackagesbaseurl=https://artifacts.elastic.co/packages/6.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md3、安裝elasticsearchyuminstall-yelasticsearch

411/22/2022環(huán)境搭建(一)

ElasticSearch安裝1、安裝ela環(huán)境搭建(一)

ElasticSearch環(huán)境搭建1、需要安裝jdk1.8版本以上的java-version2、創(chuàng)建elasticsearchdata的存放目錄，并修改該目錄的屬主屬組mkdir-p/data/es-datachown-Relasticsearch:elasticsearch/data/es-data3、修改elasticsearch的日志屬主屬組chown-Relasticsearch:elasticsearch/var/log/elasticsearch/4、修改elasticsearch的配置文件vim/etc/elasticsearch/elasticsearch.yml

511/22/2022環(huán)境搭建(一)

ElasticSearch環(huán)境搭建1、需要安/etc/elasticsearch/elasticsearch.yml編輯找到配置文件中的，打開該配置并設置集群名稱:elk-tang找到配置文件中的，打開該配置并設置節(jié)點名稱:elk-tang-1修改data存放的路徑path.data:/data/es-data修改logs日志的路徑path.logs:/var/log/elasticsearch/注釋配置內(nèi)存使用用交換分區(qū)#bootstrap.memory_lock:true監(jiān)聽的網(wǎng)絡地址network.host:開啟監(jiān)聽的端口http.port:9200增加新的參數(shù)，這樣head插件可以訪問es(5.x版本，如果沒有可以自己手動加)http.cors.enabled:truehttp.cors.allow-origin:"*"611/22/2022/etc/elasticsearch/elasticsear啟動ElasticSearch/etc/init.d/elasticsearchstart

711/22/2022啟動ElasticSearch/etc/init.d/ela創(chuàng)建開機自啟動服務chkconfigelasticsearchon811/22/2022創(chuàng)建開機自啟動服務chkconfigelasticsear其他需要修改的參數(shù)vim/etc/security/limits.conf在末尾追加以下內(nèi)容（elk為啟動用戶，當然也可以指定為*）elksoftnofile65536elkhardnofile65536elksoftnproc2048elkhardnproc2048elksoftmemlockunlimitedelkhardmemlockunlimitedvim/etc/security/limits.d/XXX-nproc.conf將里面的1024改為2048（ES最少要求為2048）*softnproc2048vim/etc/elasticsearch/elasticsearch.yml加入以下內(nèi)容bootstrap.system_call_filter:false911/22/2022其他需要修改的參數(shù)vim/etc/security/lim再次啟動/etc/init.d/elasticsearchrestart1011/22/2022再次啟動/etc/init.d/elasticsearch環(huán)境搭建（二）

安裝elasticsearch-head插件安裝node.jssudocurl-sL-o/etc/yum.repos.d/khara-nodejs.repo/coprs/khara/nodejs/repo/epel-7/khara-nodejs-epel-7.reposudoyuminstall-ynodejsnodejs-npm安裝headgitclonegit:///mobz/elasticsearch-head.gitcdelasticsearch-headnpminstallnpmrunstart1111/22/2022環(huán)境搭建（二）

安裝elasticsearch-head插件環(huán)境搭建（三）

安裝Logstash環(huán)境Logstash需要安裝到產(chǎn)生日志的服務器上rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearchyuminstall-ylogstashrpm-qllogstashln-s/usr/share/logstash/bin/logstash/bin/1211/22/2022環(huán)境搭建（三）

安裝Logstash環(huán)境Logstash需要Logstash配置input{file{path=>["/var/log/nginx/access.log"]start_position=>"beginning"ignore_older=>0}}filter{grok{patterns_dir=>"/opt/logstash/patterns"match=>{"message"=>"%{NGINXACCESS}"}add_field=>[“resp_code”,“%{response}”]}geoip{source=>"http_x_forwarded_for"target=>"geoip"database=>"/etc/logstash/GeoLite2-City.mmdb"add_field=>["[geoip][coordinates]","%{[geoip][longitude]}"]add_field=>["[geoip][coordinates]","%{[geoip][latitude]}"]}mutate{convert=>["[geoip][coordinates]","float"]convert=>["response","integer"]convert=>["bytes","integer"]replace=>{"type"=>"nginx_access"}remove_field=>"message"}date{match=>["timestamp","dd/MMM/yyyy:HH:mm:ssZ"]}mutate{remove_field=>"timestamp"}}output{elasticsearch{hosts=>[":9200"]index=>"logstash-nginx-access-%{+YYYY.MM.dd}"}stdout{codec=>rubydebug}}1311/22/2022Logstash配置input{1311/22/2022建立grok使用的表達式mkdir-pv/opt/logstash/patternsvi/opt/logstash/patterns/nginxNGUSERNAME[a-zA-Z\.\@\-\+_%]+NGUSER%{NGUSERNAME}NGINXACCESS%{IPORHOST:clientip}-%{NOTSPACE:remote_user}\[%{HTTPDATE:timestamp}\]\"(?:%{WORD:verb}%{NOTSPACE:request}HTTP/%{NUMBER:httpversion}|%{DATA:rawrequest})\"%{NUMBER:response}(?:%{NUMBER:bytes}|-)%{QS:referrer}%{QS:agent}\"(?:%{IPV4:http_x_forwarded_for}|-)\"1411/22/2022建立grok使用的表達式mkdir-pv/opt/logGeoIP的數(shù)據(jù)庫解析ipwget/download/geoip/database/GeoLite2-City.tar.gztar-xzvfGeoLite2-City.tar.gzmvGeoLite2-City_20181030/GeoLite2-City.mmdb/etc/logstash/.1511/22/2022GeoIP的數(shù)據(jù)庫解析ipwgethttp://geoli測試配置文件并啟動Logstash服務logstash-t-f./elk.confnohuplogstash–f./elk.conf2>&1>/dev/null&1611/22/2022測試配置文件并啟動Logstash服務logstash-t環(huán)境搭建（四）

Kibanawgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz#注意需要與ES對應的版本tar-xzfkibana-6.4.2-linux-x86_64.tar.gzmvkibana-6.4.2-linux-x86_64/usr/localln-s/usr/local/kibana-6.4.2-linux-x86_64//usr/local/kibanavim/usr/local/kibana/config/kibana.yml1711/22/2022環(huán)境搭建（四）

Kibanawgethttps://art/usr/local/kibana/config/kibana.yml編輯server.port:5601server.host:""elasticsearch.url:"http://localhost:9200"kibana.index:".kibana"1811/22/2022/usr/local/kibana/config/kiban安裝screen,以便于kibana在后臺運行yum-yinstallscreenscreen/usr/local/kibana/bin/kibana1911/22/2022安裝screen,以便于kibana在后臺運行yum-yKibana安裝完成打開瀏覽器并設置對應的indexhttp://localhost:56012011/22/2022Kibana安裝完成打開瀏覽器并設置對應的indexhtt在Kibana上安裝sentinl插件用于發(fā)送郵件提醒1、到/sirensolutions/sentinl/releases上選擇合適的版本2、在服務器上運行：/usr/local/kibana/bin/kibana-plugininstall/sirensolutions/sentinl/releases/download/tag-6.4.2-0/sentinl-v6.4.2.zip3、重啟kibana4、在kibana界面上配置sentinl2111/22/2022在Kibana上安裝sentinl插件用于發(fā)送郵件提醒1、到注意：需要定時清理日志文件定期清理nginx日志文件echo'::1--[03/Nov/2018:20:28:03+0800]"GET/HTTP/1.1"2000"-""Mozilla/5.0(X11;Linuxx86_64;rv:52.0)Gecko/20100101Firefox/52.0""-"'>/var/log/nginx/access.log定期清理ES中的日志文件curl-XDELETEhttp://localhost:9200/nginx-*-`date+%Y-%m-%d-d"-$ndays"`2211/22/2022注意：需要定時清理日志文件定期清理nginx日志文件2211防火墻配置所有端口僅對內(nèi)網(wǎng)開放2311/22/2022防火墻配置所有端口僅對內(nèi)網(wǎng)開放2311/22/2022結(jié)束張人杰2018.11.42411/22/20222411/22/2022ELK日志監(jiān)控報警實戰(zhàn)磁云科技張人杰2018.10.16252ELK日志監(jiān)控報警實戰(zhàn)磁云科技張人杰12什么是ELKE:

ElasticSearchL:

LogstashK:

Kibana2611/22/2022什么是ELKE:ElasticSearch211/22/2運行效果當服務器宕機時,立即發(fā)送郵件通知2711/22/2022運行效果當服務器宕機時,立即發(fā)送郵件通知311/22/202環(huán)境搭建(一)

ElasticSearch安裝1、安裝elasticsearch的yum源的密鑰rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch2、配置elasticsearch的yum源vim/etc/yum.repos.d/elasticsearch.repo[elasticsearch-6.x]name=Elasticsearchrepositoryfor6.xpackagesbaseurl=https://artifacts.elastic.co/packages/6.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md3、安裝elasticsearchyuminstall-yelasticsearch

2811/22/2022環(huán)境搭建(一)

ElasticSearch安裝1、安裝ela環(huán)境搭建(一)

ElasticSearch環(huán)境搭建1、需要安裝jdk1.8版本以上的java-version2、創(chuàng)建elasticsearchdata的存放目錄，并修改該目錄的屬主屬組mkdir-p/data/es-datachown-Relasticsearch:elasticsearch/data/es-data3、修改elasticsearch的日志屬主屬組chown-Relasticsearch:elasticsearch/var/log/elasticsearch/4、修改elasticsearch的配置文件vim/etc/elasticsearch/elasticsearch.yml

2911/22/2022環(huán)境搭建(一)

ElasticSearch環(huán)境搭建1、需要安/etc/elasticsearch/elasticsearch.yml編輯找到配置文件中的，打開該配置并設置集群名稱:elk-tang找到配置文件中的，打開該配置并設置節(jié)點名稱:elk-tang-1修改data存放的路徑path.data:/data/es-data修改logs日志的路徑path.logs:/var/log/elasticsearch/注釋配置內(nèi)存使用用交換分區(qū)#bootstrap.memory_lock:true監(jiān)聽的網(wǎng)絡地址network.host:開啟監(jiān)聽的端口http.port:9200增加新的參數(shù)，這樣head插件可以訪問es(5.x版本，如果沒有可以自己手動加)http.cors.enabled:truehttp.cors.allow-origin:"*"3011/22/2022/etc/elasticsearch/elasticsear啟動ElasticSearch/etc/init.d/elasticsearchstart

3111/22/2022啟動ElasticSearch/etc/init.d/ela創(chuàng)建開機自啟動服務chkconfigelasticsearchon3211/22/2022創(chuàng)建開機自啟動服務chkconfigelasticsear其他需要修改的參數(shù)vim/etc/security/limits.conf在末尾追加以下內(nèi)容（elk為啟動用戶，當然也可以指定為*）elksoftnofile65536elkhardnofile65536elksoftnproc2048elkhardnproc2048elksoftmemlockunlimitedelkhardmemlockunlimitedvim/etc/security/limits.d/XXX-nproc.conf將里面的1024改為2048（ES最少要求為2048）*softnproc2048vim/etc/elasticsearch/elasticsearch.yml加入以下內(nèi)容bootstrap.system_call_filter:false3311/22/2022其他需要修改的參數(shù)vim/etc/security/lim再次啟動/etc/init.d/elasticsearchrestart3411/22/2022再次啟動/etc/init.d/elasticsearch環(huán)境搭建（二）

安裝elasticsearch-head插件安裝node.jssudocurl-sL-o/etc/yum.repos.d/khara-nodejs.repo/coprs/khara/nodejs/repo/epel-7/khara-nodejs-epel-7.reposudoyuminstall-ynodejsnodejs-npm安裝headgitclonegit:///mobz/elasticsearch-head.gitcdelasticsearch-headnpminstallnpmrunstart3511/22/2022環(huán)境搭建（二）

安裝elasticsearch-head插件環(huán)境搭建（三）

安裝Logstash環(huán)境Logstash需要安裝到產(chǎn)生日志的服務器上rpm--importhttps://artifacts.elastic.co/GPG-KEY-elasticsearchyuminstall-ylogstashrpm-qllogstashln-s/usr/share/logstash/bin/logstash/bin/3611/22/2022環(huán)境搭建（三）

安裝Logstash環(huán)境Logstash需要Logstash配置input{file{path=>["/var/log/nginx/access.log"]start_position=>"beginning"ignore_older=>0}}filter{grok{patterns_dir=>"/opt/logstash/patterns"match=>{"message"=>"%{NGINXACCESS}"}add_field=>[“resp_code”,“%{response}”]}geoip{source=>"http_x_forwarded_for"target=>"geoip"database=>"/etc/logstash/GeoLite2-City.mmdb"add_field=>["[geoip][coordinates]","%{[geoip][longitude]}"]add_field=>["[geoip][coordinates]","%{[geoip][latitude]}"]}mutate{convert=>["[geoip][coordinates]","float"]convert=>["response","integer"]convert=>["bytes","integer"]replace=>{"type"=>"nginx_access"}remove_field=>"message"}date{match=>["timestamp","dd/MMM/yyyy:HH:mm:ssZ"]}mutate{remove_field=>"timestamp"}}output{elasticsearch{hosts=>[":9200"]index=>"logstash-nginx-access-%{+YYYY.MM.dd}"}stdout{codec=>rubydebug}}3711/22/2022Logstash配置input{1311/22/2022建立grok使用的表達式mkdir-pv/opt/logstash/patternsvi/opt/logstash/patterns/nginxNGUSERNAME[a-zA-Z\.\@\-\+_%]+NGUSER%{NGUSERNAME}NGINXACCESS%{IPORHOST:clientip}-%{NOTSPACE:remote_user}\[%{HTTPDATE:timestamp}\]\"(?:%{WORD:verb}%{NOTSPACE:request}HTTP/%{NUMBER:httpversion}|%{DATA:rawrequest})\"%{NUMBER:response}(?:%{NUMBER:bytes}|-)%{QS:referrer}%{QS:agent}\"(?:%{IPV4:http_x_forwarded_for}|-)\"3811/22/2022建立grok使用的表達式mkdir-pv/opt/logGeoIP的數(shù)據(jù)庫解析ipwget/download/geoip/database/GeoLite2-City.tar.gztar-xzvfGeoLite2-City.tar.gzmvGeoLite2-City_20181030/GeoLite2-City.mmdb/etc/logstash/.3911/22/2022GeoIP的數(shù)據(jù)庫解析ipwgethttp://geoli測試配置文件并啟動Logstash服務logstash-t-f./elk.confnohuplogstash–f./elk.conf2>&1>/dev/null&4011/22/2022測試配置文件并啟動Logstash服務logstash-t環(huán)境搭建（四）

Kibanawgethttps://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz#注意需要與ES對應的版本tar-xzfkibana-6.4.2-linux-x86_64.tar.gzmvkibana-6.4.2-linux-x86_64/usr/localln-s/usr/local/kibana-6.4.2-linux-x86_64/