1 背景介紹
成都創(chuàng)新互聯(lián)公司主營(yíng)煙臺(tái)網(wǎng)站建設(shè)的網(wǎng)絡(luò)公司,主營(yíng)網(wǎng)站建設(shè)方案,成都app軟件開發(fā),煙臺(tái)h5重慶小程序開發(fā)搭建,煙臺(tái)網(wǎng)站營(yíng)銷推廣歡迎煙臺(tái)等地區(qū)企業(yè)咨詢
中間件應(yīng)用運(yùn)行在Docker容器中能夠感受到容器快速創(chuàng)建快速銷毀確實(shí)帶來了很大的便利性、靈活性。但由于Docker應(yīng)用容器銷毀不管人為操作,還是容器掛掉甚至容器應(yīng)用導(dǎo)致其崩潰,它運(yùn)行的數(shù)據(jù)都會(huì)隨著它的生命周期結(jié)束而結(jié)束,這種情況是非常適合部署無狀態(tài)服務(wù)。卻不適合有狀態(tài)的應(yīng)用部署。但問題來了,假如運(yùn)維同事需要對(duì)容器應(yīng)用發(fā)生的故障進(jìn)行下一步分析,而在容器中這分部有狀態(tài)的應(yīng)用日志也會(huì)隨著容器銷毀而銷毀。日志都丟了還談什么故障分析定位,運(yùn)維同事面臨盲目被動(dòng)狀態(tài)。然而在大的容器集群環(huán)境下如果直接將容器應(yīng)用日志持久化到本地磁盤目錄,那樣日志會(huì)出現(xiàn)目錄結(jié)構(gòu)臨亂和容器日志文件也會(huì)面臨覆蓋問題。
2 日志轉(zhuǎn)存需求
1)將容器中的單個(gè)或多個(gè)業(yè)務(wù)應(yīng)用新增日志內(nèi)容按照一定目錄結(jié)構(gòu)存放在遠(yuǎn)程日志服務(wù)器。
2)存儲(chǔ)日志的目錄結(jié)構(gòu)形式需要以下方式存儲(chǔ):
/logs/app_id/service_id/container_id/app_name/xxx.log
3 工具介紹
1) Filebeat是一個(gè)日志文件托運(yùn)工具,在你的服務(wù)器上安裝客戶端后,filebeat會(huì)監(jiān)控日志目錄或者指定的日志文件,追蹤讀取這些文件(追蹤文件的變化,不停的讀),并且轉(zhuǎn)發(fā)這些信息到logstarsh中存放。
2) Logstash是一款輕量級(jí)的日志搜集處理框架,可以方便的把分散的、多樣化的日志搜集起來,并進(jìn)行自定義的處理,然后傳輸?shù)街付ǖ奈恢谩?/p>
4 Logstash日志服務(wù)器
部署日志服務(wù)器logstash配置要求如下。
系統(tǒng) |
Centos7.0 X86_64 以上 |
CPU |
4核 |
內(nèi)存 |
16G |
存儲(chǔ) |
外部存儲(chǔ)500G以上 |
Logstash軟件對(duì)JDK也是有要求的,建議在JDK1.8.0以上版本來運(yùn)行l(wèi)ogstash。
4.1 安裝JDK軟件
直接去oracle官方下載JDK版本1.8軟件來安裝
tar xvf jdk1.8.0_131.tar.gz –C /usr
然后配置JDK環(huán)境變量vi /etc/profile
JAVA_HOME=/usr/jdk1.8.0_131
CLASSPATH=.:$JAVA_HOME/lib.tools.jar
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME CLASSPATH PATH
用命令source /etc/profile生效環(huán)境變量
4.2 安裝logstash軟件
軟件下載
https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.tar.gz
軟件安裝
執(zhí)行命令tar -xvf /opt/ logstash-6.0.0.tar.gz 解壓logstash
啟動(dòng)logstash
/opt/logstash/bin/logstash -f /opt/logstash/logstash.conf
在日志服務(wù)器上新建一個(gè)/logs目錄用于存儲(chǔ)大量應(yīng)用容器日志。
5 Filebeat軟件安裝
直接將filebeat軟件和nginx,php-fpm軟件聯(lián)合一起封裝成一個(gè)新的基礎(chǔ)鏡像,事先我們需要知道哪些應(yīng)用日志文件需要提取出來。注意以下是需要提取容器中的應(yīng)用日志:
Nginx日志容器路徑
/var/log/nginx
Php-fpm日志窗口路徑:
/var/opt/remi/php70/log/php-fpm
下載filebeat軟件
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-linux-x86_64.tar.gz
以下紅色字體部署就是規(guī)范nginx與php-fpm應(yīng)用日志,并將日志文件映射到/logs目錄下。同時(shí)安裝filebeat軟件。
FROM centos MAINTAINER jaymarco@shsnc.com #Install system library RUN rpm -ivh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm && rpm -ivh http://rpms.remirepo.net/enterprise/remi-release-7.rpm && \ yum install -y php70-php-gd.x86_64 php70-php-mcrypt.x86_64 php70-php-fpm.x86_64 php70-php-pecl-redis.x86_64 python-setuptools \ php70-php-mbstring.x86_64 php70-php-snmp.x86_64 php70-php-pecl-zip.x86_64 php70-php-xml.x86_64 \ php70-php-MySQLnd.x86_64 php70-php-pecl-mysql.x86_64 gcc gcc-c++ automake libtool make cmake openssl openssl-devel pcre-devel && \ yum clean all #Make install nginx RUN rpm -ivh http://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.10.3-1.el7.ngx.x86_64.rpm COPY nginx.conf /etc/nginx/nginx.conf #set php www.conf config RUN sed -e 's/127.0.0.1:9000/9000/' \ -e '/allowed_clients/d' \ -e '/catch_workers_output/s/^;//' \ -e '/error_log/d' \ -e 's/;listen.backlog = 511/listen.backlog = 1024/' \ -e 's/pm.max_children = 50/pm.max_children = 300/' \ -e 's/pm.start_servers = 5/pm.start_servers = 30/' \ -e 's/pm.min_spare_servers = 5/pm.min_spare_servers = 30/' \ -e 's/pm.max_spare_servers = 35/pm.max_spare_servers = 60/' \ -e 's/;pm.max_requests = 500/pm.max_requests = 10240/' \ -e 's/;request_slowlog_timeout = 0/request_slowlog_timeout = 2/' \ -e 's/;request_terminate_timeout = 0/request_terminate_timeout = 20/' \ -e 's/;rlimit_files = 1024/rlimit_files = 65535/' \ -i /etc/opt/remi/php70/php-fpm.d/www.conf && \ sed -e 's/max_execution_time = 30/max_execution_time = 150/' \ -e 's/max_input_time = 60/max_input_time = 300/' \ -i /etc/opt/remi/php70/php.ini && \ sed -e 's/daemonize = yes/daemonize = no/' \ -e 's/;rlimit_files = 1024/rlimit_files = 65535/' -i /etc/opt/remi/php70/php-fpm.conf && \ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &&\ echo 'Asia/Shanghai' >/etc/timezone RUN easy_install supervisor && \ mkdir -p /var/log/supervisor && \ mkdir -p /var/run/sshd && \ mkdir -p /var/run/supervisord #Add supervisord conf ADD supervisord.conf /etc/supervisord.conf #copy start script ADD startserv.sh /startserv.sh RUN chmod +x /startserv.sh #Set port EXPOSE 9000 # For collecting logs, install filebeat plugin RUN mkdir /logs RUN ln -s /var/log/nginx /logs/ RUN ln -s /var/opt/remi/php70/log/php-fpm /logs ADD filebeat-6.0.0-linux-x86_64.tar.gz /var/log/ RUN chmod +x /var/log/filebeat/filebeat #Start web server #ENTRYPOINT ["/var/log/filebeat/init.sh"] CMD ["/startserv.sh"] |
以下配置是對(duì)nginx服務(wù)的一些性能指標(biāo)參數(shù)來優(yōu)化,并一起打包到基礎(chǔ)鏡像中。
user nginx; worker_processes 2; worker_cpu_affinity auto; error_log /var/log/nginx/error.log error; worker_rlimit_nofile 10240; worker_priority -2; events { use epoll; accept_mutex on; worker_connections 10240; } http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"' 'upstream_addr:"$upstream_addr" ' 'upstream_cache_status:"$upstream_cache_status" ' 'upstream_status:"$upstream_status" '; access_log /var/log/nginx/access.log main; sendfile on; sendfile_max_chunk 512k; aio threads; directio 4m; keepalive_timeout 65; open_log_file_cache max=1000 inactive=20s valid=1m min_uses=2; gzip on; gzip_comp_level 4; gzip_disable "MSIE [1-6]."; gzip_min_length 10k; gzip_http_version 1.0; gzip_types text/plain text/css text/xml text/javascript application/xml application/x-javascript application/xml+rss application/javascript application/json; gzip_vary on; client_max_body_size 2m; include /etc/nginx/conf.d/*.conf; } |
5.3 supervisord.conf配置參數(shù)
容器中只能運(yùn)行一個(gè)進(jìn)程,如果需要運(yùn)行多個(gè)進(jìn)程我們使用了supervisord后臺(tái)管理進(jìn)程工具,方便多進(jìn)程啟動(dòng)監(jiān)控。以下紅色字體加入了filebeat啟動(dòng)命令。
[unix_http_server] file=/tmp/supervisor.sock ; (the path to the socket file) [supervisord] logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log) logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB) logfile_backups=10 ; (num of main logfile rotation backups;default 10) loglevel=info ; (log level;default info; others: debug,warn,trace) pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid) nodaemon=true ; (start in foreground if true;default false) minfds=1024 ; (min. avail startup file descriptors;default 1024) minprocs=200 ; (min. avail process descriptors;default 200) user=root ; ; the below section must remain in the config file for RPC ; (supervisorctl/web interface) to work, additional interfaces may be ; added by defining them in separate rpcinterface: sections [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket [program:php-fpm] command=/opt/remi/php70/root/usr/sbin/php-fpm -F [program:nginx] command=/usr/sbin/nginx -c /etc/nginx/nginx.conf [program:filebeat] command=/var/log/filebeat/filebeat -c /var/log/filebeat/filebeat.yml |
5.4 startserv.sh啟動(dòng)腳本
以下紅色字體部分的內(nèi)容主要是為了生成一個(gè)filebeat.yml文件,讓filebeat程序加載對(duì)應(yīng)的app_id,service_id,host_name,logstash等參數(shù)值,并將服務(wù)接起來。
#!/bin/sh ip=`ip a|grep -w inet|grep -v -w lo|awk '{print $2}'|awk -F'/' '{print $1}'` LOGS="/logs/" #FILE=`ls -l $LOGS |awk '/^d/ {print $NF}'` FILE=`ls $LOGS` HOME="/var/log/filebeat" BAK="$HOME/bak" CONF="$HOME/filebeat.yml" HOST_NAME=`hostname` cp $BAK $CONF for name in $FILE do sed -i "/paths/a\ - $LOGS$name/*.log" $CONF done sed -i "s/#APP_ID#/$APP_ID/g" $CONF sed -i "s/#ip#/$ip/g" $CONF sed -i "s/#SERVICE_ID#/$SERVICE_ID/g" $CONF sed -i "s/#HOST_NAME#/$HOST_NAME/g" $CONF sed -i "s/#LOGSTASH_HOST#/$LOGSTASH_HOST/g" $CONF /usr/bin/supervisord -n -c /etc/supervisord.conf |
filebeat.yml例子:
filebeat: spool_size: 10240 idle_timeout: "10s" prospectors: - paths: - /logs/php-fpm/*.log - /logs/nginx/*.log fields: app_id: "6db116df" service_id: "_6db116df_64a00233" host_name: "139b3e343614" fields_under_root: true tail_files: true document_type: "172.17.0.2" processors: - drop_fields: fields: ["input_type", "beat", "offset"] output.logstash: hosts: ["XX.XX.XX.XX:5044"] worker: 2 |
打包應(yīng)用鏡像
docker build –t acitivty_front:6.0-201711221420 .
5.5 啟動(dòng)應(yīng)用容器
以上操作已經(jīng)封裝成一個(gè)新的應(yīng)用鏡像acitivty_front:6.0-201711221420,然后通過docker運(yùn)行指令來啟動(dòng)應(yīng)用鏡像。
docker run -itd -p 80:80 -e APP_ID=6db116df -e SERVICE_ID=_6db116df_64a00233 -e LOGSTASH_HOST=<日志服務(wù)器IP> acitivty_front:6.0-201711221420
6 提取日志結(jié)果
應(yīng)用容器拉起來后同時(shí)會(huì)啟動(dòng)動(dòng)應(yīng)用和filebeat插件,容器就會(huì)自動(dòng)對(duì)應(yīng)用日志統(tǒng)一收集并推送到日志服務(wù)器。以下是日志服務(wù)器中看到的提取出來的容器中的應(yīng)用日志。
Logstash日志目錄解析
日志服務(wù)器logstash接收日志
新聞標(biāo)題:基于Docker應(yīng)用容器日志文件收集
轉(zhuǎn)載來于:http://aaarwkj.com/article36/gjcdsg.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供外貿(mào)網(wǎng)站建設(shè)、企業(yè)建站、網(wǎng)站改版、定制開發(fā)、微信公眾號(hào)、用戶體驗(yàn)
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來源: 創(chuàng)新互聯(lián)