江乐的笔记

rsyslogd + graylog2 日志管理方案(2)

 tech  1928  4 分钟

graylog2安装

graylog2 是一个开源的日志存储系统，它由下面几部分组成：

  • 一个由Java写的server，接受TCP、UDP及AMQP的syslog日志信息用来存储日志内容的MongoDB服务器
  • 一个web管理界面使用ruby编写，基于Rails框架，让你可以可视化地管理你的日志

主要特性

  • 通过 TCP/UDP 接收syslog信息
  • 基于 MongoDB 的快速后端存储
  • GELF (Graylog Extended Log Format)
  • 黑名单
  • 过滤器
  • 统计

相关汇总

  • 在/opt/graylog2-server目录安装graylog2-server
  • 在/opt/graylog2-web-interface目录安装graylog2-web-interface in
  • graylog2-web-interface以普通用户graylog身份运行
  • 通过RVM安装Ruby 1.9.3和Rails 3.x环境
  • 通过Passenger-standalone，在http://127.0.0.1:3001/运行graylog2-web-interface
  • Apache监听tcp/80端口，通过mod_proxy转发3001端口的graylog2-web-interface

安装MongoDB

通过 EPEL yum repo 安装mongodb

# yum install mongodb mongodb-server
# mkdir /var/lib/mongodb
# chown -R mongodb:mongodb /var/lib/mongodb/

设置开机自动启动mongod

# chkconfig mongod on

开启mongod(日志: /var/log/mongodb/mongodb.log)

# service mongod start

运行mongo，在admin collection里创建一个admin用户，新建graylog2 collection还有graylog用户。

# mongo
    use admin
    db.addUser('admin', 'admin-mongo-passwd')
    db.auth('admin', 'admin-mongo-passwd')
    use graylog2
    db.addUser('grayloguser', 'grayloguser-mongo-passwd')

安装elasticsearch

首先是下载

cd /opt
wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.7.tar.gz	tar xzfv elasticsearch-0.18.7.tar.gz

基本设置

vi /opt/elasticsearch/config/elasticsearch.yml
`network.host:` ，设置成`127.0.0.1`
`path.logs`，设置成`/var/log/elasticsearch` 
`path.data`，设置成`/var/data/elasticsearch`
`cluster.name`，设置成`graylog2`

下载elasticsearch-servicewrapper (tanuki-wrapper)

cd /opt/elasticsearch/bin/
wget https://github.com/elasticsearch/elasticsearch-servicewrapper/zipball/master	mv master elasticsearch-servicewrapper.zip && unzip elasticsearch-servicewrapper.zip	mv elasticsearch-elasticsearch-servicewrapper-*/* . && rm -rf elasticsearch-elasticsearch-servicewrapper-*

启动elasticsearch

/opt/elasticsearch/bin/service/elasticsearch start

查看是否启动成功

tail /var/log/elasticsearch/graylog2.log
正常情况会显示文件内容

设置自动启动

vi /etc/rc.local
加上一行
/opt/elasticsearch/bin/service/elasticsearch start

安装graylog2-server

下载最新版graylog2-server压缩包: https://github.com/Graylog2/graylog2-server/downloads

mkdir /opt
cd /opt
wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz

安装到/opt/graylog2-server

tar xzvf graylog2-server-0.9.6.tar.gz
ln -s graylog2-server-0.9.6 graylog2-server

修改配置文件/etc/graylog2.conf

cp /opt/graylog2-server/graylog2.conf.example /etc/graylog2.conf
编辑/etc/graylog2.conf 设置monogdb_user, 	mongodb_password,syslog_listen_port = 1514 syslog_protocol = tcp

启动脚本/etc/init.d/graylog2-server

#!/bin/sh
#
# graylog2-server: graylog2 message collector
#
# chkconfig: - 98 02
# description: This daemon listens for syslog and GELF messages and stores them in mongodb
#
CMD=$1
NOHUP=`which nohup`
JAVA_HOME=/usr/java/latest
JAVA_CMD=$JAVA_HOME/bin/java
GRAYLOG2_SERVER_HOME=/opt/graylog2-server
start() {
echo "Starting graylog2-server ..."
$NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
}
stop() {
PID=`cat /tmp/graylog2.pid`
echo "Stopping graylog2-server ($PID) ..."
kill $PID
}
restart() {
echo "Restarting graylog2-server ..."
stop
start
}
case "$CMD" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo "Usage $0 {start|stop|restart}"
RETVAL=1
esac
view raw graylog2-server hosted with ❤ by GitHub

日志切割配置/etc/logrotate.d/graylog2-server

/var/log/graylog2.log {
daily
rotate 90
copytruncate
delaycompress
compress
notifempty
missingok
}
view raw graylog2-server hosted with ❤ by GitHub

注册graylog2-server启动脚本

chmod +x /etc/init.d/graylog2-server
chkconfig --add graylog2-server
chkconfig graylog2-server on

启动graylog2-server

service graylog2-server start
ps aux | grep graylog2
    root     23197  2.3  0.1 1196204 18428 pts/0   Sl   12:37   0:00 java -jar ../graylog2-server.jar

安装graylog2-web-interface

安装RVM，通过RVM安装ruby 1.9.3

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
rvm install 1.9.3

新建graylog用户

adduser -m graylog

下载最新版graylog2-web-interface

cd /opt/
wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
tar xzvf graylog2-web-interface-0.9.6.tar.gz
ln -s graylog2-web-interface-0.9.6 graylog2-web-interface
chown -R graylog /opt/graylog2-web-interface-0.9.6

使用bundler安装必要的gem rvm use 1.9.3 cd /opt/graylog2-web-interface gem install bundler bundle install

配置graylog2-web-interface
配置文件路径/opt/graylog2-web-interface/config 根据你的环境修改下面三个文件

  • email.yml
  • general.yml
  • mongoid.yml(请使用之前设置的用户名密码)

开启graylog2-web-inteface

rvm use 1.9.3
cd /opt/graylog2-web-interface
RAILS_ENV=production scripts/rails server

访问http://x.x.x.x:3000/ x.x.x.x是服务器ip地址。如果没问题，graylog2会要求你创建第一个用户。创建了第一个用户以后，按ctrl+c关闭graylog2-web-interface。

安装passenger-standalone

yum -y install curl-devel
rvm use 1.9.3
gem install passenger
gem install file-tail
 
cd /opt/graylog2-web-interface
mkdir tmp log
chmod -R 777 tmp log
passenger start

passenter会下载，编译需要的组件。如果passenger工作正常，按ctrl+c结束passenger。

创建启动脚本/etc/init.d/graylog2-web-interface

#!/bin/bash
#
# graylog2-web-interface: graylog2 web interface
#
# chkconfig: - 98 02
# description: Starts graylog2-web-interface using passenger-standalone. \
# Uses RVM to use switch to a specific ruby version.
#
# config
USER=graylog
APP_DIR=/opt/graylog2-web-interface
RVM_RUBY=1.9.2
ADDR=127.0.0.1
PORT=3000
ENVIRONMENT=production
LOG_FILE=/var/log/graylog2-web-interface.log
# --
CMD_START="cd $APP_DIR; rvm use $RVM_RUBY; passenger start -d \
-a $ADDR \
-p $PORT \
-e $ENVIRONMENT \
--user $USER"
CMD_STOP="cd $APP_DIR; rvm use $RVM_RUBY; passenger stop -p $PORT"
CMD_STATUS="cd $APP_DIR; rvm use $RVM_RUBY; passenger status -p $PORT"
. /lib/lsb/init-functions
case "$1" in
start)
echo "Starting graylog2-web-interface"
su - $USER -c "$CMD_START"
;;
stop)
echo "Stopping graylog2-web-interface"
su - $USER -c "$CMD_STOP"
;;
status)
su - $USER -c "$CMD_STATUS"
;;
*)
echo "Usage: $0 start|stop|status" &gt;&amp;2
exit 3
;;
esac
view raw graylog2-web-interface hosted with ❤ by GitHub

创建日志切割配置/etc/logroate.d/graylog2-web-interface

/opt/graylog2-web-interface/log/*log
       size=256M
       rotate 90
       copytruncate
       delaycompress
       compress
       notifempty
       missingok
}

注册graylog2-web-interface服务

chmod +x /etc/init.d/graylog2-web-interface
chkconfig --add graylog2-web-interface
chkconfig graylog2-web-interface on

开启graylog2-web-interface

service graylog2-web-interface start

设置apache，将80端口的请求转发到passenter

建立配置文件/etc/httpd/conf.d/graylog2.conf

NameVirtualHost *:80
 
    ServerName log.mydomain.com
    ServerAlias graylog2.mydomain.com
    ProxyPreserveHost On
    ProxyPass        / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:3000/
 
        CustomLog /var/log/httpd/log.mydomain.com-access_log common

重启服务

service httpd configtest
service httpd restart

参考资料

updatedupdated2020-07-232020-07-23

相关文章：