fluentdを使ってnginxのログを溜めてみるメモ

まずはローカルのファイルにログをためてみる

fluentdインストール

td-agentというパッケージを使う。

参考
Installing td-agent for Redhat and CentOS / Installing td-agent daemon / Knowledge Base - Treasure Data Platform Support

[root@localhost ~]# vim /etc/yum.repos.d/td.repo
[root@localhost ~]# cat /etc/yum.repos.d/td.repo
[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/$basearch
gpgcheck=0
[root@localhost ~]# yum install td-agent

ログを置くフォルダの準備

[root@localhost ~]# mkdir /var/log/fluent
[root@localhost ~]# chown td-agent:td-agent /var/log/fluent

nginxのログフォーマット変更

[root@localhost ~]#
log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time';
…(省略）
access_log /var/log/nginx/hayazaki.access.log custom;

nginx再起動

[root@localhost ~]# /etc/init.d/nginx stop
[root@localhost ~]# /etc/init.d/nginx start

/etc/td-agent/td-agent.conf（fluentd設定ファイル）

type tail
format /^(?[^ ]*) [^ ]* (?[^ ]*) \[(?[^\]]*)\] "(?\S+)(?: +(?[^ ]*) +\S*)?" (?[^ ]*) (?[^ ]*)(?: "(?[^\"]*)" "(?[^\"]*)" (?[^ ]*))?$/
time_format %d/%b/%Y:%H:%M:%S %z
path /var/log/nginx/hayazaki.access.log
tag nginx.access.hayazaki



type file
path /var/log/archive/nginx.access.hayazaki
time_slice_format %Y%m%d_%H%M
time_slice_wait 30s
time_format %Y-%m-%d %H:%M:%S

fluentd起動

[root@localhost ~]# service td-agent start

mongodbにためてみる

mongodbインストール

[root@localhost ~]# vim /etc/yum.repos.d/10gen.repo
[root@localhost ~]# cat /etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
[root@localhost ~]# yum install mongo-10gen-server
[root@localhost ~]# service mongod start

この時点では当然まだmongoは空

[root@localhost ~]# mongo
MongoDB shell version: 2.0.2
connecting to: test
> show dbs;
local (empty)

fluent-plugin-mongoのインストール

[root@localhost ~]# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mongo

/etc/td-agent/td-agent.conf

ファイルに出力していた部分をmongoに出力するように書き換え

type mongo
host localhost
database fluent
collection nginx

td-agent再起動

[root@localhost ~]# /etc/init.d/td-agent restart

下のメッセージが出る

Starting td-agent: Able to load bson_ext version 1.4.0, but >= 1.5.2 is required.

**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
　You can install the extension as follows:
　gem install bson_ext

　If you continue to receive this message after installing, make sure that the
　bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.

bson_extをインストール後、td-agent再起動

[root@localhost ~]# /usr/lib64/fluent/ruby/bin/gem install bson_ext
[root@localhost ~]# /etc/init.d/td-agent restart

mongodbに出力されているか確認

[root@localhost ~]# curl http://localhost/
[root@localhost ~]# mongo
MongoDB shell version: 2.0.6
connecting to: test
> use fluent
switched to db fluent
> db.nginx.find();
{ "_id" : ObjectId("4fdce3b95ef6b1741f000001"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "151", "referer" : "-", "agent" : "curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2", "response_time" : "\"0.000\"", "time" : ISODate("2012-06-16T19:50:24Z") }

こんな感じでクエリを投げれば404のログだけ抽出できる

> db.nginx.find({code: "404"});