読者です 読者をやめる 読者になる 読者になる

zakihayaメモ

RubyとRailsのことが中心

fluentdを使ってnginxのログを溜めてみるメモ

まずはローカルのファイルにログをためてみる

fluentdインストール

td-agentというパッケージを使う。

参考
Installing td-agent for Redhat and CentOS / Installing td-agent daemon / Knowledge Base - Treasure Data Platform Support

[root@localhost ~]# vim /etc/yum.repos.d/td.repo
[root@localhost ~]# cat /etc/yum.repos.d/td.repo
[treasuredata]
name=TreasureData
baseurl=http://packages.treasure-data.com/redhat/$basearch
gpgcheck=0
[root@localhost ~]# yum install td-agent

ログを置くフォルダの準備

[root@localhost ~]# mkdir /var/log/fluent
[root@localhost ~]# chown td-agent:td-agent /var/log/fluent

nginxのログフォーマット変更

[root@localhost ~]#
log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time';
…(省略)
access_log /var/log/nginx/hayazaki.access.log custom;

nginx再起動
[root@localhost ~]# /etc/init.d/nginx stop
[root@localhost ~]# /etc/init.d/nginx start

/etc/td-agent/td-agent.conf(fluentd設定ファイル)


type tail
format /^(?[^ ]*) [^ ]* (?[^ ]*) \[(?

fluentd起動

[root@localhost ~]# service td-agent start

mongodbにためてみる

mongodbインストール

[root@localhost ~]# vim /etc/yum.repos.d/10gen.repo
[root@localhost ~]# cat /etc/yum.repos.d/10gen.repo
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
[root@localhost ~]# yum install mongo-10gen-server
[root@localhost ~]# service mongod start

この時点では当然まだmongoは空

[root@localhost ~]# mongo
MongoDB shell version: 2.0.2
connecting to: test
> show dbs;
local (empty)

fluent-plugin-mongoのインストール

[root@localhost ~]# /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mongo

/etc/td-agent/td-agent.conf

ファイルに出力していた部分をmongoに出力するように書き換え


type mongo
host localhost
database fluent
collection nginx


td-agent再起動
[root@localhost ~]# /etc/init.d/td-agent restart

下のメッセージが出る
Starting td-agent: Able to load bson_ext version 1.4.0, but >= 1.5.2 is required.

**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
 You can install the extension as follows:
 gem install bson_ext

 If you continue to receive this message after installing, make sure that the
 bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.


bson_extをインストール後、td-agent再起動
[root@localhost ~]# /usr/lib64/fluent/ruby/bin/gem install bson_ext
[root@localhost ~]# /etc/init.d/td-agent restart

mongodbに出力されているか確認

[root@localhost ~]# curl http://localhost/
[root@localhost ~]# mongo
MongoDB shell version: 2.0.6
connecting to: test
> use fluent
switched to db fluent
> db.nginx.find();
{ "_id" : ObjectId("4fdce3b95ef6b1741f000001"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "151", "referer" : "-", "agent" : "curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.12.7.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2", "response_time" : "\"0.000\"", "time" : ISODate("2012-06-16T19:50:24Z") }

こんな感じでクエリを投げれば404のログだけ抽出できる
> db.nginx.find({code: "404"});