Graylog2 on centos 6.2: easy visual log parsing

Here is a quick install for graylog2. This tool is a bit like sawmill or splunk but fully free ! You may discover some about your infrastructure today :D

Install Centos like usual, graphical interface is not needed ;)

 Ensure you have correct DNS and reverse set-up

This is a fine way to ensure all name resolution will be done properly. In my case :

# host `hostname`
graylog.feedback.eu.org has address 172.16.86.34
graylog.feedback.eu.org has IPv6 address 2001:470:b6db:0:20c:29ff:fe54:ddea
# host 172.16.86.34
34.86.16.172.in-addr.arpa domain name pointer graylog.feedback.eu.org.
# host 2001:470:b6db:0:20c:29ff:fe54:ddea
a.e.d.d.4.5.e.f.f.f.9.2.c.0.2.0.0.0.0.0.b.d.6.b.0.7.4.0.1.0.0.2.ip6.arpa domain name pointer graylog.feedback.eu.org.

You may not have IPv6 ;)

 Add the secondary software repositories

We will need EPEL and passenger repositories. EPEL is a must for me as it provides nearly all that lack on main repository. Passenger is for the Apache ruby module. It will be needed for the web set-up

# rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
[...]
1:epel-release ########################################### [100%]
[root@graylog ~]# yum install http://passenger.stealthymonkeys.com/rhel/6/passenger-release.noarch.rpm
...

Install all needed package

#yum -y install lsb gcc make java-1.6.0-openjdk mongodb-server rubygems.noarch ruby-devel mod_passenger
[...]

Installation can be a bit long… but this way, all prerequisites are installed in one time

 Configure mongodb

Ok only 2 lines need to be changed :

# cat /etc/mongodb.conf
#bind_ip = 127.0.0.1
auth = true

 Start mongo DB and configure accesses

Mongo is a standard service to run :

# /etc/init.d/mongod start
Starting mongod: [ OK ]
# chkconfig mongod on

To give access, we need to create db and users :

# mongo
MongoDB shell version: 1.8.2
connecting to: test
> use admin
switched to db admin
> db.addUser('admin', 'xxx')
{
"user" : "admin",
"readOnly" : false,
"pwd" : "xxx"
}
> db.auth('admin', 'xxx')
1
> use graylog2
switched to db graylog2
> db.addUser('grayloguser', 'xxx')
{
"user" : "grayloguser",
"readOnly" : false,
"pwd" : "xxx"
}
> exit
bye

 Install elasticsearch

This new prerequisite does not exist yet as RPM

# wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.6.zip
# unzip /home/fjacquet/elasticsearch-0.18.6.zip
Archive: /home/fjacquet/elasticsearch-0.18.6.zip
creating: elasticsearch-0.18.6/
[...]
inflating: elasticsearch-0.18.6/bin/plugin 

# /usr/local/elasticsearch-0.18.6/bin/elasticsearch

Will need to make a service wrapper …

Extract graylog2 server

Get the software

# wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
# tar xzvf /tmp/graylog2-server-0.9.6.tar.gz
[...]
# cp graylog2.conf.example graylog2.conf
# mv graylog2-server-0.9.6/ /usr/local/

Trick the configuration location

# ln -s /usr/local/graylog2-server-0.9.6/graylog2.conf /etc/

Fix the root folder

# vim /usr/local/graylog2-server-0.9.6/bin/graylog2ctl
$NOHUP java -jar /usr/local/graylog2-server-0.9.6/graylog2-server.jar &

Starting graylog2-server … As it listen to syslog, you need to run as root ( port < 1024 … )

# /usr/local/graylog2-server-0.9.6/bin/graylog2ctl start
# /usr/bin/nohup: appending output to `nohup.out'

Web front installation

Get the software

# wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
# tar xzvf /tmp/graylog2-web-interface-0.9.6.tar.gz
# mv graylog2-web-interface-0.9.6/ /usr/local/
# cd /usr/local/graylog2-web-interface-0.9.6

Install ruby modules installer

# gem install bundler
Successfully installed bundler-1.0.21
1 gem installed
Installing ri documentation for bundler-1.0.21...
Installing RDoc documentation for bundler-1.0.21...

# bundle install
Fetching source index for http://rubygems.org/
[...]
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Backup and edit configuration files

# cd config
# for i in *.yml; do j=$(echo $i.orig); cp $i $j; done
# vim *yml

yml file must be configured with same login informations for mongo. The rest is likely to stay untouched. New Relic may be or not installed

Start as apache site

# cd /etc/httpd/conf.d
# vim passenger.conf

LoadModule passenger_module modules/mod_passenger.so
<IfModule mod_passenger.c>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.11
PassengerRuby /usr/bin/ruby
PassengerTempDir /var/run/passenger
</IfModule>
<VirtualHost *:80>
    ServerAdmin root@localhost
    DocumentRoot /usr/local/graylog2-web-interface-0.9.6/public
    <Directory /usr/local/graylog2-web-interface-0.9.6/public>
        Allow from all
        Options -MultiViews
    </Directory>
    ErrorLog /var/log/httpd/error.log
    LogLevel warn
    CustomLog /var/log/httpd/access.log combined
</VirtualHost>

Start Apache and enjoy

# /etc/init.d/httpd restart
Stopping httpd: [FAILED]
Starting httpd: [OK]
About these ads
This entry was posted in Features and tagged , , , , . Bookmark the permalink.

7 Responses to Graylog2 on centos 6.2: easy visual log parsing

  1. Elasticseach-servicewrapper
    Download elasticsearch-servicewrapper (tanuki-wrapper) into your elasticserach/bin installation directory and unpack it there
    wget https://github.com/elasticsearch/elasticsearch-servicewrapper/zipball/master
    mv master elasticsearch-servicewrapper.zip && unzip elasticsearch-servicewrapper.zip
    mv elasticsearch-elasticsearch-servicewrapper-*/* . && rm -rf elasticsearch-elasticsearch-servicewrapper-*

  2. pvradu says:

    I have a strange feeling that events are not added to the database. I have successful logins from the graylog server to MongoDB (I know that because I stopped the web interface while testing, and mongod.log shows successful authentication – from the graylog server), I can see syslog packets coming on udp/514 using tcpdump, I have java listening on udp/514, but there’s nothing in my MongoDB nor web interface. Any ideas on why my DB is still empty?

    Thanks!

    • Dick Davies says:

      All the events end up in Elasticsearch, not Mongo (as of 0.9.6). Check the elastic search DB settings are the same in -server and -web-interface.

  3. Tom says:

    I have the same problem, zero message count. I can see messages via tcpdump, and graylog2 is listening, but the message count in mongo is zero. Any ideas?

  4. pvradu says:

    Upgrade it to 0.9.6p1 ..or the release candidate. It’s because you’re not sending the standard syslog format to Graylog server. There’s a switch in the new version’s config file that will solve that.

  5. Anders says:

    i can get it to listen on tcp 514 i only get logs from rsyslog.?? get permision denied in gray log
    FATAL: org.graylog2.Main – Could not start syslog server core thread. Do you have permissions to listen on port 514?
    on ipv6 does it listen..
    root@Partylog2 /etc/modprobe.d# netstat -an | grep 514
    tcp 0 0 127.0.0.1:27017 127.0.0.1:39514 ESTABLISHED
    tcp6 0 0 127.0.0.1:39514 127.0.0.1:27017 ESTABLISHED
    udp6 0 0 :::514 :::*

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s