Logdog is yet another sparrow plugin to analyze various log files.
One my use logdog for various tasks like finding errors in nginx/apache logs or inspecting suspicious entries in your sshd log file.
One killer feature of logdog is it parsing log lines for the given period of time. Like one can ask to find only entries for last hour or for last 2 days or even 15 minutes. It is very handy as often this is probably what you need - to identify what have happened recently and to not re-examine all the events happened a long time before.
Let's see how one can monitor BAD ssh logins using logdog. The rest part of this blog will look like a formal documentation, but I hope you won't be frustrated! :)
Installing logdog
First of all we need sparrow and a minimal prerequisites:
$ yum install curl
$ cpanm Sparrow
$ sparrow index update
$ sparrow plg install logdog
Now comes configuration part
Configure monitoring suite
# let's create project - it just a container for sparrow monitoring suites
$ sparrow project create system
# sparrow checkpoint is a configurable sparrow plugin
$ sparrow check add system ssh-bad-logins
# let's bind ssh-bad-logins checkpoint to logdog plugin
$ sparrow check set system ssh-bad-logins logdog
# and provide configuration:
$ export EDITOR=nano
$ sparrow check ini system ssh-bad-logins
Configuration parameters will be:
[logdog]
# this is where sshd keep logs
file = /var/log/auth.log
# we will be interested in only entries related to `bad logins' failures
filter = Failed password
# as on my server timestamps are in format
# kinds Apr 8 11:35:23
# this regexp pattern will be use to identify entries times:
time_pattern = (\S+)\s+(\d+)\s+(\S+)\s
# this is required to convert found times ( given by time_pattern )
# into Time Objects and make proper time calculation
time_format = %b %e %T
# This timezone is used on my server
# The list of acceptable timezone
# could be found here
# https://metacpan.org/pod/DateTime::TimeZone::Catalog
timezone = Europe/Moscow
# key_filed is pattern to group found entries
# as on my server they looks like:
# Failed password for $LOGIN from $IP_ADDRESS port 18367 ssh2
# I want to group them by user login and IP address
key_field = password for\s+(.*)\s+from\s+(\S+)
# density is additionally to key_field
# it sets number of entries in groups
# if you need to take into account only frequent events
# you may increase density
# like density = 10
# means >= 10 bad login attempts for given user from given IP_ADDRESS for
# given period of time
density = 1
# I only need a last 10 minutes activity
history = 10 minutes
# setting check_mode to zero
# means if more then zero entries are found
# it will be treated as check failure
# yeah, in other words probably someone tries to break your server
# over ssh
check_mode = zero
Now let's give it a try. But before doing this we need to "generate" some unsuccessful logins:
$ ssh foo@127.0.0.1
foo@127.0.0.1's password:
Permission denied (publickey,password).
And so on ... Now let's see if our monitoring suite detect our activity:
$ sparrow check run system ssh-bad-logins
# cd /root/sparrow/plugins/public/logdog && carton exec 'strun --root ./ --ini /root/sparrow/projects/system/checkpoints/ssh-bad-logins/suite.ini '
/tmp/.outthentic/29141/root/sparrow/plugins/public/logdog/story.t ..
# history: 10 minutes
# filter: Failed password
# density: 1
# group invalid user foo 127.0.0.1 count: 9
# Apr 12 16:16:02 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:05 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:08 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:12 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:16 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:19 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:22 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# Apr 12 16:16:26 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# Apr 12 16:16:29 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# group invalid user bar 127.0.0.1 count: 3
# Apr 12 16:16:38 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
# Apr 12 16:16:41 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
# Apr 12 16:16:44 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
ok 1 - output match /lines count: (\d+)/
ok 2 - output match /Failed password/
not ok 3 - zero groups found
1..3
# Failed test 'zero groups found'
# at /root/sparrow/plugins/public/logdog/local/lib/perl5/Outthentic.pm line 144.
# Looks like you failed 1 test of 3.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests
Test Summary Report
-------------------
/tmp/.outthentic/29141/root/sparrow/plugins/public/logdog/story.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 3
Non-zero exit status: 1
Files=1, Tests=3, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.25 cusr 0.00 csys = 0.27 CPU)
Result: FAIL
Now, to get things completely automated let's define a cron job staring every 10 minutes:
$ crontab -l
*/10 * * * * sparrow check run system ssh-bad-logins --cron
Conclusion
That is it. As we could see logdog is quite flexible tool. As I already told it could be used for various task, probably eventually I have to say something about monitoring nginx errors or something like that :)
Dev & Devops
---
Then I beheld all the work of God, that a man cannot find out the work that is done under the sun: because though a man labour to seek it out, yet he shall not find it; yea further; though a wise man think to know it, yet shall he not be able to find it.
(Ecclesiastes 8:17)