Hi!

It seems Sparrow project is getting stabilized with version 0.1.*, I don't think I am going to change API radically in the future ... The same for Outthentic which accordingly hit version 0.1.* as well.

So to sum things up:

• Outthentic - is multi purposes, multi languages scenarios framework. You choose between one of the three options when writing scenarios on Outthentic - Perl, Ruby or Bash.

• Who is supposed to use Outthentic? Anybody looking for process automation - developers, IT or devops.

• One of the intriguing of Outthentic feature is "embedded" testing system based on Outthentic::DSL resulting in scenarios report in TAP format. You write your scenario to solve your issues, you test your scenario using Outthentic::DSL - this is indivisible approach.

• Outthentic does not force you to use a "special" language to write your scenarios , you choose Perl, Ruby or Bash ( More languages in plans , Perl6 , huh ? ;). Instead Outthentic provides your some integration, testing facilities to make scenario development process more effective ...

• Sparrow is a kinda orchestration tool to make outthentic scenarios development and distribution process more simple and agile. Outthentic scenarios could be packaged, distributed, grouped and configured to make it easier scenarios usage in a industrial way.

• SparrowHub - is a central repository for reusable multi purposes scenarios. An only few plugins could be found here right now but this will be a good start point for other sparrow plugin developers on how to create a new plugins.

That is it.

Have a useful scenario? Excellent! Wrap it up into sparrow plugin and upload it into SparrowHub! That is so easy ...

--- Alexey Melezhik

Logdog is yet another sparrow plugin to analyze various log files.

One my use logdog for various tasks like finding errors in nginx/apache logs or inspecting suspicious entries in your sshd log file.

One killer feature of logdog is it parsing log lines for the given period of time. Like one can ask to find only entries for last hour or for last 2 days or even 15 minutes. It is very handy as often this is probably what you need - to identify what have happened recently and to not re-examine all the events happened a long time before.

Let's see how one can monitor BAD ssh logins using logdog. The rest part of this blog will look like a formal documentation, but I hope you won't be frustrated! :)

Installing logdog

First of all we need sparrow and a minimal prerequisites:

$ yum install curl
$ cpanm Sparrow
$ sparrow index update
$ sparrow plg install logdog

Now comes configuration part

Configure monitoring suite

 # let's create project - it just a container for sparrow monitoring suites
 $ sparrow project create system
 # sparrow checkpoint is a configurable sparrow plugin 
 $ sparrow check add system ssh-bad-logins
 # let's bind ssh-bad-logins checkpoint to logdog plugin
 $ sparrow check set system ssh-bad-logins logdog
 # and provide configuration:
 $ export EDITOR=nano
 $ sparrow check ini system ssh-bad-logins

Configuration parameters will be:

[logdog]
# this is where sshd keep logs
file = /var/log/auth.log

# we will be interested in only entries related to `bad logins' failures 
filter = Failed password

# as on my server timestamps are in format
# kinds Apr  8 11:35:23 
# this regexp pattern will be use to identify entries times:
time_pattern = (\S+)\s+(\d+)\s+(\S+)\s

# this is required to convert found times ( given by time_pattern )
# into Time Objects and make proper time calculation
time_format = %b %e %T

# This timezone is used on my server
# The list of acceptable timezone
# could be found here
# https://metacpan.org/pod/DateTime::TimeZone::Catalog
timezone = Europe/Moscow

# key_filed is pattern to group found entries 
# as on my server they looks like:
# Failed password for $LOGIN from $IP_ADDRESS port 18367 ssh2
# I want to group them by user login and IP address 
key_field = password for\s+(.*)\s+from\s+(\S+)

# density is additionally to key_field
# it sets number of entries in groups
# if you need to take into account only frequent events  
# you may increase density
# like density = 10 
# means >= 10 bad login attempts for given user from given IP_ADDRESS for 
# given period of time 
density = 1

# I only need a last 10 minutes activity 
history = 10 minutes 



# setting check_mode to zero
# means if more then zero entries are found
# it will be treated as check failure
# yeah, in other words probably someone tries to break your server 
# over ssh
check_mode = zero

Now let's give it a try. But before doing this we need to "generate" some unsuccessful logins:

$ ssh foo@127.0.0.1 
foo@127.0.0.1's password: 
Permission denied (publickey,password).

And so on ... Now let's see if our monitoring suite detect our activity:

$ sparrow check run system ssh-bad-logins  


# cd /root/sparrow/plugins/public/logdog && carton exec 'strun --root ./  --ini /root/sparrow/projects/system/checkpoints/ssh-bad-logins/suite.ini '

/tmp/.outthentic/29141/root/sparrow/plugins/public/logdog/story.t .. 
# history: 10 minutes
# filter: Failed password
# density: 1
# group invalid user foo 127.0.0.1 count: 9
# Apr 12 16:16:02 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:05 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:08 melezhik-pc sshd[29099]: Failed password for invalid user foo from 127.0.0.1 port 55267 ssh2
# Apr 12 16:16:12 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:16 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:19 melezhik-pc sshd[29103]: Failed password for invalid user foo from 127.0.0.1 port 55268 ssh2
# Apr 12 16:16:22 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# Apr 12 16:16:26 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# Apr 12 16:16:29 melezhik-pc sshd[29107]: Failed password for invalid user foo from 127.0.0.1 port 55270 ssh2
# group invalid user bar 127.0.0.1 count: 3
# Apr 12 16:16:38 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
# Apr 12 16:16:41 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
# Apr 12 16:16:44 melezhik-pc sshd[29111]: Failed password for invalid user bar from 127.0.0.1 port 55271 ssh2
ok 1 - output match /lines count: (\d+)/
ok 2 - output match /Failed password/
not ok 3 - zero groups found
1..3

#   Failed test 'zero groups found'
#   at /root/sparrow/plugins/public/logdog/local/lib/perl5/Outthentic.pm line 144.
# Looks like you failed 1 test of 3.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests 

Test Summary Report
-------------------
/tmp/.outthentic/29141/root/sparrow/plugins/public/logdog/story.t (Wstat: 256 Tests: 3 Failed: 1)
Failed test:  3
Non-zero exit status: 1
Files=1, Tests=3,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.25 cusr  0.00 csys =  0.27 CPU)
Result: FAIL

Now, to get things completely automated let's define a cron job staring every 10 minutes:

$ crontab -l

*/10  * * * * sparrow check run  system ssh-bad-logins   --cron

Conclusion

That is it. As we could see logdog is quite flexible tool. As I already told it could be used for various task, probably eventually I have to say something about monitoring nginx errors or something like that :)

Sometimes for some reasons processes on your server work unexpectedly long or don't die on time, this might cause many issues basically because of your server resources start to exhaust.

Stale-proc-check is sparrow plugin to show you if any some "stale" processes exists on your server. It depends on ps utility , so will probably work on many linux/unix boxes ...

Below is short manual.

INSTALL

$ sparrow plg install stale-proc-check

USAGE

Once plugin is installed you need to define configuration for it by using sparrow checkpoint container, which just an abstraction for configurable sparrow plugin.

You need to provide 2 parameters:

• filter - perl regular expression to match a desired process

• history - a time period to determine that processes found are fresh enough

In others words if any processes older then $history parameter found it will be treated as bad situation and check will fail.

Ok, now we are ready to do configuration:

$ sparrow project create system

$ sparrow check add system stale-ssh-sessions

$ sparrow check set system stale-ssh-sessions stale-proc-check

$ export EDITOR=nano && sparrow check ini system stale-ssh-sessions

[stale-proc-check]
# lets find all knife ssh processes running more than halve an hour
filter = knife\s+ssh
history = 30  minutes

In the example here I will be looking all knife ssh processes running more then halve an hour, on our production system it typical when for some reasons knife ssh commands do not die even though a parent process is terminated. Well we also have a bugs :)

Now let's run a check:

$ sparrow check run system stale-ssh-sessions

# running cd /root/sparrow/plugins/public/stale-proc-check && carton
exec 'strun --root ./  --ini
/root/sparrow/projects/system/checkpoints/stale-ssh-sessions/suite.ini
' ...

/tmp/.outthentic/25883/root/sparrow/plugins/public/stale-proc-check/story.t ..
# filter: knife\s+ssh
# history: 30 minutes
# 0 stale processes found
ok 1 - output match /count: (\d+)/
ok 2 - zero stale processes found
1..2
ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.09 cusr
0.00 csys =  0.11 CPU)
Result: PASS

Hurrah, no stale process here ... But at the end of this post let me show a negative case as well. Let's start a few sleep commands and checks if they are still running, indeed they should! :)

$ sleep 1000 &
$ sleep 1000 &
$ sleep 1000 &
$ sleep 1000 &


$ sparrow check add system sleepyheads

$ sparrow check set system sleepyheads stale-proc-check

$ export EDITOR=nano && sparrow check ini system sleepyheads

[stale-proc-check]
# I want to see "sleep commands" only
filter = sleep

# running more then
history = 5  minutes

Now let's see who overstates ( we should wait for about 5 minutes before runing our check ... ) :

 $ sparrow check run system sleepyheads 


# running cd /home/melezhik/sparrow/plugins/public/stale-proc-check && carton exec 'strun --root ./  --ini /home/melezhik/sparrow/projects/system/checkpoints/sleepyheads/suite.ini ' ...

/tmp/.outthentic/5584/home/melezhik/sparrow/plugins/public/stale-proc-check/story.t .. 
# filter: sleep
# history: 5  minutes
# 7 stale processes found
# pid: 3117 command: sleep 1000                        delta: minutes: 5 seconds: 16 
# pid: 3118 command: sleep 1000                        delta: minutes: 5 seconds: 16 
# pid: 3119 command: sleep 1000                        delta: minutes: 5 seconds: 15 
# pid: 3120 command: sleep 1000                        delta: minutes: 5 seconds: 15 
# pid: 3121 command: sleep 1000                        delta: minutes: 5 seconds: 14 
# pid: 3122 command: sleep 1000                        delta: minutes: 5 seconds: 14 
# pid: 3123 command: sleep 1000                        delta: minutes: 5 seconds: 13 
ok 1 - output match /count: (\d+)/
ok 2 - [b] output match 'start_proc_data'
ok 3 - [b] output match 'pid'
ok 4 - [b] output match 'command'
ok 5 - [b] output match 'time'
ok 6 - [b] output match 'delta'
ok 7 - [b] output match 'end_proc_data'
not ok 8 - zero stale processes found
1..8

#   Failed test 'zero stale processes found'
#   at /home/melezhik/sparrow/plugins/public/stale-proc-check/local/lib/perl5/Outthentic.pm line 130.
# Looks like you failed 1 test of 8.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/8 subtests 

Test Summary Report
-------------------
/tmp/.outthentic/5584/home/melezhik/sparrow/plugins/public/stale-proc-check/story.t (Wstat: 256 Tests: 8 Failed: 1)
  Failed test:  8
  Non-zero exit status: 1
Files=1, Tests=8,  1 wallclock secs ( 0.01 usr  0.01 sys +  0.05 cusr  0.01 csys =  0.08 CPU)
Result: FAIL

Minion-check is a sparrow plugin to monitor failed minion tasks.

One could easily verify if any minion jobs are failed for a certain period of time. Which could be important to understand if you have any failures for your long running task executed by minion.

The installation:

$ sparrow plg install minion-check 
$ sparrow project create myhost
$ sparrow check add myhost minion
$ sparrow check set myhost minion minion-check

And the configuration:

Here you need to provide bash command to run your minion command and optionally period of time to look up failed minion tasks:

$ export EDITOR=nano
$ sparrow check ini myhost minion


# bash script to run minion command
command = cd /foo/bar/app/path && carton exec ./app.pl minion

 # check failed tasks for last 5 minutes, 10 hours, 2 days , etc ...
 history = 10 minutes

That' ok. Now let's run a plugin on the host where your minion workers run:

$ sparrow check run  myhost minion 
# running cd /home/vagrant/sparrow/plugins/public/minion-check && carton exec 'strun --root ./  --ini /home/vagrant/sparrow/projects/myhost/checkpoints/minion/suite.ini ' ...

/tmp/.outthentic/18157/home/vagrant/sparrow/plugins/public/minion-check/failed-tasks/story.t ..
ok 1 - stdout is already set
ok 2 - stdout saved to /tmp/.outthentic/18157/3r2uXbq_kM
ok 3 - output match /Q=(1|0)/
ok 4 - output match /(\d+)\s+failed/
ok 5 - stdout is already set
ok 6 - stdout saved to /tmp/.outthentic/18157/HpQ2V5mxm4
# foo_task (default, failed, p0, r0)
# []
# "100 at app.pl line 11, <DATA> line 742.\n"
# 2016-04-01T19:06:24Z (created)
# 2016-04-01T19:08:52Z (started)
# 2016-04-01T19:08:53Z (finished)
ok 7 - output match /(.*)/
ok 8 - output match /finished/
ok 9 - '2016-04-01T19:08:53Z (finished)' match /(\d\d\d\d-\d\d-\d\d)T(\S+)Z.*/
not ok 10 - 0 failed jobs found for period 10 minutes
1..10

#   Failed test '0 failed jobs found for period 10 minutes'
#   at /home/vagrant/sparrow/plugins/public/minion-check/local/lib/perl5/Outthentic.pm line 130.
# Looks like you failed 1 test of 10.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests

Test Summary Report
-------------------
/tmp/.outthentic/18157/home/vagrant/sparrow/plugins/public/minion-check/failed-tasks/story.t (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  10
  Non-zero exit status: 1
Files=1, Tests=10,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.60 cusr  0.05 csys =  0.68 CPU)
Result: FAIL

Running by cron.

Thus, one may monitor minion tasks failures by cron every 10 minutes:

$ crontab -l

*/10 * * * * sparrow check run  myhost minion --cron

Running sparrow check with --cron options means giving output only in case of test failures ...

Hi all!

SparrowHub - is a test/monitoring suites repository. SparrowHub repository contains collections of reusable testing/monitoring suites which are also called plugins. These are test suites for various use cases. From testing web services to checking disk available amount on your server. Sparrow suites are runnable via console script called sparrow which produce TAP output. It could be used in whatever monitoring , testing task and could be easily integrated into existed monitoring / deployment / configuration / automation systems. Sparrow plugins to be written on outthentic DSL and to be extended by Perl.

I have just added two 5 minutes tutorials for those who:

• wants just to use sparrow suites in monitoring / testing purpose
• wants to develop new sparrow plugins

continuous integration using sparrow tool chain

Sparrow is a tool to automate testing infrastructure. Automated testing is essential part of continuous integration processes as it provides fast feedback on cases when something goes wrong.

Consider simple example of how sparrow could be used to build up some basic parts of your infrastructure.

web application development

We have a simple Dancer2 application we are going to deploy on development environment:

app.psgi

#!/usr/bin/env perl
use Dancer2;

get '/' => sub {
    "Hello World!"
};

dance;

Let's keep source code at git repository:

git init
git add app.psgi
git commit -a -m 'my web application'

git remote add origin https://github.com/melezhik/webapp.git
git push -u origin master

development server

We are going to deploy application on dedicated server used for development environment:

ssh dev.server
git clone https://github.com/melezhik/webapp.git
cd webapp
cpanm Dancer2
plackup

In these lines we fetch source code from remote git repository and run dancer application. Good so far. These steps could be automated in various ways ( jenkins, crontab , whatever your favorite CI tool ).

Last command should emit following:

HTTP::Server::PSGI: Accepting connections at http://0:5000/

Which means our application is running.

building up test harness

As we need to ensure that app is running correctly after being deployed we have to add some integration tests for it. With sparrow it is as simple as writing a few lines of code:

git init # let's keep test suite case under git

echo 127.0.0.1:5000 > host

nano get.txt
    200 OK
    Hello World!

echo '{}' > sparrow.json

echo "requires 'swat';" > cpanfile

git add .
git commit -a -m 'basic test suite'
git remote add origin https://github.com/melezhik/webapp-basic-check.git
git push -u origin master

Now when we are done with creating a very simple test suite let's go to development server and create some check points:

ssh dev.server
cpanm Sparrow

echo "basic-check https://github.com/melezhik/webapp-basic-check.git" > ~/sparrow.list      
sparrow index update
sparrow plg install basic-check

sparrow project create webapp
sparrow check add webapp basic
sparrow check set  webapp basic basic-check

sparrow check run webapp basic

Output of last command will be:

# running cd /home/vagrant/sparrow/plugins/private/basic-check && carton exec 'swat ./   ' ...

/home/vagrant/.swat/.cache/30818/prove/00.GET.t ..
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -f -D - '127.0.0.1:5000/'
ok 1 - server returned successful response
ok 2 - output match '200 OK'
ok 3 - output match 'Hello World!'
1..3
ok
All tests successful.
Files=1, Tests=3,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.04 cusr  0.00 csys =  0.07 CPU)
Result: PASS

Ok, we see that our tests succeed and we can continue with development.

adding new feature to web application

Let's add authentication to your application:

app.psgi

#!/usr/bin/env perl

use Dancer2;
use Dancer2::Plugin::Auth::Tiny;

set show_errors => 1;
set session     => 'Simple';


get '/' => sub {
    "Hello World!"
};


get '/public' => sub { return 'public area' };

get '/private' => needs login => sub { return 'private area' };

get '/login' => sub {
    session "user" => "Robin Good";
    return "login and to back to " . params->{return_url};
};

get '/logout' => sub {
    app->destroy_session;
    redirect uri_for('/public');
};

dance;

Let's create a check list we need to ensure:

• when user hits /public route he sees 'public area'
• when user hits /private route for a first time he gets redirected to /login page and then gets a session cookies
• when user hits /private routes for a second time he sees 'private area'

Now create another suite case for the stories above, we are going to keep under another git repository and then deliver tests as another sparrow plugin as we did with the basic suite case:

creating test suite skeleton

git init
echo 127.0.0.1:5000 > host
echo '{}'> sparrow.json
echo "requires 'swat';" > cpanfile
git add .
git commit -a -m 'authentication test suite'
git remote add origin https://github.com/melezhik/webapp-auth-check.git
git push -u origin master

public area test

mkdir public

nano public/get.txt
    200 OK
    public area

private area first time

mkdir private-first-time
nano private-first-time/hook.pm

    run_swat_module( GET => '/logout' );
    run_swat_module( GET => '/private' , { auth => 0 } );
    set_response('done');

echo done > private-first-time/get.txt

mkdir logout/
echo swat_module=1 > logout/swat.ini
echo 200 OK > logout/get.txt

mkdir private/

nano private/swat.ini

    swat_module=1
    curl_params="-b ${test_root_dir}/cook.txt"

nano private/get.txt

    generator: [ module_variable('auth') ? 'private area' : 'login and to back to' ]

private area second time

mkdir private-second-time
nano private-second-time/hook.pm

    run_swat_module( GET => '/login' );
    run_swat_module( GET => '/private' , { auth => 1 } );
    set_response('done');

mkdir login/

nano login/swat.ini

    swat_module=1
    curl_params="-c ${test_root_dir}/cook.txt"

echo 200 OK > login/get.txt

And finally we can commit changes to git and push them to remote.

git add .
git commit -a -m 'authentication check test suite'
git push

testing new features

As we did with basic test suite we now are able to run tests for authentication feature:

ssh dev.server
echo "auth-check https://github.com/melezhik/webapp-auth-check.git" >> ~/sparrow.list      
sparrow index update
sparrow plg install auth-check

sparrow check add webapp auth
sparrow check set  webapp auth auth-check

sparrow check run webapp auth

The output of the second test suite will be:

# running cd /home/vagrant/sparrow/plugins/private/testapp2 && carton exec 'swat ./   ' ...

/home/vagrant/.swat/.cache/1085/prove/private-first-time/00.GET.t ...
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -f -D - '127.0.0.1:5000/logout'
ok 1 - server returned successful response
ok 2 - output match '200 OK'
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -b /home/vagrant/.swat/.cache/1085/prove/cook.txt -f -D - '127.0.0.1:5000/private'
ok 3 - server returned successful response
ok 4 - output match 'login and to back to'
ok 5 - server response is spoofed
# response saved to /home/vagrant/.swat/.cache/1085/prove/xDQuPGlVog
ok 6 - output match 'done'
1..6
ok
/home/vagrant/.swat/.cache/1085/prove/public/00.GET.t ...............
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -f -D - '127.0.0.1:5000/public'
ok 1 - server returned successful response
ok 2 - output match '200 OK'
ok 3 - output match 'public area'
1..3
ok
/home/vagrant/.swat/.cache/1085/prove/private-second-time/00.GET.t ..
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -c /home/vagrant/.swat/.cache/1085/prove/cook.txt -f -D - '127.0.0.1:5000/login'
ok 1 - server returned successful response
ok 2 - output match '200 OK'
# trying ... curl -X GET -k --connect-timeout 20 -m 20 -L -b /home/vagrant/.swat/.cache/1085/prove/cook.txt -f -D - '127.0.0.1:5000/private'
ok 3 - server returned successful response
ok 4 - output match 'private area'
ok 5 - server response is spoofed
# response saved to /home/vagrant/.swat/.cache/1085/prove/hvtwlrTqEZ
ok 6 - output match 'done'
1..6
ok
All tests successful.
Files=3, Tests=15,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.17 cusr  0.01 csys =  0.22 CPU)
Result: PASS

Examining your test infrastructure

Running project show command one may obtain available check points for webapp project. It is handy when eventually a lot test suites come in here.

sparrow project show webapp
[project webapp]

    [checkpoints]

      auth
      basic

Summary

Sparrow tool chains has following features:

• integration tests suites are decoupled from application code

• tests could be grouped by types/environments and developed/delivered as dedicated test suites - aka sparrow plugins

• sparrow test infrastructure is easy to bootstrap on different environments (dev,test,prod) - everybody involved is able now to run desired test suite with easiness

• swat output is intended to simplify debugging/troubleshooting process, in most cases you just rerun curl command ( against tested server ) with parameters extracted from test output.

PS

paper source code

the source code for this post could be found here

code examples :

• webapp
• webapp-basic-check
• webapp-auth-check

The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun. Is there any thing whereof it may be said, See, this is new? it hath been already of old time, which was before us. (Ecclesiastes 1:9,10)

Outthentic is a tool for rapid plugin developments. When I say "plugin" I mean a little utility to solve a specific task.

Having a risk to reinvent the wheel, from time to time I still start to create some useful stuff, at least useful from my point of view ...

Disk usage monitoring

This little plugin does what is stated at the header:

• checks disk space in partition by partition basis
• alerts in case of shortage
• a threshold parameter could be set and equal 80 default, which means before 80 percentage of disk usage we are ok.

Sparrow

This plugin is written on outthentic and better way to install is sparrow - outthentic plugin manager. So here we go:

# a basic prerequisites  
apt-get install curl
cpanm Sparrow

# now we install plugin
sparrow index update
sparrow plg install df-check
sparrow project create system
sparrow check add system disk
sparrow check set system disk df-check
sparrow check ini system disk # skip this step if you want default settings

Once plugin is installed one my run it:

sparrow check run  system disk | perl -n -e 'print "    ", $_'
# running cd /home/vagrant/sparrow/plugins/public/df-check && carton exec 'strun --root ./  --ini /home/vagrant/sparrow/projects/system/checkpoints/disk/suite.ini ' ...

/tmp/.outthentic/9034/home/vagrant/sparrow/plugins/public/df-check/disk-shortage/story.t ..
ok 1 - perl /home/vagrant/sparrow/plugins/public/df-check/disk-shortage/story.pl succeeded
ok 2 - stdout saved to /tmp/.outthentic/9034/gJBohrjXGg
# threshhold: 93
# verify ... /dev/sda1
# verify ... udev
# verify ... tmpfs
# verify ... tmpfs
# verify ... tmpfs
# verify ... tmpfs
# verify ... none
# verify ... none
ok 3 - output match /(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)/
ok 4 - enough disk space (74%) on /dev/sda1
ok 5 - enough disk space (0%) on udev
ok 6 - enough disk space (1%) on tmpfs
ok 7 - enough disk space (1%) on tmpfs
ok 8 - enough disk space (0%) on tmpfs
ok 9 - enough disk space (0%) on tmpfs
ok 10 - enough disk space (90%) on none
ok 11 - enough disk space (90%) on none
1..11
ok
All tests successful.
Files=1, Tests=11,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.04 cusr  0.00 csys =  0.06 CPU)
Result: PASS

And finally make it cronable

crontab -e 
*/30 * * * *  sparrow check run  system disk --cron

Is it a better wheel?

Well, ... I think so )))), at least my thoughts are:

• it is written on perl
• it has clear and simple installation and configuration layout
• it has minimal dependencies - a few perl modules and df utility itself
• it is hack-able - consider some improvements? check out a source code and make a MR!

Once upon a Friday

Are you a smart web developer using Mojolicious, Dancer2, Kelp, Limper or just CGI ? Sure, yes you are! ;-) You have just finished yet another web application ready to get deployed and awaited with impatience by your customers. All your 't/' are fine, you for sure use Plack::Test or similar tools to tests your application not only in internal but external way. Everything is going to be fine, even though it's Friday today and is not probably good time to make a release, but you so anxious to see a results and get users input, so you are going to talk to John - devops guy and send him a git URL where your source code lives in so he could start the deploy procedure ...

... But, yes, there are some "hurdles" always. John kindly asking you to provides some monitoring capabilities for your freshly baked application, so he could make it sure it won't break on weekend silently without alarming anybody ...

Ah, yeah you talk about monitoring ... ?

"Ah, monitoring John, yes, sure ..." - thoughts have passed in your mind.

"Well John, you could try a curl or check_http or something else and write your own bash script to get this done. It's easy for sure, you only have to take care about cookies get sent back upon successful authentication, you know curl "knows" cookies, you know cookie as well, do you? Also there is a database endpoint you need to call periodically to ensure application does not lost DB connection.

That's it!

And ... yeah, you know the a route for this called GET /database-info ... - Just use it!"

Saying all these phrases you are looking at John's face and have noticed as him is getting unhappy, a moment after you start to realize why.

" Ah, Alex ( this is your name ) - ... Cookies, cookies ... what you are talking about ? Sorry, I am pretty swamped right now, you know a couple of applications coming from next department have been waiting to get deployed since yesterday."

"I am really sorry, but can't do what you talk about right now. What if you come to me on Monday, and we sit together and drop some tests for your application monitoring? You know I'd be happy to deploy it, but I need some tests first ..."

Now it's time for you to be unhappy ...

Swat / Sparrow - Don't be unhappy

Swat is a DSL for rapid web tests development. And sparrow is the tool to make some "glue" to push your swat tests into production.

Let's see how we could use both of these tools to make Alex life more happy and letting him find better collaboration with his colleagues.

Alex's application besides that cool and important for customers, exposes some essential endpoints we have to talk about a little bit:

• POST /login
• GET /database-status

The first one, obviously is for letting user sign in and should be considered as vital part of the application. Upon successfull login server returns session cookie and redirect user to profile page with greetings:

 Hello user! This is profile page

And the second one, is not intended to be accessed by human - as it name hints - but by ... right monitoring script! This endpoint was accidentally named by Alex as `GET /database-info' in his conversation with John.

Upon hitting /database-status we will have:

database is running!

Which means connection between application and database server is perstist and we don't have to worry about it.

Ok, having this we could drop some tests for two routes, in a minimal time and with minimal efforts:

First of all let's create a directory to hold our tests scenarios:

$ mkdir app-monitoring
$ cd app-monitoring

Then let's define mentioned routes:

$ mkdir login/
$ mkdir database-status/

Now, we are almost done. We need to define a content which we expect to get when request the routes.

One for login:

$ echo '200 OK' > login/post.txt
$ echo 'Hello user! This is profile page' >> login/post.txt

And one for database-status

$ echo '200 OK' > database-status/get.txt
$ echo 'database is running!' >> database-status/get.txt

And, yes, our application send cookies back and make http redirect to /profile page upon successful login, so we need to take care about this:

$ nano login/swat.ini
curl_params="-d username=$username -d password=$password --cookie-jar ${test_root_dir}/cookie.txt"

And finally let run our tests manually to see they works:

username=alex password=123456 swat ./ 127.0.0.1:3000

vagrant@Debian-jessie-amd64-netboot:~/projects/papers/blogs-perl-org/app-monitoring$ username=alex password=123456 swat ./ 127.0.0.1:3000
/home/vagrant/.swat/.cache/18844/prove/database-status/00.GET.t ..
ok 1 - GET 127.0.0.1:3000/database-status succeeded
# response saved to /home/vagrant/.swat/.cache/18844/prove/sL6VpL2brR
ok 2 - output match '200 OK'
ok 3 - output match 'database is running!'
1..3
ok
/home/vagrant/.swat/.cache/18844/prove/login/00.POST.t ...........
ok 1 - POST 127.0.0.1:3000/login succeeded
# response saved to /home/vagrant/.swat/.cache/18844/prove/CZEawnHwv_
ok 2 - output match '200 OK'
ok 3 - output match 'Hello user! This is profile page'
1..3
ok
All tests successful.
Files=2, Tests=6,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.10 cusr  0.00 csys =  0.12 CPU)
Result: PASS

Hurrah! We have just succeeded with monitoring script ready to use, but how we deliver it to John?

Little sparrow to ship your monitoring scripts

Well, sparrow is another part of this story. Not that long, as sparrow is easy to set up and use.

In a few words sparrow acts like cpan client for swat test suites. One may package swat tests into sparrow plugin and upload it into SparrowHub - swat test suites repository, so that others could use it.

This is what we need to do ...

First of all let's install sparrow:

$ sudo cpanm Sparrow

Then get registered at SparrowHub and have an API token required for sparrow client authentication.

$ cat ~/sparrowhub.json

{
    "user"  : "alex",
    "token" : "here will be UUID string"
}

And finally upload a plugin for your swat tests suite:

$ cd app-monitoring
$ nano sparrow.json

Sparrow.json file is like Build.PL or Makefile.PL for your swat test suite. It provides some meta information for plugin successfully gets uploaded into SparrowHub.

This is minimal information we need to provide:

$ cat sparrow.json

{
    "version" => "0.0.1"
    "name" => "alex-web-test",
    "description" => "this is a great plugin to monitor alex web application!"
}

OK, we are almost setup. A final step, need to declare CPAN modules dependencies if have any. We are going to use cpanfile / carton for this. A minimal dependency we have here is swat itself, as we run our tests by swat.

$ cd app-monitoring
$ nano cpanfile

require 'swat';

Now we ready to share our plugin via SparrowHub.

$ cd app-monitoring
$ sparrow plg upload

This is the end of Alex's story. And now let's go back to John.

This is a sparrow, John!

Alex said merrily coming into his room. "Now it's really easy to setup monitoring you asked me!"

All that John have to do is to grab his keyboards and type a few commands.

First of all, guess what? yeah - `sudo cpanm Sparrow', as he needs it too!

Then create some monitoring setup.

$ sparrow project create alex-project
$ sparrow check add alex-project brand-new-web-service

Last two command just create a sparrow project for Alex's application and checkpoint called `brand-new-web-service' to run monitoring checks against it.

Then John probably need to install a plugin Alex just uploaded into SparrowHub. No registration required to install sparrow plugins, as it absolutely free stuff:

$ sparrow index update
$ sparrow plg install alex-web-test

Then having plugin installed John need to configure a checkpoint for it:

$ sparrow check set alex-project brand-new-web-service -u 127.0.0.1 -p alex-web-test

Having this means web application will be accessible at http://127.0.0.1 and will be tested by alex-web-test sparrow plguin test suite.

And finally let's set credentials required for tests:

$ sparrow check set_swat alex-project brand-new-web-service

    username=whatever_test_user
    password=test_user_password

Let cron do it's job

Now it's safe for John to put Alex application into production and go to home:

$ crontab -l

MAILTO='john@company.com'

*/10 * * * * sparrow check run alex-project brand-new-web-service --cron

This will run monitoring checks every 10 minutes and in case of errors a swat test report will be emailed back to John.

PS

That's it!

--- Happy new year and have fun tests with Swat and Sparrow!

INTRO

It's been so many times when I wrote about swat and sparrow - a tool-chain for rapid web test automation development .

Well, it's time to represent a SparrowHub - a central repository of sparrow plugins.

About swat and sparrow in few words.

SWAT

Swat - is a web test automation framework for quick and simple web test development. There are a plenty of documentation at swat project page HOW to write swat test suites and a lot examples as well.

SPARROW

Sparrow was designed as evolution for swat related development. Sparrow provides infrastructure for swat related projects. It means you could orchestrate and manage various swat tests and run them as well against dedicated web services. Right now it's quite simple console client with ordinary, naive look but I have a bigger plans for it ;-)

SPARROW PLUGINS

Now, let me tell you about sparrow plugins - an essential part of swat/sparrow ecosystem. Sparrow plugins are shareable swat tests suites. Like a bit of tests logic for some application. Like nginx, apache, tomcat or some REST API - whatever you could test over http.

Ok. One could write plugins and then upload them into central repository and finally someone else could enjoy the fruit of his great job by installing plugin and running swat tests from it!

All the workflow to upload/download/run sparrow plugins is provided by sparrow client.

Think about it as of CPAN repository / CPAN client for PERL, but in context of test automation.

SparrowHub and CPAN

Wait, how does sparrowhub, sparrow relates CPAN? Would not it be easier to ship sparrow plugins as regular CPAN modules?

Well yes an no ...

Sparrow plugins are very like cpan modules, internally they might utilize a Carton to declare perl dependencies a plugin author wants to use in a test suite. But after all sparrow plugins are not CPAN modules, they are much simpler in way of distribution process.

Also swat "utilize" PERL as the way to extent your tests, but swat it self is yet another DSL, not PERL. For this and other reasons I decided not to use CPAN and create a dedicated package management system but still fine integrated with PERL and CPAN.

How can I help?

Swat , sparrow are still quite young and could be considered in beta stage, although I work hard to make them mature and stable, and of course it's possible to start playing with this system right now. What you could do:

• Install Sparrow
• Get registered at https://sparrowhub.org
• Install and play with existed plugins ( a few ones now, but see below )
• Write your own ( cool and useful ) sparrow plugin and share with others - by uploading to SparrowHub
• Report issues, features , ideas to swat/sparrow git projects
• Join a contributors team

Whatever you suggest to do to make swat/sparrow better - you are welcome! ;-)

Why?

Probably it'd be the first section of this post ...

I think I'd like to start web tests shareable automation in/for PERL community, so that many people ( not only those using PERL ) could create and distribute automated tests in easy and smart way.

Hi! This is sparrow / swat related article http://habrahabr.ru/post/272245/ written on Russian language.

Hi, this is very simple and short example, but quite expository.

install sparrow

$ sudo apt-get install curl
$ sudo cpanm Sparrow

installs nginx sparrow plugin

$ sparrow index update
$ sparrow plg install swat-nginx

setup monitored host

$ sparrow project foo create
$ sparrow check add foo nginx-server
$ sparrow check set foo nginx-server -p swat-nginx -u 127.0.0.1

and finally ... run test suite

    $ sparrow check run foo nginx-server

    # sparrow environment initialzed at /home/vagrant/sparrow
    /home/vagrant/.swat/.cache/20277/prove/00.GET.t ..
    ok 1 - GET 127.0.0.1/ succeeded
    # response saved to /home/vagrant/.swat/.cache/20277/prove/jgxseRgqg2
    ok 2 - output match '200 OK'
    ok 3 - output match /Server: nginx\/(\S+)/
    ok 4 - valid nginx version: 1.6.2
    1..4
    ok
    All tests successful.
    Files=1, Tests=4,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.05 cusr  0.00 csys =   0.08 CPU)
    Result: PASS

PS

Visit https://github.com/melezhik/sparrow to know more.