Results tagged “sparrow”

How to list BitBucket repositories with sparrow

Listing BitBucket repositories could be annoying task even though BitBucket exposes a Rest API for this, and the reason for it is pagination - BitBucket sends result back spited by pages, so you need to request a next page till the end. This is hard to automate and prevent me form using BitBucket API directly.

Well, I have dropped a small sparrow plugin to handle with this task. At least it works for me. It lists ( in plain text format ) all the repositories for given project and team. There a lot of option of plugin you will find at documentation but the usual workflow is:

install plugin

$ sparrow plg install bitbucket-repo-list

run plugin

Here you lists repositories for given project and team. You should supply your Bitbucket credentials to request team/project information:

$ sparrow plg run bitbucket-repo-list \
--param login=superuser --param password=keep-it-secret \
--param team=heroes \
--param project=humans

That is it. Hopefully will be useful for someone deal with BitBucket repositories.

An informal comparison of sparrow and ansible eco systems

Hi!

Here is my latest post about Sparrowdo configuration management tool written in Perl6 - Sparrow plugins vs ansible modules - an attempt of informal comparison of sparrow and ansible eco systems.

Regards.

Alexey

Sparrow plugins evolution

Hi! Sparrowdo is a modern configuration management tool written on Perl6. If you wonder how sparrowdo makes a difference you may read this article - Sparrow plugins evolution - an informal introduction to core part of sparrowdo - sparrow plugins.

A blog site for Sparrowdo project

Hi ! Sparrowdo is lightweight and very flexible configuration management system written on Perl6. I have started a blog site where going to drop technical notes and others news related sparrowdo project, welcome to https://sparrowdo.wordpress.com !

Testing Mojolicious applications with Sparrow

A bits of theory

In this post I am going to show how you can use Sparrow to test Mojolicious applications.

Sparrow approach to test things differs from convenient unit tests approach, practically this means:

  • A tested code is treated as black box rather than unit test way where you rely deeply on inner application structure.

  • A sparrow test suites are not a part of CPAN distribution ( the one you keep under t/* )

  • A sparrow test suite code is decoupled from tested application code and is better to be treated as third party tests for your application

  • Sparrow tests suites have it's own life cycle and get released in parallel with tested application

  • Sparrow acts like toolchain to choose proper tools and then analyze script output

  • IMHO writing sparrow tests sometimes is much simpler and takes less efforts and time, but I don't say you don't need a unit tests but sometime it worths it to take an alternative testing approaches / tools and see the difference , so if you are interested , please read below ...

Ok, let's go to the practical example.

Simple test

A Mojolicious comes with some handy tools to invoke a http requests against web application.

Consider simple mojolicious code:

#!/usr/bin/env perl

use Mojolicious::Lite;

get '/' => {text => 'hello world'};

app->start;

Now we can quickly test a GET / route with help of mojolicious get command:

./app.pl get /
[Sun Dec 11 17:23:38 2016] [debug] GET "/"
[Sun Dec 11 17:23:38 2016] [debug] 200 OK (0.000456s, 2192.982/s)
hello world

That's ok. This is going to be a base for out first sparrow test:

$ nano story.bash

$project_root_dir/app.pl get /


$ nano story.check

hello world


$ strun

/ started

[Sun Dec 11 17:45:28 2016] [debug] GET "/"
[Sun Dec 11 17:45:28 2016] [debug] 200 OK (0.000469s, 2132.196/s)
hello world
ok      scenario succeeded
ok      output match 'hello world'
STATUS  SUCCEED

What we have done here.

  • created a story to run GET / against mojolicious application - file named story.bash
  • created a story check file to validate data returned from http request.
  • finally run a test suite ( story ) with the help of so called story runner - strun

More about stories and check files could be found at Outthentic - a module for execution sparrow scripts.

Consider a negative result here, when test fails. For this let's change a check file to express we need another string returned from calling GET / route:

$ nano story.check

hello sparrow


$ strun

/ started

[Sun Dec 11 18:18:07 2016] [debug] GET "/"
[Sun Dec 11 18:18:07 2016] [debug] 200 OK (0.001237s, 808.407/s)
hello world
ok      scenario succeeded
not ok  output match 'hello sparrow'
STATUS  FAILED (256)

Debugging test scripts

As our test is just bash script it easy to add some debugging facilities into it:

$ nano story.bash

set -x

$project_root_dir/app.pl get /

And run the script:

/ started

++ /home/melezhik/projects/mojolicious-app-smoke/app.pl get /
[Tue Dec 13 13:16:07 2016] [debug] GET "/"
[Tue Dec 13 13:16:07 2016] [debug] Routing to a callback
[Tue Dec 13 13:16:07 2016] [debug] 200 OK (0.000328s, 3048.780/s)
hello world
ok  scenario succeeded
ok  output match 'hello world'
STATUS  SUCCEED

Right now it looks pretty useless but becomes very handy for more complex scripts.

Splitting test suite on many simple tests

As your application grows it comprises many routes, let's how we organize our test suite layout to test them all.

$ cat app.pl

#!/usr/bin/env perl

use Mojolicious::Lite;

get '/' => sub {
  my $c = shift;
  $c->render( text => 'welcome page')

};

get '/hello' => sub  {

  my $c = shift;
  $c->render( text => 'hello '.($c->param('name')))

};

get '/bye' => sub {

  my $c = shift;
  $c->render( text => 'bye '.($c->param('name')))

};

app->start;

Here is the story modules:

$ mkdir -p modules/welcome-page modules/hello modules/buy

# welcome page
$ echo '$project_root_dir/app.pl get /' > modules/welcome-page/story.bash
$ echo 'welcome page' > modules/welcome-page/story.check

# GET /hello
$ echo '$project_root_dir/app.pl get /hello?name=sparrow' > modules/hello/story.bash
$ echo 'hello sparrow' > modules/hello/story.check


# GET /bye
$ echo '$project_root_dir/app.pl get /bye?name=sparrow' > modules/bye/story.bash
$ echo 'bye sparrow' > modules/bye/story.check

And main story-container to call them all:

$ echo 'Smoke tests for app.pl' > meta.txt
$ nano hook.bash

run_story welcome-page
run_story hello
run_story bye


$ strun

/ started

Smoke tests for app.pl

/modules/welcome-page/ started

[Sun Dec 11 19:34:08 2016] [debug] GET "/"
[Sun Dec 11 19:34:08 2016] [debug] Routing to a callback
[Sun Dec 11 19:34:08 2016] [debug] 200 OK (0.00091s, 1098.901/s)
welcome page
ok      scenario succeeded
ok      output match 'welcome page'

/modules/hello/ started

[Sun Dec 11 19:34:09 2016] [debug] GET "/hello"
[Sun Dec 11 19:34:09 2016] [debug] Routing to a callback
[Sun Dec 11 19:34:09 2016] [debug] 200 OK (0.000871s, 1148.106/s)
hello sparrow
ok      scenario succeeded
ok      output match 'hello sparrow'

/modules/bye/ started

[Sun Dec 11 19:34:09 2016] [debug] GET "/bye"
[Sun Dec 11 19:34:09 2016] [debug] Routing to a callback
[Sun Dec 11 19:34:09 2016] [debug] 200 OK (0.001051s, 951.475/s)
bye sparrow
ok      scenario succeeded
ok      output match 'bye sparrow'
STATUS  SUCCEED

What we have done here.

  • created 3 story modules each for a route (GET /, GET /hello , GET /bye )
  • create a story-container to call a story modules
  • run strun to execute test suite

More about story modules and story containers be found at Outthentic - a module for execution sparrow scripts.

Parameterizing test suite

We hardcoded a string sparrow get passed as parameter to routes GET /hello and GET /buy. Let's parametrize out test suite:

$ nano modules/hello/story.bash

name=$(story_var name)
$project_root_dir/app.pl get '/hello?name='$name


$ nano hook.bash

run_story welcome-page
run_story hello name Mojolicious
run_story bye

And then run our test suite:

# some output truncated ...

/modules/hello/ started

[Mon Dec 12 12:42:04 2016] [debug] GET "/hello"
[Mon Dec 12 12:42:04 2016] [debug] Routing to a callback
[Mon Dec 12 12:42:04 2016] [debug] 200 OK (0.000845s, 1183.432/s)
hello Mojolicious
ok      scenario succeeded
not ok  output match 'hello sparrow'

/modules/bye/ started

Obviously our test failed as we need to change a check list:

$ nano modules/hello/story.check

generator: [ "hello ".(story_var('name')) ]

Generators are way to create story check list in runtime. More on this read at Outthentic doc pages.

story_var is bash helper function letting our stories to get the parameter they are called from story container. By the way as sparrow is kind language agnostic framework we could write our code on plain Perl:

$ nano modules/hello/story.pl

my $name =  story_var('name');

my $cmd = project_root_dir()."/app.pl get '/hello?name=$name'";

print `$cmd`

Or even Ruby:

$ nano modules/hello/story.rb

name =  story_var 'name' 

puts `#{project_root_dir}/app.pl get '/hello?name=#{name}'`

I did not tell you, one day I am going to add Python support ... :)

Ok, now let's make our name parameter configurable via configuration file

Configuring test suites

Sparrow test suites maintain three well known configuration formats:

  • Config::General ( probably most convenient way to describe various configurations )
  • JSON
  • YAML

Let's go with Config::General. As our example quite trivial the configuration won't be too complicated:

$ nano suite suite.ini

name Sparrow


$ nano hook.bash

name=$(config name)
run_story welcome-page
run_story hello name $name
run_story bye name $name

We can even use nested configuration parameters:

$ nano suite suite.ini

<name>
  bird Sparrow
  animal Fox
</name>


$ nano hook.bash

bird_name=$(config name.bird)
animal_name=$(config name.animal)

run_story welcome-page
run_story hello name $bird_name
run_story bye name $animal_name

And finally we can override parameters via command line:

$ strun --param name.animal='Bear'

As I said we could use another configuration formats, like for example JSON:

$ nano suite.json

{

  "name" : {
    "bird"    : "sparrow",
    "animal"  : "bear"
  }
}


$ strun --json suite.json

Processing output data

Sometimes we need to process output data to make it testable via Sparrow. It's very common when dealing with application emitting JSON:

$ cat app.pl

#!/usr/bin/env perl

use Mojolicious::Lite;

get '/echo-name' => sub {
  my $c = shift;
  $c->render( json => { name => $c->param('name') } )

};

app->start;

Sparrow does not provide any built in capabilities to parse JSON, but it rather acts as tool where one can add any desired modules into "pipeline":

$ nano hook.bash

name=$(config name)

run_story echo-name name $name

$ mkdir -p modules/name

$ nano modules/echo-name/story.bash

name=$(story_var name)

$project_root_dir/app.pl get '/echo-name?name='$name \
| perl -MJSON -e 'print decode_json(join "", <STDIN>)->{name}'

Now let's run the test:

/ started

Smoke tests for app.pl

/modules/echo-name/ started

[Tue Dec 13 12:00:18 2016] [debug] GET "/echo-name"
[Tue Dec 13 12:00:18 2016] [debug] Routing to a callback
[Tue Dec 13 12:00:18 2016] [debug] 200 OK (0.00054s, 1851.852/s)
sparrow
ok  scenario succeeded
ok  output match 'sparrow'
STATUS  SUCCEED

This approach could be generalized to any data processing like YMAL/XML/CSS. Instead of defining data parsing at test logic we filter/process output data to "map" it to sparrow testing format - just a lines of text where we could make regexp/text search.

Ok let's keep moving. Prepare our test suite for distribution.

Distributing tests

A one thing we should pay attention to. A mojolicious application we write tests for in practice is distributed separately from sparrow test suite. There are some cases:

  • an application gets installed and launched as script available at system $PATH
  • an application gets tested via some CI pipelines, for example Travis

Whatever case we consider it's not hard to adopt our test suite to new reality. For example if there is script called our_mojolicious_application we just need to change small bit of code to rely on system wide installation instead of local:

$ nano modules/hello/story.bash

name=$(story_var name)

our_mojolicious_application get '/hello?name='$name

That is it.

Declaring dependencies

Sparrow works smoothly with Perl/Ruby well known dependency managers to work out on this, just create a cpanfile with dependencies in case of Perl:

$ nano cpanfile

requires 'JSON'; # indeed not required for recent Perls

Providing test suite meta information

This is important point where we define some data to make it possible upload our test suite to SparrowHub - a sparrow scripts repository

$ nano sparrow.json

{
    "name"            : "mojolicious-app-smoke"
    "description"     : "smoke tests for our Mojolicious application",
    "version"         : "0.0.1",
    "url"             : "https://github.com/melezhik/mojolicious-app-smoke"
}

We name our test suite, give it a version, provide short description and provide source code link (github).

Also let's provide a small README.md to let other understand what it is:

$ nano README.md

# SYNOPSIS

Make some smoke tests against our mojolicious application

# INSTALL

$ sparrow plg install mojolicious-app-smoke

# USAGE

$ sparrow plg run mojolicious-app-smoke

# Author

Alexey Melezhik

Now let's commit our changes

$ git init
$ git add .
$ git commit -a 'first commit'
$ git remote add origin https://github.com/melezhik/mojolicious-app-smoke.git
$ git push -u origin master

Finlay we are ready to upload a test suite to SparrowHub:

$ sparrow plg upload

sparrow.json file validated ... 
plugin mojolicious-app-smoke version 0.000001 upload OK

Once you make changes to your test suite you bump a version and release a new stuff into Sparrowhub. As I told you a sparrow test suite has it own life cycle separated from application being tested.

Finally If for security reasons you don't want to make your test suite public Sparrow allow you to host your scripts at private git repositories ( github/bitbuket).

Get it run

Ok. Now having a test suite as sparrow plugin we could easily install and run it somewhere we need:

$ sparrow plg install mojolicious-app-smoke
$ sparrow plg run mojolicious-app-smoke

We can even override a test suite parameter via command line:

$ sparrow plg run mojolicious-app-smoke --param name.bird=woodpecker

Or create a configuration via sparrow task:

$ sparrow project create webapp
$ sparrow task add webapp smoke-test mojolicious-app-smoke
$ sparrow task ini webapp/smoke-test

<name>
  bird Woodpecker
  animal Fox
</name>

To get plugin documentation simply run:

$ sparrow plg man mojolicious-app-smoke

Or view plugin page at SparrowHub site - https://sparrowhub.org/info/mojolicious-app-smoke .

There are many other fun things you could do with Sparrow API, please follow Sparrow documentation.

Summary

What we have just learned:

  • How the sparrow approach differs from classic unit tests for Perl
  • How to create sparrow scenarios to test Mojolicious applications
  • How to distribute sparrow tests suites with the notion of so called sparrow plugins

Regards

Alexey Melezhik

How to upload a script to SparrowHub

This is informal response to David Farrell's article How to upload a script to CPAN.

Preamble: CPAN is great. This post in no way should be treated as Sparrow VS CPAN attempt. Sparrow is just an alternative method to distribute your scripts. Ok, let's go.

So let’s say I’ve got this Perl script:

$ cat story.pl

use strict;
use warnings;
use DateTime;

my $text = 'bar';
my $dt = DateTime->now;

if ($dt->mon == 1 && $dt->day == 31) {
   $text = reverse $text;
}

print "$text\n";
exit 0;

Notice I have called my script story.pl? This is a naming convention for sparrow scripts.

Ok, let's move on.

Setup your distribution directory

All we need is to:

  • create a story check file
  • create a cpanfile to declare script dependencies
  • create a plugin meta file
  • optionally create README.md

story check file

Sparrow scripts should be accompanied by a check file. It's just a text file with some patterns to match against stdout emitted by an executed script. For example:

$ cat story.check  
regexp: (bar|rab)

Here we just require that script yields into stdout one of two lines - bar or rab. That is it.

Sometimes we don't need to check script stdout , that's ok just leave the story.check file empty:

$ echo > story.check

cpanfile

As we have an external dependency (the DateTime module) let's put it in a cpanfile:

$ cat cpanfile
requires 'DateTime'

Sparrow uses carton to run script with dependencies. That is it.

plugin meta file

In a plugin meta file one defines essential information required for script upload to SparrowHub. The structure is quite simple, there should be a JSON format file with these fields:

  • name - a plugin name
  • version - a plugin version
  • description - short plugin description
  • url - plugin web page url (optional)

In other words the sparrow meta file is the way to "convert" an existing script into sparrow plugin:

$ cat sparrow.json
{
    "name" : "bar-script",
    "version" : "0.0.1",
    "description" : "print bar or rab",
    "url" : "https://github.com/melezhik/bar-script"
}

Readme file

You might want to add some documentation to the script. Simply create a README.md file with documentation in markdown format:

$ cat README.md

# SYNOPSIS

print `bar` or `rab`

# INSTALL

    $ sparrow plg install bar-script

# USAGE

    $ sparrow plg run bar-script


# Author

[Alexey Melezhik](melezhik@gmail.com)

# Disclosure

An initial script code borrowed from David Farrell article [How to upload a script to CPAN](http://perltricks.com/article/how-to-upload-a-script-to-cpan/)

Finally we have the following project structure:

$ tree

.
├── cpanfile
├── README.md
├── sparrow.json
├── story.check
└── story.pl

0 directories, 5 files

Test script

To see that the script does what you want simply run strun inside the project root directory:

$ carton # install dependencies
$ carton exec strun


/ started

bar
ok      scenario succeeded
ok      output match /(bar|rab)/
STATUS  SUCCEED

Strun - is utility comes with Sparrow to run sparrow scripts, it is used by plugin developers.

Upload script to Sparrowhub

Provided you have an account on SparrowHub, just do this:

$ sparrow plg upload
sparrow.json file validated ...
plugin bar-script version 0.000001 upload OK

Now you can browse the script information at SparrowHub.

Run script

To run the script you need to install and run it with the sparrow client:

$ sparrow index update
$ sparrow plg install bar-script
$ sparrow plg run bar-script

sparrow-bar-script.png

Further reading

PS. Thanks to David Farrell for "giving" an idea of writing this post ( after reading his article ) and for fixing some typos and style errors in this document.

sparrow remote tasks - way to package your scripts and configurations.

Idea behind a sparrow is quite simple. You have a script, you package it as sparrow plugin and you can re-use it somewhere else.

Give me list of available plugins:

$ sparrow plg search    

type    name

public  app-cpm
public  bash
public  bitbucket-repo
public  check-tomcat-deploy
public  chemaxon-check
public  cpan-package
public  df-check
public  directory
public  docker-engine
public  file
public  foo-generic
public  foo-test
public  git-async-push
public  git-base
public  git-svn-export
public  gitprep
public  group
public  http-status
public  logdog
public  minion-check
public  module-release
public  mongodb
public  nano-setup
public  net-ping
public  nginx-check
public  nginx-example
public  outth-mysql-cookbook
public  package-generic
public  perl-app
public  perlbrew
public  proc-validate
public  ruby-test
public  sendmail
public  service
public  sph-check
public  ssh-sudo-check
public  ssh-sudo-try
public  sshd-check
public  stale-proc-check
public  svn-to-git-submodules
public  swat-nginx
public  swat-pintod
public  swat-test
public  templater
public  user
private package-generic-dev

This plugin makes a trivial nano.rc file configuration:

$ sparrow plg install nano-setup

Project is just a container for plugins to group them logically:

$ sparrow project create utils

Task is a plugin with parameters, in other words task bind plugin with parameters to project:

$ sparrow task add utils nano-rc nano-setup
$ sparrow task ini utils nano-rc
   tabsize 2

Now let's run our task:

$ sparrow task run utils nano-rc

<nano-rc>

/ started

rc file generated from template
rc file updated
ok      scenario succeeded
ok      [b] output match 'rc file generated from template'
ok      [b] output match 'rc file updated'
STATUS  SUCCEED

You can even override task parameters via command line:

 $ sparrow task run utils nano-rc --param tabsize=4

So sparrow plugins are packed scripts, but I want to package my configuration ...

A task initialization process could be the same if we want to reuse not only scripts but configuration. Every time I ssh on new server I want to apply the same nano.rc as it good for me. Here are remote sparrow tasks:

$ sparrow remote task upload utils/nano-rc 'my nano.rc setup'
task updated
task ini data updated OK

I have just uploaded a nano-rc task and it's configuration to my SparrowHub account. Now having this I could ssh to another server and re-apply my task:

$ ssh some-server
$ sparrow remote task run  utils/nano-rc


task meta data saved to /home/vagrant/sparrow/cache/meta/utils/nano-rc.json ...
public@nano-setup is uptodate (0.1.0)
project utils already exists - nothing to do here ...
task utils/nano-rc already exists, update task parameters
task - set plugin to public@nano-setup
task utils/nano-rc successfully created
loaded test suite ini from /home/vagrant/sparrow/cache/meta/utils/nano-rc.ini OK

<nano-rc>

/ started

rc file generated from template
rc file updated
ok      scenario succeeded
ok      [b] output match 'rc file generated from template'
ok      [b] output match 'rc file updated'
STATUS  SUCCEED

Pretty handy, huh?

Once remote task gets run for the first time , later you can run it as local sparrow task :

 $ sparrow task run utils nano-rc

Ok, if I find my task quite common I even can share it with others:

$ sparrow remote task share  utils/nano-rc
task nano-rc shared OK
now everyone could see it!

Everyone now can run my task:

$ sparrow remote task run melezhik@utils/nano-rc

The list of available remote tasks could be fetched like this:

$ sparrow remote task public-list 

2016-10-24 12:59:06 melezhik@packages/toplist | install my favorite packages
2016-10-24 12:58:22 melezhik@utils/nano-rc | my nano.rc setup

The list of private remote tasks related to your account is taken as:

$ sparrow remote task list

You may take other actions on your remote tasks:

$ sparrow remote task hide app/secure # to hide task from others ( make it private )
$ sparrow remote task remove app/secure # to remove task

Further steps

Follow sparrow docs on remote tasks API. Or create useful task and share with others!

At the end of my post is a simple but illuminative example of remote task to install my favorite packages:

$ sparrow project create packages
$ sparrow task add packages toplist package-generic
$ sparrow task ini packages toplist
    list       nano mc ncdu telnet tree 

$ sparrow remote task upload packages/toplist 'installs my favorite packages'

Regards

Alexey Melezhik

SparrowHub Plugins Request

A quick search at github on something like Perl/Bash scripts gives me a lot of results.

As developers we more think about modules and libraries when talk about software reuse. But in our day-to-day life scripts still take a vital part. Scripts like little, but useful commands to get our job done. This is why people from many programming languages/environments tend to create a places for such a tools called repositories, so other could easily find, install and run scripts or utilities. To list a few:

Well. A SparrowHub is a attempt to make a such effort in Perl communities to create a repository of useful automation scripts. Sparrow is Perl friendly in many ways:

  • it is written on Perl
  • it gets installed as CPAN module
  • it respects carton/cpanfile dependency management for your scripts
  • it supports Config::General format for script configuration ( among JSON, YAML and command line parameters )

Well let me now turn to the essential point of my post.

I know there are a plenty of cool software written by Perl community we know as CPAN, and many, many Perl modules get shipped with some useful scripts, even more some module's are only public interface - some command line scripts. So why not to upload such a script into a repository so other could use it? I know not all the cases are good fit, but let me outline some criteria if your script could be a good candidate to be uploaded as sparrow plugin:

  • you have some command line tools based on existed Perl/CPAN/Bash scripts and want to share it somehow with others or even keep it for yourself so to repeat script installation / execution process in the same way every time you setup a new server.

  • script takes a lot of input parameters or parameters with the structure is hard to be expressed in command line manner ( Hashes, Lists, Groups, so on ).

  • you need to add some middle-ware steps to your script logic , but don't want to add this into scripts source code.

  • you need to build a more complex tool based on your low level script/utility.

  • script possibly gets run often and / or on multiple servers.

  • script requires some extra checks so one could run it safely.

  • script is a generic tool acting differently depending on input parameters.

So if this sounds like about your scripts, you may start playing with SparrowHub. After all it's just distribution platform, having minimal impact at your script code:

  • You write/already have a script ( plus some dependencies software like CPAN module )
  • You upload script / or script based tool to SparrowHub as sparrow plugin
  • Others may find, install and run your script ( via SparrowHub ).

If this sounds interesting there are simple `how to contribute' steps:

  • Get registered at SparrowHub
  • Convert your script into sparrow plugin
  • Upload new sparrow plugin

In case you still have a questions or need help on converting your scripts into sparrow plugin, please:

Finally, if you feel like you have no time to dive into new things and study docs, but you are still interested in the subject, just let me know ( creating an issue at sparrow/github ) about script you wish add sparrowhub to and I will try to do it soon.

Regards.

Alexey Melezhik

Monitoring minion workers/tasks using Sparrow

Hi! This is small post concerning sparrow plugin to monitor minion workers. I had some post on this recently, but now I have made some improvements at existed plugin, so I am writing a new one ...

On my production server I use Minion to send emails to my clients. For some reasons there are faults on executing minion tasks or even minion workers get stopped sometimes for reasons unknown. As sending emails is a vital part of the service registration system I need to know if everything goes bad with my email system. Here is minon-check sparrow plugin to risqué.

Install sparrow

$ cpanm Sparrow

Install sparrow plugin

$ sparrow index update # get latest plugins index from Sparrowhub
$ sparrow plg install minion-check # install plugin

Set plugin up

$ sparrow project create webapp # create a projects to hold web applications tasks
$ sparrow task add webapp minion-health minion-check # this is 
# minion health task which is "tied" to minion-check plugin

$ EDITOR=nano  sparrow task ini webapp minion-health # set task parameters. 
# on my server this will be:

# I use carton install, so:
command = cd /path/to/your/mojo/app && carton exec ./app.pl minion

# sets worker footprint to lookup at processes list
worker_cmd = minion worker

A few comments on plugin setup here:

  • A command parameter define a system command to handle your minion jobs/tasks , An app.pl is Mojolicious application to run my web application, so I set my minion command as mojolicious command.

  • A worker_cmd parameter is just a string to look up inside a process list, to identify that minion worker is alive. A simple stings minion worker works for me.

Let's run a task and see the output:

$ sparrow task run webapp minion-health


<minion-tasks>

/modules/check-worker/ started

look up {{{minion worker}}} at proccess list ...
web-app       2748  0.3  3.4 202704 35216 pts/0    S    23:17   0:00 perl ./app.pl minion worker -m production -I 15 -j 2
done
ok      scenario succeeded
ok      output match 'done'

/modules/last-task/ started

Q=0
ok      scenario succeeded
ok      output match /Q=(1|0)/

/ started

no failed tasks found
ok      0 failed jobs found
STATUS  SUCCEED

So what sparrow plugin minion-check does?

  • check if your minion worker(s) is running.

  • check if you have no tasks marked as FAILED for the given period of time. ( I do not set it up explicitly here, but for default it uses 10 minutes interval , for details follow minion-check documentation ).

Finally I setup cron job to run a sparrow plugin every 10 minutes:

$ sparrow task run webapp minion-health --cron

A --cron flag makes sparrow client silent unless any errors happen during execution.


Regards.

Alexey Melezhik

Sparrowdo automation. Part 6. Sparrowdo modules - getting a bigger things using light primitives.

This is what have been seen before:

Well, while keep writing a sparrowdo tutorial the tool keep growing too. Let me introduce something new and excited about sparrowdo automation - how one can easily create a higher level entities using so called sparrowdo modules.

Sparrowdo modules ...

So far we have talked about some sparrowdo primitives. They are light, they are small and they relate to a small specific tasks, under the hood they are just sparrow plugins with parameters - sparrowdo tasks.

Well, here is the list to recall a few:

  • System packages - package-generic plugin
  • CPAN packages - cpan-package plugin
  • Users and groups - are user and group plugins
  • Linux Services are represented by service plugin
  • And more and more and more ...

And so on, you can see all of them here, on sparrowhub site - https://sparrowhub.org/search. Most of sparrow plugins are just black boxes to solve a specific task. More or less plugins are just primitives.

Take a look at chef resources or ansible modules - they are probably of the same nature.

Now let's me introduce a sparrowdo module - a container for sparrow plugins.

Consider a quite common task. Installing Nginx web server. Having looked at sparrow toolkit we have all necessary bricks to "build" a running Nginx server:

  • Package-generic plugin to install nginx package
  • Service plugin to enable and run nginx service

Let's write code then:

use v6;

unit module Sparrowdo::Nginx;

use Sparrowdo;

our sub tasks (%args) {

  task_run  %(
    task => 'install nginx',
    plugin => 'package-generic',
    parameters => %( list => 'nginx' )
  );

  task_run  %(
    task => 'enable nginx',
    plugin => 'service',
    parameters => %( service => 'nginx', action => 'enable' )
  );

  task_run  %(
    task => 'start nginx',
    plugin => 'service',
    parameters => %( service => 'nginx', action => 'start' )
  );


}

Ok. The code is quite simple, I just want to add some vital comments here.

  • Sparrowdo modules are plain Perl6 modules.

  • You need to load Sparrowdo module to export some sparrowdo API functions, like task_run or others.

  • You have to define at least a tasks(%args) function gets called when someone else use your modules, see how later

  • You optionally may handle some arguments get passed into your module, see tasks function signature.

  • And finally sparrowdo module is just a container for some sparrowdo tasks get called sequentially .

Ok. Now lets use our new sparrowdo module.

First step we need to ensure that module installed at the server where from we are going to run a sparrowdo tasks:

$ panda install Sparrowdo::Nginx

Ok when we are ready with module install we have two ways here.

  1. running module as is

  2. using module inside sparrowdo scenario.

Running module as is

This is the simplest way. This is very similar to running ansible modules:

sparrowdo --host=127.0.0.1 --module_run=Nginx

install nginx server

Running module via sparrowfile

Of course one can just use sparrowdo module using sparrowdo API

$ cat sparrowfile

run_module 'Nginx';

Sparrowdo uses a convention about modules names, it cut a Sparrowdo:: prefix from module name when run it via run_module function. So the rule is simple:

| Module Perl6 Name | Run_module $name parameter |
+-------------------+----------------------------+
| Sparrow::Foo::Bar | 'Foo::Bar'                 |

A current version of Sparrowdo::Nginx ignore an arguments, one day it would be possible to call 'Nginx' module with parameters:

run_module 'Nginx', %( port => 81 );

Little helpers for developers life

Sparrowdo provides some essentials helpers to simplify some developers tasks.

Guessing a target OS

It's very usual when we need to know a target server OS name to make a right decision about server configuration. Let me show you. Recall Nginx module, for centos we need install a repository so nginx package is not here by default:

our sub tasks (%args) {

  if target_os() ~~ m/centos/ {

    task_run  %(
      task => 'install epel-release',
      plugin => 'package-generic',
      parameters => %( list => 'epel-release' )
    );

  }

Passing a sparrowdo command line parameters

Remember a post on installing CPAN packages on the servers with http proxy restrictions?

Consider this sparrwodo module to install CPAN modules:

use v6;

unit module Sparrowdo::CpanInstall;

use Sparrowdo;

our sub tasks (%args) {

  task_run  %(
    task => 'install cpan modules',
    plugin => 'cpan-package',
    parameters => %( 
      list => %args<list>,
      http_proxy => input_params('HttpProxy'), 
      https_proxy => input_params('HttpsProxy'), 
    )
  );

}

And sparrowfile:

module_run 'CpanInstall' %(
  list => 'DBI Moose Mojolicious'
);

And finally sparrowdo scenario run:

$ sparrwodo --host=<$some-host> --http_proxy=<$http_proxy>  --https_proxy=<$https_proxy>

An input_params($param_name) function will pass all the sparrowdo client parameters back to Sparrowdo::CpanInstall module.

Conclusion

Sparrowdo modules are high level members of sparrowdo echo system. What is good they are just Perl6 modules. Anybody who codes at Perl6 could easily start a new ones ( a basic knowledge of existed sparrow plugins is required though ).

Here is a short list of my ones:

https://modules.perl6.org/#q=Sparrowdo%3A%3A

Will be happy to see a new members of Sparrowdo::* family at Perl6 modules repository.

Regards and have a good weekend.


Alexey Melezhik

Sparrowdo automation. Part 5. Managing services and processes.

HI!

This time I want to tell you how to manage services and processes using sparrowdo.

Before this post a following list of topics was written by me:

As services are highly coupled with processes we will investigate them in one post.

Let's have an nginx web server gets installed on your system:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'install nginx server',
  plugin => 'package-generic',
  parameters => %( list => 'nginx' )
);

We talked about package-generic plugin at this post. We use this plugin to install system packages.

install nginx server

Ok. This is very logical now having installed an nginx to make it "bootable", so next reboot of our system will pickup an nginx and make it sure it runs too. Some people call this autoload:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run %(
  task => 'enable nginx service',
  plugin => 'service',
  parameters => %( action => 'enable', service => 'nginx' )
);

task_run %(
  task => 'start nginx service',
  plugin => 'service',
  parameters => %( action => 'start', service => 'nginx' )
);

nginx-up-and-running

A service plugin makes it possible to enable and disabling Linux services, as well as starting and stopping them. It's very simple yet useful plugin for those who want to automate Linux services on target hosts.

At example here we not only make it nginx autoloadable enabling it, but also make it sure it starts. So good so far.

Well time goes and we need to ensure that nginx server is running. There are more than one way to do this.

The simplest one is to look up in a processes tree a process related to nginx master. This is what I usually do first when troubleshoot nginx server issues.

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'check my nginx master process',
  plugin => 'proc-validate',
  parameters => %(
    pid_file => '/var/run/nginx.pid',
    footprint => 'nginx.*master'
  )
);

nginx-master-process

A proc-validate plugin takes 2 parameters at input. The first one is the path to file where PID is written, and the second optional one - Perl regular expression to identify a process at process tree. Even providing only the first parameter is enough but I also set a footprint to make my example more explanatory.

Summary

We've learned how to manage Linux services with the help of sparrowdo. It's easy and it makes your routine tasks automated. And if you want to add some "audit" to your running services, which of course sounds reasonable for maintainers jobs the easiest way to start with is using simple proc-validate plugin.


See you soon at our next topic.

Have a fun in coding and automation.

-- Alexey Melezhik

Sparrowdo automation. Part 1. Installing cpan packages.

Here I start a series of posts on sparrowdo configuration management tool usage.

Installing sparrowdo

Sparrowdo implies installing software on both master and target host. Following "push" approach one should install a sparrowdo on master host where from he starts a ssh sessions against target hosts to be configurable:

$ ssh master.host 
$ panda install Sparrowdo

Once you install a sparrowdo you need to install a sparrow client on every host where you want to start deployments. Sparrow acts like client to carry out deployment tasks initiated from master host.

$ ssh target.host 
$ cpanm Sparrow
$ yum install curl

Here is simple schema of master -> target hosts interaction:

  +-----------------+
  |                 |    ssh
  |                 |------------> < host-1 > 192.168.0.1
  | <master host>   |    ssh
  | {sparrowdo}     |------------------> < host-2 > 192.168.0.2
  |                 |    ssh 
  |                 |-----------------------> < host-N > 192.168.0.3
  |                 |
  +-----------------+


  +-------------+
  |             |
  | <host>      |
  |             |
  | {sparrow}   | 
  | {curl}      |
  |             |
  +-------------+

That is it. Now you are ready to start your very first deployment with sparrowdo. As the title of the post has we want to install cpan packages.

Sparrowdo scenarios

Sparrowdo scenarios are written in Perl6. Following Sparrowdo API basically all you need is to find a proper sparrow plugin and run it with parameters. Luckily we have a proper one to solve our task - cpan-package. In most cases Sparrowdo way to configure servers is quite simple - find a proper plugins to handle your specific tasks and use them. We could treat sparrow plugins like a useful primitives to accomplish a complex tasks. This conception is very similar to what in chef or puppet is called resources , or in ansible is called modules. In case you miss a plugin to solve your task you are encouraged to create a new one, in easy way, but this series of posts are more focused on sparrowdo end client usage rather on sparrow plugin development.

Ok, let get back to cpan-package plugin, and just use it in our sparrowdo scenario:

$ ssh master.host
$ cat sparrowfile

task_run  %(
  task => 'install some cpan packages',
  plugin => 'cpan-package',
    parameters => %(
    list => 'CGI DBI',
    install-base => '/opt/perl'
  )
);

The content of sparrowfile is quite self-explanatory. We want to install 2 packages CGI and DBI into /opt/perl install base directory. Here we go!

A few words about ssh setup

A sparrowdo ssh setup implies 2 things so everything works fine:

  • a user from master host has password-less ssh access to target host
  • a user has (password-less) sudo on target host

You may specify user explicitly when run sparrowdo scenario or current user would be used implicitly when connecting by ssh to target host. From the master host now we run:

$ sparrowdo -h target.host

He is what I have for my test target box:

$ sparrowdo --host=192.168.0.1
running sparrow tasks on 192.168.0.1 ...
running task <install some cpan packages> plg <cpan-package>
parameters: {install-base => /opt/perl, list => CGI DBI}
/tmp/.outthentic/5385/opt/sparrow/plugins/public/cpan-package/story.t ..
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI into /opt/perl ...
# Successfully installed HTML-Parser-3.72 (upgraded from 3.64)
# Successfully installed Test-Deep-1.120
# Successfully installed Sub-Uplevel-0.25
# Successfully installed Carp-1.38 (upgraded from 1.11)
# Successfully installed Test-Warn-0.30
# Successfully installed CGI-4.31 (upgraded from 3.51)
# 6 distributions installed
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install DBI into /opt/perl ...
# Successfully installed DBI-1.636
# 1 distribution installed
# install ok
ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3
ok
All tests successful.
Files=1, Tests=3, 91 wallclock secs ( 0.02 usr  0.01 sys + 72.58 cusr  8.26 csys = 80.87 CPU)
Result: PASS

A few things to say here:

  • A sparrowdo reports are TAP reports due to TDD nature of sparrow plugins.
  • You may have extra information here by adding --verbose parameter for sparrowdo client to see a ssh commands details and some other indeterminate steps taken by sparrowdo when deploying your server.

So, keep your eye on following sparrowdo presentations.

Regards.


Alexey

sparrowdo sandbox

Hi all!

Sparrowdo is a new born #perl6 project for configuration management automation. It based on sparrow plugin system.

A user unfamiliar with sparrow or sparrowdo may check out a sparrowdo-test repository containing a ready to use sparrowdo scenarios for some typical cases , like installing packages, checking linux process and so on ...

So to start use it:

  • install sparrowdo on your "master" host
  • install sparrow ( vers >= 0.1.11 ) on your "target" host

On your "master" host:

  • git clone https://github.com/melezhik/sparrowdo-test
  • cd ./sparrow-test/example/dir
  • sparrowdo --host=target-host # see other options in sparrowdo documentation

Basically I have tested it on some ubuntu/centos/debian boxes, if you have any issues let me know ...

Regards.

Alexey

Sparrowdo - a simple configuration management tool written on Perl6 and consuming sparrow plugins

Hi!

This is the very first ( to take it for a spin ) release of sparrowdo - a simple configuration management tool written on Perl6 and consuming sparrow plugins.

This is where Perl5, Perl6 could make a synergy. Consider a simple example of sparrowdo scenario:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'install my packages',
  plugin => 'package-generic',
  parameters => %( list => 'git-core carton cpanminus' )
);

task_run  %(
  task => 'install psgi app',
  plugin => 'perl-app',
  parameters => %( 
    'app_source_url' => 'https://github.com/melezhik/web-app.git',
    'git_branch' => 'master',
    'http_port' => 3030
  )
);

All it does is:

  • installing some packages required for further deployment ( git, cpanm, carton )
  • installing perl5 psgi application by fetching source code from remote repository, installing dependencies with cpanfile/carton and running service using Ubic and Starman.

Here we go. On my vagrant Ubuntu 14.04 Trusty box I will have:

$ sparrowdo --ssh_user=vagrant  --ssh_port=2200 --host=127.0.0.1  --verbose 
running sparrow tasks on 127.0.0.1 ... 
running task <install my packages> plg <package-generic> 
parameters:
{list => git-core carton cpanminus}
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow index update"'
get index updates from SparrowHub ... OK
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow project remove sparrowdo"'
project sparrowdo successfully removed

ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow plg install package-generic"'
public@package-generic is uptodate (0.1.1)
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow project create sparrowdo"'
project sparrowdo successfully created

ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow task add sparrowdo install_my_packages package-generic"'
task - set plugin to public@package-generic

task sparrowdo/install_my_packages successfully created

install_my_packages.json                                                                                                                              100%   41     0.0KB/s   00:00    
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow task run sparrowdo install_my_packages --json /tmp/install_my_packages.json"'
# cd /home/vagrant/sparrow/plugins/public/package-generic 
# export PATH=$PATH:local/bin 
# export PERL5LIB=local/lib/perl5:$PERL5LIB 
# strun --root ./ --json /tmp/install_my_packages.json --json /tmp/install_my_packages.json

/tmp/.outthentic/31112/home/vagrant/sparrow/plugins/public/package-generic/story.t .. 
# [/home/vagrant/sparrow/plugins/public/package-generic/modules/apt-get]
# Package: git-core
# Version: 1:2.1.4-2.1
# Status: install ok installed
ok 1 - output match 'Status: install ok installed'
# [/home/vagrant/sparrow/plugins/public/package-generic/modules/apt-get]
# Package: carton
# Version: 1.0.12-1
# Status: install ok installed
ok 2 - output match 'Status: install ok installed'
# [/home/vagrant/sparrow/plugins/public/package-generic/modules/apt-get]
# Package: cpanminus
# Version: 1.7014-1
# Status: install ok installed
ok 3 - output match 'Status: install ok installed'
# [/home/vagrant/sparrow/plugins/public/package-generic]
# done
ok 4 - output match 'done'
1..4
ok
All tests successful.
Files=1, Tests=4,  1 wallclock secs ( 0.01 usr  0.01 sys +  1.28 cusr  0.07 csys =  1.37 CPU)
Result: PASS
running task <install-psgi-app> plg <perl-app> 
parameters:
{app_source_url => https://github.com/melezhik/web-app.git, git_branch => master, http_port => 3030}
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow plg install perl-app"'
public@perl-app is uptodate (0.1.5)
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow project create sparrowdo"'
project sparrowdo already exists - nothing to do here ... 

ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow task add sparrowdo install-psgi-app perl-app"'
task - set plugin to public@perl-app

task sparrowdo/install-psgi-app successfully created

install-psgi-app.json                                                                                                                                 100%  110     0.1KB/s   00:00    
ssh -q -tt -p 2200 vagrant@127.0.0.1 ' sudo bash -c "export LC_ALL=en_US.UTF-8 ; sparrow task run sparrowdo install-psgi-app --json /tmp/install-psgi-app.json"'
# cd /home/vagrant/sparrow/plugins/public/perl-app 
# export PATH=$PATH:local/bin 
# export PERL5LIB=local/lib/perl5:$PERL5LIB 
# strun --root ./ --json /tmp/install-psgi-app.json --json /tmp/install-psgi-app.json

/tmp/.outthentic/31292/home/vagrant/sparrow/plugins/public/perl-app/story.t .. 
# [/home/vagrant/sparrow/plugins/public/perl-app]
# useradd: user 'perl-app' already exists
# Ubic is up to date. (1.59)
# Ubic::Service::Plack is up to date. (1.18)
# Starman is up to date. (0.4014)
# Stopping perl-app... stopped
# Already on 'master'
# Your branch is up-to-date with 'origin/master'.
# Already up-to-date.
# Installing modules using /opt/perl-app/cpanfile (deployment mode)
# Complete! Modules were installed into /opt/perl-app/local
# Starting perl-app... started
# perl-app  running
# install-ok
ok 1 - output match 'Complete! Modules were installed into'
ok 2 - output match 'Starting perl-app... started'
ok 3 - output match 'install-ok'
1..3
ok
All tests successful.
Files=1, Tests=3,  3 wallclock secs ( 0.01 usr  0.02 sys +  0.74 cusr  0.15 csys =  0.92 CPU)
Result: PASS

Asciinema video

A visual example of the sparrowdo scenario could be found here -

asciicast

Volunteers/Contributors needed!!!

Interested in Perl 6 automation for devops? Sparrowdo could be a good greenfield project to start with. Not involved in Perl 6 and use Perl 5 ? - that is fine, sparrowdo USES sparrow plugins which are to be written on Perl5|Bash|Ruby! So why not write a useful plugin to share with? Here is a short list of plugins now - use it as examples and create your own one! And finally sparrow itself is written on Perl5! Just contribute somehow into sparrow/sparrowhub/sparrowdo echo system! Merge requests are welcome!

-- Regards

Alexey

How to write a simple nginx installer with sparrow in 4 minutes - nice asciinema video!

A simple nginx installer with #sparrow - nice asciinema video!

https://asciinema.org/a/06h3ezswbyidu0pdl993rl30v

#nginx #perl #devops #automation

And ... yeah I forgot to say cpanm Sparrow at the very beginning .... ))

https://sparrowhub.org updated

HI ! https://sparrowhub.org/ site updated! A various changes of index page:

  • welcome text distilled to make it sparrow usage more clear and understandable
  • a look-and-feel screenshots added
  • a links to tutorials, documentation pages and papers added
  • a new motto proposed ))) - "Sparrow - Reusable automation scripts" !

Alexey

Building docker images with sparrow

This is a paper written in Russian on building docker images with sparrow - https://habrahabr.ru/post/302278/

Playing with Docker and Sparrow

Docker is quite popular solution to rapidly spin up developers environments. I have been playing with it and it seems fun for me. The other fun thing I found that Sparrow could be a good solution to build up new docker images.

Here is short example of how it could be. A few lines in Dockerfile and you have a GitPrep server up and running as docker container. Whew!

Here is my Dockerfile:

FROM perl:5.20
RUN apt-get update
RUN apt-get install sudo
RUN apt-get install git-core
RUN cpanm Sparrow
RUN sparrow index update
RUN sparrow index summary
RUN sparrow plg install gitprep
RUN sparrow plg run gitprep
CMD sparrow plg run gitprep --param action=start --param start_mode=foreground

I base on official perl docker image and then let all the job to be done by sparrow gitprep plugin!

This is how one can use it:

$ git clone https://github.com/melezhik/docker-projects.git
$ cd docker-projects/gitprep

Build an image with target melezhik/gitprep or whatever you want. You probably will need to run this command more than once if meet "Installing the dependencies failed: Installed version (1.636) of DBI is not in range '== 1.634'" error.

$ sudo docker build -t melezhik/gitprep .

Run gitprep server

$ sudo docker run -p 10020:10020 -d -i melezhik/gitprep

Test it!

curl 127.0.0.1:10020

Installing docker engine on Ubuntu Trusty 14.04 (LTS) using sparrow

While playing with docker I created a simple sparrow plugin to install docker engine on Ubuntu Trusty 14.04 (LTS) - https://sparrowhub.org/info/docker-engine . Please let me know if other platform to support you need! ;))

sparrow-docker.png

Sparrow plugins development tutorial in Russian.

Sparrow plugins development tutorial in Russian. https://habrahabr.ru/post/300876.

2  

About melezhik

user-pic Dev & Devops --- Then I beheld all the work of God, that a man cannot find out the work that is done under the sun: because though a man labour to seek it out, yet he shall not find it; yea further; though a wise man think to know it, yet shall he not be able to find it. (Ecclesiastes 8:17)