Sparrowdo automation. Part 6. Sparrowdo modules - getting a bigger things using light primitives.

This is what have been seen before:

Well, while keep writing a sparrowdo tutorial the tool keep growing too. Let me introduce something new and excited about sparrowdo automation - how one can easily create a higher level entities using so called sparrowdo modules.

Sparrowdo modules ...

So far we have talked about some sparrowdo primitives. They are light, they are small and they relate to a small specific tasks, under the hood they are just sparrow plugins with parameters - sparrowdo tasks.

Well, here is the list to recall a few:

  • System packages - package-generic plugin
  • CPAN packages - cpan-package plugin
  • Users and groups - are user and group plugins
  • Linux Services are represented by service plugin
  • And more and more and more ...

And so on, you can see all of them here, on sparrowhub site - https://sparrowhub.org/search. Most of sparrow plugins are just black boxes to solve a specific task. More or less plugins are just primitives.

Take a look at chef resources or ansible modules - they are probably of the same nature.

Now let's me introduce a sparrowdo module - a container for sparrow plugins.

Consider a quite common task. Installing Nginx web server. Having looked at sparrow toolkit we have all necessary bricks to "build" a running Nginx server:

  • Package-generic plugin to install nginx package
  • Service plugin to enable and run nginx service

Let's write code then:

use v6;

unit module Sparrowdo::Nginx;

use Sparrowdo;

our sub tasks (%args) {

  task_run  %(
    task => 'install nginx',
    plugin => 'package-generic',
    parameters => %( list => 'nginx' )
  );

  task_run  %(
    task => 'enable nginx',
    plugin => 'service',
    parameters => %( service => 'nginx', action => 'enable' )
  );

  task_run  %(
    task => 'start nginx',
    plugin => 'service',
    parameters => %( service => 'nginx', action => 'start' )
  );


}

Ok. The code is quite simple, I just want to add some vital comments here.

  • Sparrowdo modules are plain Perl6 modules.

  • You need to load Sparrowdo module to export some sparrowdo API functions, like task_run or others.

  • You have to define at least a tasks(%args) function gets called when someone else use your modules, see how later

  • You optionally may handle some arguments get passed into your module, see tasks function signature.

  • And finally sparrowdo module is just a container for some sparrowdo tasks get called sequentially .

Ok. Now lets use our new sparrowdo module.

First step we need to ensure that module installed at the server where from we are going to run a sparrowdo tasks:

$ panda install Sparrowdo::Nginx

Ok when we are ready with module install we have two ways here.

  1. running module as is

  2. using module inside sparrowdo scenario.

Running module as is

This is the simplest way. This is very similar to running ansible modules:

sparrowdo --host=127.0.0.1 --module_run=Nginx

install nginx server

Running module via sparrowfile

Of course one can just use sparrowdo module using sparrowdo API

$ cat sparrowfile

run_module 'Nginx';

Sparrowdo uses a convention about modules names, it cut a Sparrowdo:: prefix from module name when run it via run_module function. So the rule is simple:

| Module Perl6 Name | Run_module $name parameter |
+-------------------+----------------------------+
| Sparrow::Foo::Bar | 'Foo::Bar'                 |

A current version of Sparrowdo::Nginx ignore an arguments, one day it would be possible to call 'Nginx' module with parameters:

run_module 'Nginx', %( port => 81 );

Little helpers for developers life

Sparrowdo provides some essentials helpers to simplify some developers tasks.

Guessing a target OS

It's very usual when we need to know a target server OS name to make a right decision about server configuration. Let me show you. Recall Nginx module, for centos we need install a repository so nginx package is not here by default:

our sub tasks (%args) {

  if target_os() ~~ m/centos/ {

    task_run  %(
      task => 'install epel-release',
      plugin => 'package-generic',
      parameters => %( list => 'epel-release' )
    );

  }

Passing a sparrowdo command line parameters

Remember a post on installing CPAN packages on the servers with http proxy restrictions?

Consider this sparrwodo module to install CPAN modules:

use v6;

unit module Sparrowdo::CpanInstall;

use Sparrowdo;

our sub tasks (%args) {

  task_run  %(
    task => 'install cpan modules',
    plugin => 'cpan-package',
    parameters => %( 
      list => %args<list>,
      http_proxy => input_params('HttpProxy'), 
      https_proxy => input_params('HttpsProxy'), 
    )
  );

}

And sparrowfile:

module_run 'CpanInstall' %(
  list => 'DBI Moose Mojolicious'
);

And finally sparrowdo scenario run:

$ sparrwodo --host=<$some-host> --http_proxy=<$http_proxy>  --https_proxy=<$https_proxy>

An input_params($param_name) function will pass all the sparrowdo client parameters back to Sparrowdo::CpanInstall module.

Conclusion

Sparrowdo modules are high level members of sparrowdo echo system. What is good they are just Perl6 modules. Anybody who codes at Perl6 could easily start a new ones ( a basic knowledge of existed sparrow plugins is required though ).

Here is a short list of my ones:

https://modules.perl6.org/#q=Sparrowdo%3A%3A

Will be happy to see a new members of Sparrowdo::* family at Perl6 modules repository.

Regards and have a good weekend.


Alexey Melezhik

Outthentic does not rely on Test::More any more.

Finally I decided not to use Test::More and Perl Test-Harness in Outthentic anymore.

These are the great tools proven in many many testing projects. But Outthentic tends to be more general purpose framework to run ANY scripts, rather than being a test framework only. Some testing facilitates are still here, but they poorly use a Test::More/Test-Harness entities, so I decided to rewrite story runner to get an asserts execution results in free style - so no TAP indeed is required.

$ cat story.pl 

print "hello from perl";


$ cat story.check 
hello from perl


$ strun 

/ started

hello from perl
OK  scenario succeeded
OK  output match 'hello from perl'
---
STATUS  SUCCEED

Such a changes might result in potential breakage for some sparrow plugins but I will fix it soon.

-- Regards

Alexey Melezhik

Sparrowdo automation. Part 5. Managing services and processes.

HI!

This time I want to tell you how to manage services and processes using sparrowdo.

Before this post a following list of topics was written by me:

As services are highly coupled with processes we will investigate them in one post.

Let's have an nginx web server gets installed on your system:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'install nginx server',
  plugin => 'package-generic',
  parameters => %( list => 'nginx' )
);

We talked about package-generic plugin at this post. We use this plugin to install system packages.

install nginx server

Ok. This is very logical now having installed an nginx to make it "bootable", so next reboot of our system will pickup an nginx and make it sure it runs too. Some people call this autoload:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run %(
  task => 'enable nginx service',
  plugin => 'service',
  parameters => %( action => 'enable', service => 'nginx' )
);

task_run %(
  task => 'start nginx service',
  plugin => 'service',
  parameters => %( action => 'start', service => 'nginx' )
);

nginx-up-and-running

A service plugin makes it possible to enable and disabling Linux services, as well as starting and stopping them. It's very simple yet useful plugin for those who want to automate Linux services on target hosts.

At example here we not only make it nginx autoloadable enabling it, but also make it sure it starts. So good so far.

Well time goes and we need to ensure that nginx server is running. There are more than one way to do this.

The simplest one is to look up in a processes tree a process related to nginx master. This is what I usually do first when troubleshoot nginx server issues.

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'check my nginx master process',
  plugin => 'proc-validate',
  parameters => %(
    pid_file => '/var/run/nginx.pid',
    footprint => 'nginx.*master'
  )
);

nginx-master-process

A proc-validate plugin takes 2 parameters at input. The first one is the path to file where PID is written, and the second optional one - Perl regular expression to identify a process at process tree. Even providing only the first parameter is enough but I also set a footprint to make my example more explanatory.

Summary

We've learned how to manage Linux services with the help of sparrowdo. It's easy and it makes your routine tasks automated. And if you want to add some "audit" to your running services, which of course sounds reasonable for maintainers jobs the easiest way to start with is using simple proc-validate plugin.


See you soon at our next topic.

Have a fun in coding and automation.

-- Alexey Melezhik

Sparrowdo automation. Part 4. Managing users and groups.

Hi! These are the previous articles:

This one is going to be quite short. We will learn how to manage linux users and groups using sparrowdo.

Let's get back to our ubiquitous example of installing CPAN modules:

task_run  %(
  task => 'install 2 modules',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI DBI'
  ),
);

If we run this one we will see that CGI and DBI CPAN modules get installed into system paths, which is ok.

Consider a following case though:

  • create a user account

  • install some CPAN modules locally for this user

Basically this means that once a user gets logged into his account he should be able to use his modules:

$ ssh -l some-user $some-host
$ export PERL5LIB=~/lib/perl5/
$ perl -MDBI -e 1
$ perl -MCGI -e 1

Let's rewrite our latest code adding user account here:

task_run  %(
  task => 'create user',
  plugin => 'user',
  parameters => %( name => 'foo' )
);


task_run  %(
  task => 'install 2 modules',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI DBI',
    user => 'foo',
    install-base => '~/'
  ),
);

And have it run:

$ sparrowdo --ssh_user=vagrant  --ssh_port=2222 --host=127.0.0.1  
running sparrow tasks on 127.0.0.1 ... 
running task <create user> plg <user> 
parameters: {name => foo}
/tmp/.outthentic/1563/opt/sparrow/plugins/public/user/story.t .. 
# [/opt/sparrow/plugins/public/user/modules/create]
# uid=1002(foo) gid=1002(foo) groups=1002(foo)
# user created
ok 1 - output match 'user created'
# [/opt/sparrow/plugins/public/user]
# done
ok 2 - output match 'done'
1..2
ok
All tests successful.
Files=1, Tests=2,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.09 cusr  0.09 csys =  0.21 CPU)
Result: PASS
running task <install 2 modules> plg <cpan-package> 
parameters: {install-base => ~/, list => CGI DBI, user => foo}
/tmp/.outthentic/1659/opt/sparrow/plugins/public/cpan-package/story.t .. 
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI into ~/ user foo ...
# Successfully installed Test-Deep-1.120
# Successfully installed Sub-Uplevel-0.25
# Successfully installed Test-Warn-0.30
# Successfully installed CGI-4.32
# 4 distributions installed
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install DBI into ~/ user foo ...
# Successfully installed DBI-1.636
# 1 distribution installed
# install ok
ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3
ok
All tests successful.
Files=1, Tests=3, 68 wallclock secs ( 0.02 usr  0.01 sys + 46.61 cusr  9.56 csys = 56.20 CPU)
Result: PASS

Let's check our freshly installed CPAN modules:

$ vagrant ssh
Last login: Mon Jul 25 08:58:21 2016 from 10.0.2.2
Welcome to your Vagrant-built virtual machine.
[vagrant@epplkraw0312t1 ~]$ sudo bash 
[root@epplkraw0312t1 vagrant]# su - foo
Last login: Пн июл 25 09:02:55 UTC 2016 on pts/1
[foo@epplkraw0312t1 ~]$ PERL5LIB=~/lib/perl5/ perl -MCGI -e 1
[foo@epplkraw0312t1 ~]$ PERL5LIB=~/lib/perl5/ perl -MDBI -e 1

Ok. We successfully installed desired stuff into the user's account.

And last thing in this tutorial. Managing linux groups. The code is going to be very simple, as with creating users:

task_run  %(
  task => 'create group',
  plugin => 'group',
  parameters => %( name => 'application',
  ),
);

Goodbye and see you soon at our next tutorial. Have not yet decided what it's going to be ;)

Sparrowdo automation. Part 3. Installing system packages.

HI! I continue blogging about sparrowdo - a simple perl6 configuration management tool.

This is what we've learned so far:

Installing packages

Consider out latest example with installing CPAN packages:

$ cat sparrowfile

use v6;

use Sparrowdo;


task_run  %(
  task => 'install CGI',
  plugin => 'cpan-package',
  parameters => %( list => 'CGI' ),
);

What we are trying to do here is to install CGI CPAN module using cpan-package plugin. Here is little trick is hidden. A cpan-package implies you have a cpanm client pre-installed at your system. Let's see what will happen if it is not:

$ sparrowdo --ssh_port=2200 --host=127.0.0.1 --ssh_user=vagrant 
running sparrow tasks on 127.0.0.1 ... 
running task <install-CGI> plg <cpan-package> 
parameters: {list => CGI}
/opt/sparrow/tmp/8417/opt/sparrow/plugins/public/cpan-package/story.t .. 
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI ...
# /opt/sparrow/plugins/public/cpan-package/modules/cpanm/story.bash: line 22: cpanm: command not found
not ok 1 - scenario succeeded

#   Failed test 'scenario succeeded'
#   at /usr/local/share/perl/5.20.2/Outthentic.pm line 193.
not ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3

#   Failed test 'output match 'install ok''
#   at /usr/local/share/perl/5.20.2/Outthentic.pm line 239.
# Looks like you failed 2 tests of 3.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/3 subtests 

Test Summary Report
-------------------
/opt/sparrow/tmp/8417/opt/sparrow/plugins/public/cpan-package/story.t (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  1-2
  Non-zero exit status: 2
Files=1, Tests=3,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.10 cusr  0.00 csys =  0.12 CPU)
Result: FAIL
The spawned process exited unsuccessfully (exit code: 1)
  in block <unit> at /home/melezhik/.rakudobrew/moar-nom/install/share/perl6/site/resources/CDEEEB90951FEDF24FB0EE4EFB79E7A6061B289C line 3

Upps, obviously we see the reason of the failure here:

cpanm: command not found

So, we need to install cpanm client to our system first to make it working cpan-package plugin. One day I probably add this dependency into cpan-package itself , but for now we could use a package-generic plugin to install any system packages.

Luckily we have cpanminus ubuntu package to install cpanm client on our ubuntu box:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'install cpanm client',
  plugin => 'package-generic',
  parameters => %( list => 'cpanminus' )
);

task_run  %(
  task => 'install CGI',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI',
  ),
);

Let's give it a run ...

$ sparrowdo --ssh_port=2200 --host=127.0.0.1 --ssh_user=vagrant
running sparrow tasks on 127.0.0.1 ...
running task <install cpanm client> plg <package-generic>
parameters: {list => cpanminus}
/opt/sparrow/tmp/8792/opt/sparrow/plugins/public/package-generic/story.t ..
# [/opt/sparrow/plugins/public/package-generic/modules/apt-get]
# trying to install cpanminus ...
# installer - apt-get
# Package: cpanminus
# Version: 1.7014-1
# Status: install ok installed
ok 1 - output match 'Status: install ok installed'
# [/opt/sparrow/plugins/public/package-generic]
# package-generic-done
ok 2 - output match 'package-generic-done'
1..2
ok
All tests successful.
Files=1, Tests=2,  2 wallclock secs ( 0.02 usr  0.00 sys +  0.67 cusr  0.20 csys =  0.89 CPU)
Result: PASS
running task <install-CGI> plg <cpan-package>
parameters: {list => CGI}
/opt/sparrow/tmp/10304/opt/sparrow/plugins/public/cpan-package/story.t ..
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI ...
# CGI is up to date. (4.31)
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 2 - output match 'cpan-package-done'
1..2
ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.18 cusr  0.05 csys =  0.24 CPU)
Result: PASS

Ok, now it succeeded!

A few comments on package-generic usage:

  • A limited set of platforms are supported - CentOS, Ubuntu, Debian. At least this works for me. If you need more - let me know or make a pull request!

  • No "platform-sensitive" packages naming conventions available like in other more mature systems ( chef, ansible ).

One day I would like something like that into sparrowdo API:

task_run  %(
  task => 'install apache webserver',
  plugin => 'package-generic',
  parameters_for_platform => %(
    ubuntu => %( list => 'apache2' ),
    debian => %( list => 'apache2' ),
    cetnos => %( list => 'httpd' ),
    default => %( list => 'apache2' ),

  ),
);
  • And finally. No specific version installation is supported. The plugin is quite simple. But it works for me. If you consider more complicated cases - let me know! :)

Cleaning up old packages

In case you have Ubuntu, Debian boxes a latest version of package-generic plugin make it possible to autoremove old, useless stuff:

task_run  %(
  task => 'remove old packages',
  plugin => 'package-generic',
  parameters => %( action => 'autoremove' )
);

That is it. Approaching to the end ... :)


All the best. And see you soon. I have not thought about next topic too much, but probably I will be talking about managing users and groups with the help of sparrowdo ...