Sparrowdo automation. Part 4. Managing users and groups.

Hi! These are the previous articles:

This one is going to be quite short. We will learn how to manage linux users and groups using sparrowdo.

Let's get back to our ubiquitous example of installing CPAN modules:

task_run  %(
  task => 'install 2 modules',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI DBI'
  ),
);

If we run this one we will see that CGI and DBI CPAN modules get installed into system paths, which is ok.

Consider a following case though:

  • create a user account

  • install some CPAN modules locally for this user

Basically this means that once a user gets logged into his account he should be able to use his modules:

$ ssh -l some-user $some-host
$ export PERL5LIB=~/lib/perl5/
$ perl -MDBI -e 1
$ perl -MCGI -e 1

Let's rewrite our latest code adding user account here:

task_run  %(
  task => 'create user',
  plugin => 'user',
  parameters => %( name => 'foo' )
);


task_run  %(
  task => 'install 2 modules',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI DBI',
    user => 'foo',
    install-base => '~/'
  ),
);

And have it run:

$ sparrowdo --ssh_user=vagrant  --ssh_port=2222 --host=127.0.0.1  
running sparrow tasks on 127.0.0.1 ... 
running task <create user> plg <user> 
parameters: {name => foo}
/tmp/.outthentic/1563/opt/sparrow/plugins/public/user/story.t .. 
# [/opt/sparrow/plugins/public/user/modules/create]
# uid=1002(foo) gid=1002(foo) groups=1002(foo)
# user created
ok 1 - output match 'user created'
# [/opt/sparrow/plugins/public/user]
# done
ok 2 - output match 'done'
1..2
ok
All tests successful.
Files=1, Tests=2,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.09 cusr  0.09 csys =  0.21 CPU)
Result: PASS
running task <install 2 modules> plg <cpan-package> 
parameters: {install-base => ~/, list => CGI DBI, user => foo}
/tmp/.outthentic/1659/opt/sparrow/plugins/public/cpan-package/story.t .. 
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI into ~/ user foo ...
# Successfully installed Test-Deep-1.120
# Successfully installed Sub-Uplevel-0.25
# Successfully installed Test-Warn-0.30
# Successfully installed CGI-4.32
# 4 distributions installed
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install DBI into ~/ user foo ...
# Successfully installed DBI-1.636
# 1 distribution installed
# install ok
ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3
ok
All tests successful.
Files=1, Tests=3, 68 wallclock secs ( 0.02 usr  0.01 sys + 46.61 cusr  9.56 csys = 56.20 CPU)
Result: PASS

Let's check our freshly installed CPAN modules:

$ vagrant ssh
Last login: Mon Jul 25 08:58:21 2016 from 10.0.2.2
Welcome to your Vagrant-built virtual machine.
[vagrant@epplkraw0312t1 ~]$ sudo bash 
[root@epplkraw0312t1 vagrant]# su - foo
Last login: Пн июл 25 09:02:55 UTC 2016 on pts/1
[foo@epplkraw0312t1 ~]$ PERL5LIB=~/lib/perl5/ perl -MCGI -e 1
[foo@epplkraw0312t1 ~]$ PERL5LIB=~/lib/perl5/ perl -MDBI -e 1

Ok. We successfully installed desired stuff into the user's account.

And last thing in this tutorial. Managing linux groups. The code is going to be very simple, as with creating users:

task_run  %(
  task => 'create group',
  plugin => 'group',
  parameters => %( name => 'application',
  ),
);

Goodbye and see you soon at our next tutorial. Have not yet decided what it's going to be ;)

Sparrowdo automation. Part 3. Installing system packages.

HI! I continue blogging about sparrowdo - a simple perl6 configuration management tool.

This is what we've learned so far:

Installing packages

Consider out latest example with installing CPAN packages:

$ cat sparrowfile

use v6;

use Sparrowdo;


task_run  %(
  task => 'install CGI',
  plugin => 'cpan-package',
  parameters => %( list => 'CGI' ),
);

What we are trying to do here is to install CGI CPAN module using cpan-package plugin. Here is little trick is hidden. A cpan-package implies you have a cpanm client pre-installed at your system. Let's see what will happen if it is not:

$ sparrowdo --ssh_port=2200 --host=127.0.0.1 --ssh_user=vagrant 
running sparrow tasks on 127.0.0.1 ... 
running task <install-CGI> plg <cpan-package> 
parameters: {list => CGI}
/opt/sparrow/tmp/8417/opt/sparrow/plugins/public/cpan-package/story.t .. 
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI ...
# /opt/sparrow/plugins/public/cpan-package/modules/cpanm/story.bash: line 22: cpanm: command not found
not ok 1 - scenario succeeded

#   Failed test 'scenario succeeded'
#   at /usr/local/share/perl/5.20.2/Outthentic.pm line 193.
not ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3

#   Failed test 'output match 'install ok''
#   at /usr/local/share/perl/5.20.2/Outthentic.pm line 239.
# Looks like you failed 2 tests of 3.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/3 subtests 

Test Summary Report
-------------------
/opt/sparrow/tmp/8417/opt/sparrow/plugins/public/cpan-package/story.t (Wstat: 512 Tests: 3 Failed: 2)
  Failed tests:  1-2
  Non-zero exit status: 2
Files=1, Tests=3,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.10 cusr  0.00 csys =  0.12 CPU)
Result: FAIL
The spawned process exited unsuccessfully (exit code: 1)
  in block <unit> at /home/melezhik/.rakudobrew/moar-nom/install/share/perl6/site/resources/CDEEEB90951FEDF24FB0EE4EFB79E7A6061B289C line 3

Upps, obviously we see the reason of the failure here:

cpanm: command not found

So, we need to install cpanm client to our system first to make it working cpan-package plugin. One day I probably add this dependency into cpan-package itself , but for now we could use a package-generic plugin to install any system packages.

Luckily we have cpanminus ubuntu package to install cpanm client on our ubuntu box:

$ cat sparrowfile

use v6;

use Sparrowdo;

task_run  %(
  task => 'install cpanm client',
  plugin => 'package-generic',
  parameters => %( list => 'cpanminus' )
);

task_run  %(
  task => 'install CGI',
  plugin => 'cpan-package',
  parameters => %( 
    list => 'CGI',
  ),
);

Let's give it a run ...

$ sparrowdo --ssh_port=2200 --host=127.0.0.1 --ssh_user=vagrant
running sparrow tasks on 127.0.0.1 ...
running task <install cpanm client> plg <package-generic>
parameters: {list => cpanminus}
/opt/sparrow/tmp/8792/opt/sparrow/plugins/public/package-generic/story.t ..
# [/opt/sparrow/plugins/public/package-generic/modules/apt-get]
# trying to install cpanminus ...
# installer - apt-get
# Package: cpanminus
# Version: 1.7014-1
# Status: install ok installed
ok 1 - output match 'Status: install ok installed'
# [/opt/sparrow/plugins/public/package-generic]
# package-generic-done
ok 2 - output match 'package-generic-done'
1..2
ok
All tests successful.
Files=1, Tests=2,  2 wallclock secs ( 0.02 usr  0.00 sys +  0.67 cusr  0.20 csys =  0.89 CPU)
Result: PASS
running task <install-CGI> plg <cpan-package>
parameters: {list => CGI}
/opt/sparrow/tmp/10304/opt/sparrow/plugins/public/cpan-package/story.t ..
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI ...
# CGI is up to date. (4.31)
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 2 - output match 'cpan-package-done'
1..2
ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.18 cusr  0.05 csys =  0.24 CPU)
Result: PASS

Ok, now it succeeded!

A few comments on package-generic usage:

  • A limited set of platforms are supported - CentOS, Ubuntu, Debian. At least this works for me. If you need more - let me know or make a pull request!

  • No "platform-sensitive" packages naming conventions available like in other more mature systems ( chef, ansible ).

One day I would like something like that into sparrowdo API:

task_run  %(
  task => 'install apache webserver',
  plugin => 'package-generic',
  parameters_for_platform => %(
    ubuntu => %( list => 'apache2' ),
    debian => %( list => 'apache2' ),
    cetnos => %( list => 'httpd' ),
    default => %( list => 'apache2' ),

  ),
);
  • And finally. No specific version installation is supported. The plugin is quite simple. But it works for me. If you consider more complicated cases - let me know! :)

Cleaning up old packages

In case you have Ubuntu, Debian boxes a latest version of package-generic plugin make it possible to autoremove old, useless stuff:

task_run  %(
  task => 'remove old packages',
  plugin => 'package-generic',
  parameters => %( action => 'autoremove' )
);

That is it. Approaching to the end ... :)


All the best. And see you soon. I have not thought about next topic too much, but probably I will be talking about managing users and groups with the help of sparrowdo ...

Sparrowdo automation. Part 2. Dealing with http proxy servers.

This is a next post in Sparrowdo automation series.

The previous one was Installing CPAN packages.

Sometimes having internet restricted by http proxy usage could be a problem.

Recalling our latest post on installing CPAN packages, what if we have a server with http traffic only available under some http proxy server? Then we would fail to install any CPAN packages here, and would have a similar error from cpanm client:

$ cpanm CGI

! Finding CGI on cpanmetadb failed.
--> Working on CGI
Fetching http://www.cpan.org/authors/id/L/LE/LEEJO/CGI-4.31.tar.gz ... FAIL

Ok. Passing http proxy is possible when using sparrowdo. Let's get back to the cpan-package plugin.

$ cat sparrowfile

task_run  %(
  task => 'install CGI module',
  plugin => 'cpan-package',
  parameters => %(
    list => 'CGI',
    http_proxy =>  $Sparrowdo::HttpProxy,
    https_proxy => $Sparrowdo::HttpsProxy,
  ),
);

Now let's run sparrowdo with some http/https proxy server parameters:

$ sparrowdo --http_proxy=http://my.internal.proxy:3128 --https_proxy=http://my.internal.proxy:3128 --host=192.168.0.1

That is it. Now sparrowdo will succeed in installing CPAN module CGI on the target server with http proxy restricted internet.

Bye! And keep your eye on following sparrowdo tutorials ... Next I am going to talk about installing system packages using sparrowdo.

Sparrowdo automation. Part 1. Installing cpan packages.

Here I start a series of posts on sparrowdo configuration management tool usage.

Installing sparrowdo

Sparrowdo implies installing software on both master and target host. Following "push" approach one should install a sparrowdo on master host where from he starts a ssh sessions against target hosts to be configurable:

$ ssh master.host 
$ panda install Sparrowdo

Once you install a sparrowdo you need to install a sparrow client on every host where you want to start deployments. Sparrow acts like client to carry out deployment tasks initiated from master host.

$ ssh target.host 
$ cpanm Sparrow
$ yum install curl

Here is simple schema of master -> target hosts interaction:

  +-----------------+
  |                 |    ssh
  |                 |------------> < host-1 > 192.168.0.1
  | <master host>   |    ssh
  | {sparrowdo}     |------------------> < host-2 > 192.168.0.2
  |                 |    ssh 
  |                 |-----------------------> < host-N > 192.168.0.3
  |                 |
  +-----------------+


  +-------------+
  |             |
  | <host>      |
  |             |
  | {sparrow}   | 
  | {curl}      |
  |             |
  +-------------+

That is it. Now you are ready to start your very first deployment with sparrowdo. As the title of the post has we want to install cpan packages.

Sparrowdo scenarios

Sparrowdo scenarios are written in Perl6. Following Sparrowdo API basically all you need is to find a proper sparrow plugin and run it with parameters. Luckily we have a proper one to solve our task - cpan-package. In most cases Sparrowdo way to configure servers is quite simple - find a proper plugins to handle your specific tasks and use them. We could treat sparrow plugins like a useful primitives to accomplish a complex tasks. This conception is very similar to what in chef or puppet is called resources , or in ansible is called modules. In case you miss a plugin to solve your task you are encouraged to create a new one, in easy way, but this series of posts are more focused on sparrowdo end client usage rather on sparrow plugin development.

Ok, let get back to cpan-package plugin, and just use it in our sparrowdo scenario:

$ ssh master.host
$ cat sparrowfile

task_run  %(
  task => 'install some cpan packages',
  plugin => 'cpan-package',
    parameters => %(
    list => 'CGI DBI',
    install-base => '/opt/perl'
  )
);

The content of sparrowfile is quite self-explanatory. We want to install 2 packages CGI and DBI into /opt/perl install base directory. Here we go!

A few words about ssh setup

A sparrowdo ssh setup implies 2 things so everything works fine:

  • a user from master host has password-less ssh access to target host
  • a user has (password-less) sudo on target host

You may specify user explicitly when run sparrowdo scenario or current user would be used implicitly when connecting by ssh to target host. From the master host now we run:

$ sparrowdo -h target.host

He is what I have for my test target box:

$ sparrowdo --host=192.168.0.1
running sparrow tasks on 192.168.0.1 ...
running task <install some cpan packages> plg <cpan-package>
parameters: {install-base => /opt/perl, list => CGI DBI}
/tmp/.outthentic/5385/opt/sparrow/plugins/public/cpan-package/story.t ..
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install CGI into /opt/perl ...
# Successfully installed HTML-Parser-3.72 (upgraded from 3.64)
# Successfully installed Test-Deep-1.120
# Successfully installed Sub-Uplevel-0.25
# Successfully installed Carp-1.38 (upgraded from 1.11)
# Successfully installed Test-Warn-0.30
# Successfully installed CGI-4.31 (upgraded from 3.51)
# 6 distributions installed
# install ok
ok 1 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package/modules/cpanm]
# install DBI into /opt/perl ...
# Successfully installed DBI-1.636
# 1 distribution installed
# install ok
ok 2 - output match 'install ok'
# [/opt/sparrow/plugins/public/cpan-package]
# cpan-package-done
ok 3 - output match 'cpan-package-done'
1..3
ok
All tests successful.
Files=1, Tests=3, 91 wallclock secs ( 0.02 usr  0.01 sys + 72.58 cusr  8.26 csys = 80.87 CPU)
Result: PASS

A few things to say here:

  • A sparrowdo reports are TAP reports due to TDD nature of sparrow plugins.
  • You may have extra information here by adding --verbose parameter for sparrowdo client to see a ssh commands details and some other indeterminate steps taken by sparrowdo when deploying your server.

So, keep your eye on following sparrowdo presentations.

Regards.


Alexey

sparrowdo sandbox

Hi all!

Sparrowdo is a new born #perl6 project for configuration management automation. It based on sparrow plugin system.

A user unfamiliar with sparrow or sparrowdo may check out a sparrowdo-test repository containing a ready to use sparrowdo scenarios for some typical cases , like installing packages, checking linux process and so on ...

So to start use it:

  • install sparrowdo on your "master" host
  • install sparrow ( vers >= 0.1.11 ) on your "target" host

On your "master" host:

  • git clone https://github.com/melezhik/sparrowdo-test
  • cd ./sparrow-test/example/dir
  • sparrowdo --host=target-host # see other options in sparrowdo documentation

Basically I have tested it on some ubuntu/centos/debian boxes, if you have any issues let me know ...

Regards.

Alexey