Results tagged “tutorial”

Ubic - multiservices

(This is a long time overdue post in Ubic series. Previous posts can be found here.)

Last time I explained how any kind of service start/stop/status behavior can be encapsulated in an Ubic::Service subclass. But what if you have many similar services, and want to generate the service list dynamically?

For example, sometimes you already have a directory with daemon configs, and want to run them all without creating additional "service description" file per daemon.
Or you may wish to start ten instances of one processing script.
Or maybe you keep the list of your services in SQL database on another host.

In all these cases, multiservices come to the rescue.
Multiservice is a container object which provides part of the service tree.

For example, let's assume you have a dir /usr/share/psgi-apps/ with some psgi apps in it:


$ ls /usr/share/psgi-apps/
hello-14001.psgi world-14002.psgi

Future HTTP port number in this example is encoded in the file name.

Let's map these apps into ubic service tree:


# content of /etc/ubic/service/psgi-apps file

use parent qw(Ubic::Multiservice);
use Ubic::Service::Plack;

sub new {
return bless {} => shift;
}

sub service_names {
my @files = glob "/usr/share/psgi-apps/*.psgi";
s{^/usr/share/psgi-apps/(.*)-\d+.psgi$}{$1} for @files;
return @files;
}

sub simple_service {
my $name = $_[1];
my ($file) = glob "/usr/share/psgi-apps/$name-*.psgi";
my ($port) = $file =~ /(\d+)\.psgi$/;
return Ubic::Service::Plack->new({
server => 'Standalone',
app => $file,
app_name => $name,
port => $port,
stdout => "/var/log/psgi-apps/$name.log",
stderr => "/var/log/psgi-apps/$name.err.log",
});
}

return __PACKAGE__->new;

Assuming that /var/log/psgi-apps/ dir exists, this code is enough to run existing *and* all future psgi apps in /usr/share/psgi-apps/ dir.
"service_names" method returns plain list of service names, and "simple_service" builds service by name. It's really simple :)

Now, all psgi apps can be started simultaneously with one command:


$ sudo ubic start -f psgi-apps
psgi-apps
Starting psgi-apps.hello... started
Starting psgi-apps.world... started

... or one by one:


$ sudo ubic stop psgi-apps.world
Stopping psgi-apps.world... stopped

When someone is performing some global action, multiservices are grouped in indented trees:


$ sudo ubic status
ubic-ping running (pid 18641)
sleeping-daemon running (pid 8056)
psgi-apps
psgi-apps.hello running
psgi-apps.world off

And of course you can create several levels of nesting too if you need them.


There are two multiservice classes in core Ubic distribution.

First, there is a Ubic::Multiservice::Simple class, which can be used for cases when you just want to group several services together:


# content of /etc/ubic/service/my
use Ubic::Multiservice::Simple;
...
return Ubic::Multiservice::Simple->new({
fcgi_app => $fcgi_app_service,
memcached => $memcached_service,
});

It's also handy for generating several identical workers:


# content of /etc/ubic/service/ten-workers
use Ubic::Multiservice::Simple;
use Ubic::Service::SimpleDaemon;
return Ubic::Multiservice::Simple->new({
map {(
"worker$_" => Ubic::Service::SimpleDaemon->new({
bin => "worker.pl",
})
)} (1..10)
});

Second, there is a Ubic::Multiservice::Dir. Nobody usually uses it directly, but it is constructed for every subdir in /etc/ubic/service/, so that /etc/ubic/service/a/b file is mapped to "a.b" service.

So, summing it up, multiservices are useful for:
- grouping services together for easier management;
- dynamically generating services from external sources;
- running a large number of identical worker processes.
They really simplify some complex tasks and make ubic as unobtrusive and extensible as it is possible.

PS: PSGI example requires Ubic=1.22 and Ubic::Service::Plack=1.12. I made multiservice API slightly simpler while writing this post, and fixed one bug in Ubic::Service::Plack. Both of these modules should be available on CPAN soon.

PPS: Contacts:
our irc channel is #ubic at irc.perl.org;
source code is at http://github.com/berekuk/ubic;
mailing list is at http://groups.google.com/group/ubic-perl.

If you are using ubic or have any questions, let me know. It motivates me SO much :)

Ubic - code reuse and customizations

In my previous post I showed most trivial example of ubic service.
Now it's time to talk about more interesting stuff :)

Every ubic service is simply a perl object which implements start/stop/status methods.
It is very similar to /etc/init.d shell scripts, but since perl is a real programming language, you don't have to copy-paste tons of boilerplate code every time (Dave Rolsky, if you're reading this - your silki init script is good, but "apache2-backend" line in comments means it was copy-pasted too; is there anyone who can actually write proper init script by hands?).

So, instead of copy-pasting, you can reuse any existing Ubic::Service::* module, or implement your own if it's necessary.
For example, there is the Ubic-Service-Plack distribution on CPAN which you can use to run any PSGI applications.
Quoting Ubic::Service::Plack synopsis:


use Ubic::Service::Plack;
return Ubic::Service::Plack->new({
server => "FCGI",
server_args => { listen => "/tmp/app.sock",
nproc => 5 },
app => "/var/www/app.psgi",
app_name => 'app',
status => sub { ... },
port => 4444,
ubic_log => '/var/log/app/ubic.log',
stdout => '/var/log/app/stdout.log',
stderr => '/var/log/app/stderr.log',
user => "ppb",
});

Notice the 'status' coderef. It is optional, but if provided, it can be used as custom status check:


...
status => sub {
use IO::Socket::UNIX;
use IO::Select;
use Ubic::Result qw(result);
my $socket = IO::Socket::UNIX->new(
Peer => $self->{socket},
Timeout => 5,
) or return 'broken';
$socket->send(pack("C*", 1,9,0,0 ,0,0,8,0 , 0,0,0,0, 0,0,0,0));
my $content = "";
my $select = IO::Select->new($socket);
unless ($select->can_read(5)) {
return result('broken', 'socket timeout');
}
$socket->recv($content, 8);
my @bytes = unpack("C8", $content);
if ($bytes[1] == 10) {
return 'running';
}
else {
return 'broken';
}
},
...

We ping FCGI socket using FCGI_GET_VALUES request (thanks to unknown stackoverflow guy for this method).
In case process responds properly, 'running' string is returned, and 'broken' string is returned in other cases.
We don't have to check if process is actually running and if pidfile is valid - Ubic::Service::Plack does this for us.

This check will be called several times with increasing time intervals until status will become 'running'. Time intervals and number of checks until ubic will give up are configurable too (actually, they are implemented in Ubic::Service::Common class, which can be base class for most ubic services unless you want to do something really complex).

Now, if you have tons of PSGI applications which you run as fastcgi servers, you can implement Ubic::Service::Plack::FCGI on top of Ubic::Service::Plack.
Or maybe you are running tons of HTTP servers which all implement http://.../ping http check.
Or you may wish to restart your service on some external conditions, like Steven Haryanto wants to.
All these things can be implemented using custom status checks and then encapsulated via ineritance or delegation.

Of course, 'start', 'stop' and 'reload' methods can be replaced as well.


One last bit of information (it should have been in first post but i forgot to mention it): any ubic service can be turned into SysV init script.
Simply put this line in /etc/init.d/ and you are done:


# cat >/etc/init.d/any-ubic-service
#!/usr/bin/perl
use Ubic::Run;

Yes, that's enough to get LSB-compliant init script corresponding to ubic service named 'any-ubic-service'. Isn't ubic just great? :)


Next time I'll talk about multiservices and custom commands.
Also, there are Ubic::Service::Memcached and Ubic::Service::Lighttpd I still have to upload.

Don't forget to join our mailing list if you are interested in ubic and service managing stuff!

Ubic - how to implement your first service

I had a hidden agenda when I asked on this blog about how do you run your daemons.

There is a tool I recently opensourced. It can be compared to classic SysV init script system or to daemontools, runit or upstart, but it already is better and more flexible at least in some of its properties (otherwise I wouldn't bother to implement it :) ).

It is called Ubic and it is a toolkit for describing daemons in pure perl.

Ubic service is simply a perl object of some class inherited from base Ubic::Service class.
Ubic loads service descriptions from /etc/ubic/service/ directory.
To create your first service, you have to write something like this:


# cat >/etc/ubic/service/test
use Ubic::Service::SimpleDaemon;
return Ubic::Service::SimpleDaemon->new({ bin => "sleep 1000" });

"ubic status" command can show you status of all or some services on host:


# ubic status
test off
ubic-ping running
# ubic status test
test off

Now, as you probably already guessed, to start service you simply have to say "ubic start":


# ubic start test
Starting test... started
# ubic status
test running

What are you getting for free with this simple example?


  • ubic-watchdog - generic program that runs every minute from cron and checks every service; when 1000 seconds of sleep will be over, this 'test' service will be restarted automatically;

  • LSB compliance (it is strange how many /etc/init.d/ services even on latest Ubuntu distributions can't handle double start correctly and don't implement status check);

  • Neat colors :) You can't see them in examples above, but they are there;

  • HTTP ping service:

    # wget -q -O - 'http://localhost:12345/status/service/test'
    ok

    If you don't need it, you can always turn it off with 'ubic stop'. Or you can try to read /etc/ubic/service/ubic-ping for spoilers about more complex ubic features :)


By this point in my story:
- init.d people should already think that ubic is cool ;)
- daemontools/runit users will probably require more persuading, so I'm going to continue tomorrow.

PS: Installation note: please read "installation" section in README.md file before installing.

PPS: There is a mailing list, don't hesitate to join and ask your questions.

PPPS: This is my first opensource module which i'm trying to attract attention to, so any meta-advices from guys with tons of popular CPAN distributions about what i'm doing right/wrong or how else can I persuade anyone to try ubic are very welcome too :)

1

About Vyacheslav Matyukhin

user-pic I wrote Ubic. I worked at Yandex for many years, and now i'm building my own startup questhub.io (formerly PlayPerl). I'm also working on Flux, streaming data processing framework. CPAN ID: MMCLERIC.