Results tagged “ubic”

/usr/local/bin rant

So, imagine you're a CentOS user, and you want to install a CPAN module into the system. Let's say you want to install cpanm.

You get the root shell with sudo -s.
You enter: cpan App-cpanminus.
And then you enter: cpanm --help... and oops:

[root@localhost vagrant]# cpanm --help
bash: cpanm: command not found


Ubic - status report

It's been almost a year since my last Ubic post.
So just a quick remainder: Ubic is a polymorphic service manager which makes creating daemons easy, while being extensible in a several different ways.
(github; cpan).

This post is going to be pretty long.
If the rest of it is tl;dr for you, but you have any opinion about what constitutes a perfect service manager / daemonizer tool, or about what would convince you to use one, please consider commenting anyway :)

Technical improvements

The most important improvement was the addition of ini config files. Now you can write this:
# cat /etc/ubic/service/foo.ini
bin = sleep 100
instead of this:
# cat /etc/ubic/service/foo
use Ubic::Service::SimpleDaemon;
  bin => 'sleep 1000',
This is important for reaching to people outside of perl community. But I'll get back to this a bit later. (Of course, perl-style configs are not going anywhere.)

Ubic::Service::SimpleDaemon now accepts 'cwd' and 'env' options.
There's still a lot SimpleDaemon can't do, but I'm considering renaming it to Ubic::Service::Daemon before it's too late.

Thanks to Dmitry Yashin, we got the proper FreeBSD port.

And then there's some smaller stuff:

  • Command-line ubic script now doesn't require '--force' when operating multiservices; this feature was annoying, so I removed it. (--force is still required for starting/stopping *everything*, though.)

  • Dmitry Yashin helped to get rid of LWP::UserAgent dependency by implementing the tiny Ubic::UA http useragent;

  • process guid (identifier which makes ubic pidfiles safer) calculation code was broken for a couple of releases, but I finally fixed it now;

  • there was an annoying bug on ext4, which broke services after reboot because empty pidfiles were considered invalid (I hate non-transactional POSIX filesystem semantics...);

  • finally, I butchered quite a few releases with stupid bugs and typos; now I know better and always upload trial releases first :) (cpantesters is great, even though it's a bit unstable lately);

There were also several releases of various Ubic::Service::* distributions:

  • Yury Zavarin wrote Ubic::Service::ZooKeeper and Ubic::Service::MongoDB for running Zookeeper and MongoDB as services;

  • I wrote Ubic::Service::InitScriptWrapper as a fun proof-of-concept for representing init scripts as ubic services (why would you want to do that? well, you would magically get your service monitored by the watchdog, at very least... also, colored command-line ubic script.)

  • William Wolf uploaded Ubic::Service::Starman just two days ago. I've been very happy to discover it - this is the first time someone not from our Yandex team releases an Ubic::Service::* extension.

Marketing :)

I couldn't go to YAPC::EU last year, so Alex Kapranoff gave the talk about Ubic for me.

This year at Perl Workshop in Israel, Thomas Maier showed these slides.

I myself talked about Ubic twice in Moscow - on DevConf 2011 (there were less than 10 people there, so I almost forgot about it).
And then I gave the talk at my $job at Yandex.
Yandex is pretty big these days, and now there are 11 projects using Ubic, and even more are going to; there were just 4 a year ago (Hope I won't get fired for violating my NDA ;) ).

Another cool thing that happened: Dancer now lists Ubic as one of the ways to create a service from an app.

All in all, I feel like this is still not enough :)
There are just 5 people on's #ubic channel, including me.
There are no regular contributors besides me. (This is not a problem per se, but I usually hesitate to add features unless I'm going to use it or I know a user who's going to use it. Otherwise, it's just the dead code. "Every good work of software starts by scratching a developer's personal itch.").
So what am I going to do about it?

Where are we going now?

There are two things I'm trying to achieve with Ubic:

  • Providing the flexible and extensible platform;

  • Giving a useful functionality out-of-the-box.

Ubic was always intended to be flexible and extensible. Earlier its single-line description was "flexible perl-based service manager". I recently replaced it with "polymorphic service manager".
Replacing "flexible" with "polymorphic" is a minor change, removing "perl-based" from description is more important. It means Ubic can be used by people outside of perl community.
And at Yandex, they already do.
Our NodeJS guys use it and even try to promote it to Node community. Our sysadmins use it and generally like it. We even rewrote some system init scripts with it. (rsync init script, for example - it died because of OOM sometimes, it was pretty annoying.)
So, please recommend Ubic to non-perl folks you know :)

BTW, what does "flexible" mean?
There are 3 aspects for it:

  • ubic services are represented as perl objects which conform to Ubic::Service interface, but can have different implementations;

  • ubic service tree is usually populated by loading files from /etc/ubic/service/, but by using multiservices mechanism it can be populated dynamically;

  • you can describe services with .ini configs, and can even implement your own config loaders (implement Ubic::ServiceLoader::Ext::js, describe services in js)

Of course, Ubic still have to be extended in Perl.
...Unless you bridge Perl-based Ubic API to some IPC API,using Ubic::Service::InitScriptWrapper or some other implementation. (Ubic started as LSB-compatible service manager, but that's not the main reason for its existence anymore. You could implement Ubic::Service analogue in JS, or in IPC but return status not via exit code but via stdout.)

There are many things you can do once all your services are accessible via common API:

  • global watchdog (included in core Ubic distribution)

  • http ping service (included in core Ubic distribution)

  • html frontend with start/stop buttons (here's an attempt to build such frontend; unfortunately, its development is on indefinite hiatus right now)

  • decorating services with additional functionality (you can do it with inheritance/composition)

  • binding services from other service managers to ubic; Ubic::Service::InitScriptWrapper does this for init scripts, and we can try the similar approach for daemontools, upstart, etc.

Of course, you can't just invent an API and expect people to start using it.
Nobody would adopt PSGI if there were no Plack; you have to provide some useful stuff out-of-the-box.
So, without promising anything, here's a list of things I'd like to implement next:

  • Lots of people ask me for ulimit support in SimpleDaemon;

  • Related to the previous point, some people get confused by ubic's user/group semantics (ubic calls setuid to user as soon as possible, not after forking to daemon's process); so, we need to support the tranditional behaviour as well;

  • I'd love to add Windows support one day, and then declare Ubic "the only cross-platform service manager"... unfortunately, my knowledge of Windows is lacking;

  • reload_signal option;

  • parallel restart;

  • ubic fix-permissions command;

  • YAML configs (since ini configs allow only one level of options, and some SimpleDaemon options already require more (for example, 'env'))

I'm also thinking from time to time about adding a common mechanism for adding additional functionality to services without explicit inheritance.
Something like roles.
For example: "do you want to restart your daemon when code changes? take Ubic::Service::SimpleDaemon and add Ubic::Middleware::CodeChange". Or you might want to restart on memory leaks. Or simply once in a while.
We could then support such middlewares (wrappers? roles?) in ini/yaml configs, so that even non-perl users would use them.

Last but not least, I'm working on a proper website for Ubic.
I'm not ready to show it to public just yet, but hopefully it'll be ready soon.



  • Ubic is not just for perl folks now; please recommend it to other people who might want to try it;

  • There are many features, but even more is to be done, and the main restraining factor right now is the number of active users and contributors (I don't want to add stuff just because someone may need it...).

So I'll repeat my initial question once again: how do you think a perfect service manager should work? What would convince you to use one? Which important features are missing in Ubic right now?

Ubic progress - manuals, easy installation, cross-platform

(For those people who don't know about it yet - ubic is a flexible, powerful, extensible perl-based service manager. Something like upstart or daemontools, but better.)

Lots of cool stuff happened in last few months.

First, ubic is now cross-platform and runs on *bsd, Mac OS X and pretty much any posix-compatible OS out there (no Windows yet, sorry).

Second, installation is now as easy as possible. Just do "cpanm Ubic && ubic-admin setup", and you are done. (Pass "--batch-mode" to setup command if questions annoy you). It's even possible to install ubic in home folder without root access and use it with perlbrew or whatever you like. Even on shared hosting, if you have the ability to edit crontab there.

Third, 1.30 release finally got some new manuals. They are far from perfect, but I believe they are much better than old PODs with which it was pretty obscure for new user to figure out where to start.

If you didn't try it earlier, because of incomplete docs or complicated installation procedure, now would be a good time to try again.
If this is the first time you heard about ubic, it's also a good time to try it.

Join us at if you have anything to say, I'm always greedy for any kind of feedback :)

PS: There will be the talk on Ubic on YAPC::Europe this year. I won't be there, but kappa will tell you everything you need to know. Don't miss it.


Ever since I've promised to write about my "generic streams" framework on #moose IRC channel (it happened several weeks ago), my conscience wouldn't leave me alone.
I'm still not ready to talk about streams with enough details, so I'll talk about the bigger picture instead.

So, I'm working at Yandex company on a quite large-scale project (it's a blogsearch if you really want to know). Our codebase is separated into something like 400 separate distributions, mostly perl. Every distribution is packaged into debian package and all these packages are cross-dependent in the complicated ways. We've got several hundreds of production hosts, separate test hosts and the separate test cluster. Our group is just a small part of the whole company (less than 1 percent in fact) and so some of our modules are used outside of our blogsearch project.

Why am I telling you all this? Because the software we write and especially the software which we release in the opensource is very much shaped by this highly diverse environment. In the last few years, we encountered some common patterns in the areas of service management, software configuration and streaming data processing again and again, and came up with some interesting (IMHO) and generic solutions.

So here are three things which we can share with the world:

First, there is service management. As you can imagine, we run a lot of daemons - http services and some other stuff, and so we wrote Ubic. I wrote about it many times already, and there is a fairly high probability that you already know what Ubic is. If you don't know, just look at these slides.

Second, there is software configuration. Having a production cluster and testing cluster and separate configuration for the unit testing and maintaining it all for 400 distributions is a nightmare... unless you invent something like Morpheus. Morpheus is the "ultimate configuration engine" - it provides the unified interface to any configuration values, which then can be provided from any source (config files in any format, DB, environment, command line arguments) without any changes to code at all.
Think "Log::Any", but apply its idea to the configs.
I'm really excited about Morpheus and I'll talk about it in detail on Saint Perl 2 in Russia, in St. Petersburg on December, 18. I'll have some slides by then, so if you like the Morpheus idea too and want to know more, just wait for 2 weeks.
(If you are really impatient, you can go on and read the code, but there are almost no docs by now).

Third, streaming data processing.
Creating blogsearch means that you have to fetch a lot of web pages, then put them into storage, then index all the stuff you put into storage, and also extract links from this "new posts" stream, then process these links more (expand short links, for example), also export link to another services, also use these links to calculate antispam metrics, etc.
In my mind, it looks like a giant graph with vertices representing data stored on different hosts and groups of hosts in various forms (it can be log or it can be a DB table, or it can be a rsync share on another host), and nodes between them representing processing programs.
Also, I like to think about this picture.
Thus, streaming framework. Or more like set of libraries, some of which define common APIs, some providing specific implementations (Stream::Log, Stream::DB, etc), some helping to integrate all this stuff into the big picture. I still don't know when I'll find the will to open it into public, not because it's not ready - we are using it in production - but because I'm still trying to find the right approach to decouple it and separate useful parts from the policy as much as possible, and keep it easy to use and deploy at the same time.

Besides these three major directions we also have several small useful modules.
Some of them are already out there in separate distrubtions (Log::Unrotate, AnyEvent::HTTP::LWP::UserAgent, DBD::Safe), some are on their way.
About two of these small modules I'm uncertain if they would be useful to anyone as separate distributions, but they are still too helpful and I had to bundle them with Ubic: Ubic::Lockf, Ubic::Persistent. If you like one of them, tell me and I'll upload them immediately as the separate distributions. Especially if you'll suggest an appropriate name :)

Well... that's all by now.

New Ubic slides

Here are the slides from my recent talk at meeting.
They are more detailed and hopefully less cryptic than my inital presentation.

Ubic - multiservices

(This is a long time overdue post in Ubic series. Previous posts can be found here.)

Last time I explained how any kind of service start/stop/status behavior can be encapsulated in an Ubic::Service subclass. But what if you have many similar services, and want to generate the service list dynamically?

For example, sometimes you already have a directory with daemon configs, and want to run them all without creating additional "service description" file per daemon.
Or you may wish to start ten instances of one processing script.
Or maybe you keep the list of your services in SQL database on another host.

In all these cases, multiservices come to the rescue.
Multiservice is a container object which provides part of the service tree.

For example, let's assume you have a dir /usr/share/psgi-apps/ with some psgi apps in it:

$ ls /usr/share/psgi-apps/
hello-14001.psgi world-14002.psgi

Future HTTP port number in this example is encoded in the file name.

Let's map these apps into ubic service tree:

# content of /etc/ubic/service/psgi-apps file

use parent qw(Ubic::Multiservice);
use Ubic::Service::Plack;

sub new {
return bless {} => shift;

sub service_names {
my @files = glob "/usr/share/psgi-apps/*.psgi";
s{^/usr/share/psgi-apps/(.*)-\d+.psgi$}{$1} for @files;
return @files;

sub simple_service {
my $name = $_[1];
my ($file) = glob "/usr/share/psgi-apps/$name-*.psgi";
my ($port) = $file =~ /(\d+)\.psgi$/;
return Ubic::Service::Plack->new({
server => 'Standalone',
app => $file,
app_name => $name,
port => $port,
stdout => "/var/log/psgi-apps/$name.log",
stderr => "/var/log/psgi-apps/$name.err.log",

return __PACKAGE__->new;

Assuming that /var/log/psgi-apps/ dir exists, this code is enough to run existing *and* all future psgi apps in /usr/share/psgi-apps/ dir.
"service_names" method returns plain list of service names, and "simple_service" builds service by name. It's really simple :)

Now, all psgi apps can be started simultaneously with one command:

$ sudo ubic start -f psgi-apps
Starting psgi-apps.hello... started
Starting started

... or one by one:

$ sudo ubic stop
Stopping stopped

When someone is performing some global action, multiservices are grouped in indented trees:

$ sudo ubic status
ubic-ping running (pid 18641)
sleeping-daemon running (pid 8056)
psgi-apps.hello running off

And of course you can create several levels of nesting too if you need them.

There are two multiservice classes in core Ubic distribution.

First, there is a Ubic::Multiservice::Simple class, which can be used for cases when you just want to group several services together:

# content of /etc/ubic/service/my
use Ubic::Multiservice::Simple;
return Ubic::Multiservice::Simple->new({
fcgi_app => $fcgi_app_service,
memcached => $memcached_service,

It's also handy for generating several identical workers:

# content of /etc/ubic/service/ten-workers
use Ubic::Multiservice::Simple;
use Ubic::Service::SimpleDaemon;
return Ubic::Multiservice::Simple->new({
map {(
"worker$_" => Ubic::Service::SimpleDaemon->new({
bin => "",
)} (1..10)

Second, there is a Ubic::Multiservice::Dir. Nobody usually uses it directly, but it is constructed for every subdir in /etc/ubic/service/, so that /etc/ubic/service/a/b file is mapped to "a.b" service.

So, summing it up, multiservices are useful for:
- grouping services together for easier management;
- dynamically generating services from external sources;
- running a large number of identical worker processes.
They really simplify some complex tasks and make ubic as unobtrusive and extensible as it is possible.

PS: PSGI example requires Ubic=1.22 and Ubic::Service::Plack=1.12. I made multiservice API slightly simpler while writing this post, and fixed one bug in Ubic::Service::Plack. Both of these modules should be available on CPAN soon.

PPS: Contacts:
our irc channel is #ubic at;
source code is at;
mailing list is at

If you are using ubic or have any questions, let me know. It motivates me SO much :)

Ubic - code reuse and customizations

In my previous post I showed most trivial example of ubic service.
Now it's time to talk about more interesting stuff :)

Every ubic service is simply a perl object which implements start/stop/status methods.
It is very similar to /etc/init.d shell scripts, but since perl is a real programming language, you don't have to copy-paste tons of boilerplate code every time (Dave Rolsky, if you're reading this - your silki init script is good, but "apache2-backend" line in comments means it was copy-pasted too; is there anyone who can actually write proper init script by hands?).

So, instead of copy-pasting, you can reuse any existing Ubic::Service::* module, or implement your own if it's necessary.
For example, there is the Ubic-Service-Plack distribution on CPAN which you can use to run any PSGI applications.
Quoting Ubic::Service::Plack synopsis:

use Ubic::Service::Plack;
return Ubic::Service::Plack->new({
server => "FCGI",
server_args => { listen => "/tmp/app.sock",
nproc => 5 },
app => "/var/www/app.psgi",
app_name => 'app',
status => sub { ... },
port => 4444,
ubic_log => '/var/log/app/ubic.log',
stdout => '/var/log/app/stdout.log',
stderr => '/var/log/app/stderr.log',
user => "ppb",

Notice the 'status' coderef. It is optional, but if provided, it can be used as custom status check:

status => sub {
use IO::Socket::UNIX;
use IO::Select;
use Ubic::Result qw(result);
my $socket = IO::Socket::UNIX->new(
Peer => $self->{socket},
Timeout => 5,
) or return 'broken';
$socket->send(pack("C*", 1,9,0,0 ,0,0,8,0 , 0,0,0,0, 0,0,0,0));
my $content = "";
my $select = IO::Select->new($socket);
unless ($select->can_read(5)) {
return result('broken', 'socket timeout');
$socket->recv($content, 8);
my @bytes = unpack("C8", $content);
if ($bytes[1] == 10) {
return 'running';
else {
return 'broken';

We ping FCGI socket using FCGI_GET_VALUES request (thanks to unknown stackoverflow guy for this method).
In case process responds properly, 'running' string is returned, and 'broken' string is returned in other cases.
We don't have to check if process is actually running and if pidfile is valid - Ubic::Service::Plack does this for us.

This check will be called several times with increasing time intervals until status will become 'running'. Time intervals and number of checks until ubic will give up are configurable too (actually, they are implemented in Ubic::Service::Common class, which can be base class for most ubic services unless you want to do something really complex).

Now, if you have tons of PSGI applications which you run as fastcgi servers, you can implement Ubic::Service::Plack::FCGI on top of Ubic::Service::Plack.
Or maybe you are running tons of HTTP servers which all implement http://.../ping http check.
Or you may wish to restart your service on some external conditions, like Steven Haryanto wants to.
All these things can be implemented using custom status checks and then encapsulated via ineritance or delegation.

Of course, 'start', 'stop' and 'reload' methods can be replaced as well.

One last bit of information (it should have been in first post but i forgot to mention it): any ubic service can be turned into SysV init script.
Simply put this line in /etc/init.d/ and you are done:

# cat >/etc/init.d/any-ubic-service
use Ubic::Run;

Yes, that's enough to get LSB-compliant init script corresponding to ubic service named 'any-ubic-service'. Isn't ubic just great? :)

Next time I'll talk about multiservices and custom commands.
Also, there are Ubic::Service::Memcached and Ubic::Service::Lighttpd I still have to upload.

Don't forget to join our mailing list if you are interested in ubic and service managing stuff!

Ubic - how to implement your first service

I had a hidden agenda when I asked on this blog about how do you run your daemons.

There is a tool I recently opensourced. It can be compared to classic SysV init script system or to daemontools, runit or upstart, but it already is better and more flexible at least in some of its properties (otherwise I wouldn't bother to implement it :) ).

It is called Ubic and it is a toolkit for describing daemons in pure perl.

Ubic service is simply a perl object of some class inherited from base Ubic::Service class.
Ubic loads service descriptions from /etc/ubic/service/ directory.
To create your first service, you have to write something like this:

# cat >/etc/ubic/service/test
use Ubic::Service::SimpleDaemon;
return Ubic::Service::SimpleDaemon->new({ bin => "sleep 1000" });

"ubic status" command can show you status of all or some services on host:

# ubic status
test off
ubic-ping running
# ubic status test
test off

Now, as you probably already guessed, to start service you simply have to say "ubic start":

# ubic start test
Starting test... started
# ubic status
test running

What are you getting for free with this simple example?

  • ubic-watchdog - generic program that runs every minute from cron and checks every service; when 1000 seconds of sleep will be over, this 'test' service will be restarted automatically;

  • LSB compliance (it is strange how many /etc/init.d/ services even on latest Ubuntu distributions can't handle double start correctly and don't implement status check);

  • Neat colors :) You can't see them in examples above, but they are there;

  • HTTP ping service:

    # wget -q -O - 'http://localhost:12345/status/service/test'

    If you don't need it, you can always turn it off with 'ubic stop'. Or you can try to read /etc/ubic/service/ubic-ping for spoilers about more complex ubic features :)

By this point in my story:
- init.d people should already think that ubic is cool ;)
- daemontools/runit users will probably require more persuading, so I'm going to continue tomorrow.

PS: Installation note: please read "installation" section in file before installing.

PPS: There is a mailing list, don't hesitate to join and ask your questions.

PPPS: This is my first opensource module which i'm trying to attract attention to, so any meta-advices from guys with tons of popular CPAN distributions about what i'm doing right/wrong or how else can I persuade anyone to try ubic are very welcome too :)

How do you run your daemons?

How do you start and daemonize your programs (PSGI apps, fastcgi apps, gearmand, several memcached instances, custom handcoded daemons)?

Do you write custom init scripts?
Use daemontools or upstart?
Do you simply invoke plackup in terminal? :)
Maybe you spawn your fcgi apps directly from your webserver?

Do you need watchdogs? Do you actually write them?

Announcing Ubic

Ubic is a toolkit for describing services on Linux in perl. It's been implemented in-house at Yandex as a better way to run daemons than writing ugly SysV-style /etc/init.d scripts, and i've put it on github and CPAN recently.

I also gave a lightning talk about it at YAPC::Russia::2010 in Kiev several days ago, it should explain most of its traits:

So... any feedback? :)

PS: Several Ubic::Service::* modules are not in public domain yet - i'll have to do some cleanup first, hopefully in next week or two.


About Vyacheslav Matyukhin

user-pic I wrote Ubic. I worked at Yandex for many years, and now i'm building my own startup (formerly PlayPerl). I'm also working on Flux, streaming data processing framework. CPAN ID: MMCLERIC.