So, imagine you're a CentOS user, and you want to install a CPAN module into the system. Let's say you want to install cpanm.
You get the root shell with sudo -s.
You enter: cpan App-cpanminus.
And then you enter: cpanm --help... and oops:
[root@localhost vagrant]# cpanm --help
bash: cpanm: command not found
WAT?
It's been almost a year since my last Ubic post.
So just a quick remainder: Ubic is a polymorphic service manager which makes creating daemons easy, while being extensible in a several different ways.
(github; cpan).
This post is going to be pretty long.
If the rest of it is tl;dr for you, but you have any opinion about what constitutes a perfect service manager / daemonizer tool, or about what would convince you to use one, please consider commenting anyway :)
# cat /etc/ubic/service/foo.ini [options] bin = sleep 100instead of this:
# cat /etc/ubic/service/foo
use Ubic::Service::SimpleDaemon;
Ubic::Service::SimpleDaemon->new({
bin => 'sleep 1000',
});
This is important for reaching to people outside of perl community. But I'll get back to this a bit later.
(Of course, perl-style configs are not going anywhere.)
Ubic::Service::SimpleDaemon now accepts 'cwd' and 'env' options.
There's still a lot SimpleDaemon can't do, but I'm considering renaming it to Ubic::Service::Daemon before it's too late.
Thanks to Dmitry Yashin, we got the proper FreeBSD port.
And then there's some smaller stuff:
There were also several releases of various Ubic::Service::* distributions:
I couldn't go to YAPC::EU last year, so Alex Kapranoff gave the talk about Ubic for me.
This year at Perl Workshop in Israel, Thomas Maier showed these slides.
I myself talked about Ubic twice in Moscow - on DevConf 2011 (there were less than 10 people there, so I almost forgot about it).
And then I gave the talk at my $job at Yandex.
Yandex is pretty big these days, and now there are 11 projects using Ubic, and even more are going to; there were just 4 a year ago (Hope I won't get fired for violating my NDA ;) ).
Another cool thing that happened: Dancer now lists Ubic as one of the ways to create a service from an app.
All in all, I feel like this is still not enough :)
There are just 5 people on irc.perl.org's #ubic channel, including me.
There are no regular contributors besides me. (This is not a problem per se, but I usually hesitate to add features unless I'm going to use it or I know a user who's going to use it. Otherwise, it's just the dead code. "Every good work of software starts by scratching a developer's personal itch.").
So what am I going to do about it?
There are two things I'm trying to achieve with Ubic:
Ubic was always intended to be flexible and extensible. Earlier its single-line description was "flexible perl-based service manager". I recently replaced it with "polymorphic service manager".
Replacing "flexible" with "polymorphic" is a minor change, removing "perl-based" from description is more important. It means Ubic can be used by people outside of perl community.
And at Yandex, they already do.
Our NodeJS guys use it and even try to promote it to Node community. Our sysadmins use it and generally like it. We even rewrote some system init scripts with it. (rsync init script, for example - it died because of OOM sometimes, it was pretty annoying.)
So, please recommend Ubic to non-perl folks you know :)
BTW, what does "flexible" mean?
There are 3 aspects for it:
Of course, Ubic still have to be extended in Perl.
...Unless you bridge Perl-based Ubic API to some IPC API,using Ubic::Service::InitScriptWrapper or some other implementation. (Ubic started as LSB-compatible service manager, but that's not the main reason for its existence anymore. You could implement Ubic::Service analogue in JS, or in IPC but return status not via exit code but via stdout.)
There are many things you can do once all your services are accessible via common API:
Of course, you can't just invent an API and expect people to start using it.
Nobody would adopt PSGI if there were no Plack; you have to provide some useful stuff out-of-the-box.
So, without promising anything, here's a list of things I'd like to implement next:
I'm also thinking from time to time about adding a common mechanism for adding additional functionality to services without explicit inheritance.
Something like roles.
For example: "do you want to restart your daemon when code changes? take Ubic::Service::SimpleDaemon and add Ubic::Middleware::CodeChange". Or you might want to restart on memory leaks. Or simply once in a while.
We could then support such middlewares (wrappers? roles?) in ini/yaml configs, so that even non-perl users would use them.
Last but not least, I'm working on a proper website for Ubic.
I'm not ready to show it to public just yet, but hopefully it'll be ready soon.
Summarizing:
So I'll repeat my initial question once again: how do you think a perfect service manager should work? What would convince you to use one? Which important features are missing in Ubic right now?
(For those people who don't know about it yet - ubic is a flexible, powerful, extensible perl-based service manager. Something like upstart or daemontools, but better.)
So.
Lots of cool stuff happened in last few months.
First, ubic is now cross-platform and runs on *bsd, Mac OS X and pretty much any posix-compatible OS out there (no Windows yet, sorry).
Second, installation is now as easy as possible. Just do "cpanm Ubic && ubic-admin setup", and you are done. (Pass "--batch-mode" to setup command if questions annoy you). It's even possible to install ubic in home folder without root access and use it with perlbrew or whatever you like. Even on shared hosting, if you have the ability to edit crontab there.
Third, 1.30 release finally got some new manuals. They are far from perfect, but I believe they are much better than old PODs with which it was pretty obscure for new user to figure out where to start.
So.
If you didn't try it earlier, because of incomplete docs or complicated installation procedure, now would be a good time to try again.
If this is the first time you heard about ubic, it's also a good time to try it.
Join us at irc.perl.org/#ubic if you have anything to say, I'm always greedy for any kind of feedback :)
PS: There will be the talk on Ubic on YAPC::Europe this year. I won't be there, but kappa will tell you everything you need to know. Don't miss it.
Ever since I've promised to write about my "generic streams" framework on #moose IRC channel (it happened several weeks ago), my conscience wouldn't leave me alone.
I'm still not ready to talk about streams with enough details, so I'll talk about the bigger picture instead.
So, I'm working at Yandex company on a quite large-scale project (it's a blogsearch if you really want to know). Our codebase is separated into something like 400 separate distributions, mostly perl. Every distribution is packaged into debian package and all these packages are cross-dependent in the complicated ways. We've got several hundreds of production hosts, separate test hosts and the separate test cluster. Our group is just a small part of the whole company (less than 1 percent in fact) and so some of our modules are used outside of our blogsearch project.
Why am I telling you all this? Because the software we write and especially the software which we release in the opensource is very much shaped by this highly diverse environment. In the last few years, we encountered some common patterns in the areas of service management, software configuration and streaming data processing again and again, and came up with some interesting (IMHO) and generic solutions.
So here are three things which we can share with the world:
First, there is service management. As you can imagine, we run a lot of daemons - http services and some other stuff, and so we wrote Ubic. I wrote about it many times already, and there is a fairly high probability that you already know what Ubic is. If you don't know, just look at these slides.
Second, there is software configuration. Having a production cluster and testing cluster and separate configuration for the unit testing and maintaining it all for 400 distributions is a nightmare... unless you invent something like Morpheus. Morpheus is the "ultimate configuration engine" - it provides the unified interface to any configuration values, which then can be provided from any source (config files in any format, DB, environment, command line arguments) without any changes to code at all.
Think "Log::Any", but apply its idea to the configs.
I'm really excited about Morpheus and I'll talk about it in detail on Saint Perl 2 in Russia, in St. Petersburg on December, 18. I'll have some slides by then, so if you like the Morpheus idea too and want to know more, just wait for 2 weeks.
(If you are really impatient, you can go on and read the code, but there are almost no docs by now).
Third, streaming data processing.
Creating blogsearch means that you have to fetch a lot of web pages, then put them into storage, then index all the stuff you put into storage, and also extract links from this "new posts" stream, then process these links more (expand short links, for example), also export link to another services, also use these links to calculate antispam metrics, etc.
In my mind, it looks like a giant graph with vertices representing data stored on different hosts and groups of hosts in various forms (it can be log or it can be a DB table, or it can be a rsync share on another host), and nodes between them representing processing programs.
Also, I like to think about this picture.
Thus, streaming framework. Or more like set of libraries, some of which define common APIs, some providing specific implementations (Stream::Log, Stream::DB, etc), some helping to integrate all this stuff into the big picture. I still don't know when I'll find the will to open it into public, not because it's not ready - we are using it in production - but because I'm still trying to find the right approach to decouple it and separate useful parts from the policy as much as possible, and keep it easy to use and deploy at the same time.
Besides these three major directions we also have several small useful modules.
Some of them are already out there in separate distrubtions (Log::Unrotate, AnyEvent::HTTP::LWP::UserAgent, DBD::Safe), some are on their way.
About two of these small modules I'm uncertain if they would be useful to anyone as separate distributions, but they are still too helpful and I had to bundle them with Ubic: Ubic::Lockf, Ubic::Persistent. If you like one of them, tell me and I'll upload them immediately as the separate distributions. Especially if you'll suggest an appropriate name :)
Well... that's all by now.
Here are the slides from my recent talk at Moscow.pm meeting.
They are more detailed and hopefully less cryptic than my inital presentation.
(This is a long time overdue post in Ubic series. Previous posts can be found here.)
Last time I explained how any kind of service start/stop/status behavior can be encapsulated in an Ubic::Service subclass. But what if you have many similar services, and want to generate the service list dynamically?
For example, sometimes you already have a directory with daemon configs, and want to run them all without creating additional "service description" file per daemon.
Or you may wish to start ten instances of one processing script.
Or maybe you keep the list of your services in SQL database on another host.
In all these cases, multiservices come to the rescue.
Multiservice is a container object which provides part of the service tree.
For example, let's assume you have a dir /usr/share/psgi-apps/ with some psgi apps in it:
$ ls /usr/share/psgi-apps/
hello-14001.psgi world-14002.psgi
Let's map these apps into ubic service tree:
# content of /etc/ubic/service/psgi-apps file
use parent qw(Ubic::Multiservice);
use Ubic::Service::Plack;
sub new {
return bless {} => shift;
}
sub service_names {
my @files = glob "/usr/share/psgi-apps/*.psgi";
s{^/usr/share/psgi-apps/(.*)-\d+.psgi$}{$1} for @files;
return @files;
}
sub simple_service {
my $name = $_[1];
my ($file) = glob "/usr/share/psgi-apps/$name-*.psgi";
my ($port) = $file =~ /(\d+)\.psgi$/;
return Ubic::Service::Plack->new({
server => 'Standalone',
app => $file,
app_name => $name,
port => $port,
stdout => "/var/log/psgi-apps/$name.log",
stderr => "/var/log/psgi-apps/$name.err.log",
});
}
return __PACKAGE__->new;
Assuming that /var/log/psgi-apps/ dir exists, this code is enough to run existing *and* all future psgi apps in /usr/share/psgi-apps/ dir.
"service_names" method returns plain list of service names, and "simple_service" builds service by name. It's really simple :)
Now, all psgi apps can be started simultaneously with one command:
$ sudo ubic start -f psgi-apps
psgi-apps
Starting psgi-apps.hello... started
Starting psgi-apps.world... started
... or one by one:
$ sudo ubic stop psgi-apps.world
Stopping psgi-apps.world... stopped
When someone is performing some global action, multiservices are grouped in indented trees:
$ sudo ubic status
ubic-ping running (pid 18641)
sleeping-daemon running (pid 8056)
psgi-apps
psgi-apps.hello running
psgi-apps.world off
And of course you can create several levels of nesting too if you need them.
There are two multiservice classes in core Ubic distribution.
First, there is a Ubic::Multiservice::Simple class, which can be used for cases when you just want to group several services together:
# content of /etc/ubic/service/my
use Ubic::Multiservice::Simple;
...
return Ubic::Multiservice::Simple->new({
fcgi_app => $fcgi_app_service,
memcached => $memcached_service,
});
It's also handy for generating several identical workers:
# content of /etc/ubic/service/ten-workers
use Ubic::Multiservice::Simple;
use Ubic::Service::SimpleDaemon;
return Ubic::Multiservice::Simple->new({
map {(
"worker$_" => Ubic::Service::SimpleDaemon->new({
bin => "worker.pl",
})
)} (1..10)
});
Second, there is a Ubic::Multiservice::Dir. Nobody usually uses it directly, but it is constructed for every subdir in /etc/ubic/service/, so that /etc/ubic/service/a/b file is mapped to "a.b" service.
So, summing it up, multiservices are useful for:
- grouping services together for easier management;
- dynamically generating services from external sources;
- running a large number of identical worker processes.
They really simplify some complex tasks and make ubic as unobtrusive and extensible as it is possible.
PS: PSGI example requires Ubic=1.22 and Ubic::Service::Plack=1.12. I made multiservice API slightly simpler while writing this post, and fixed one bug in Ubic::Service::Plack. Both of these modules should be available on CPAN soon.
PPS: Contacts:
our irc channel is #ubic at irc.perl.org;
source code is at http://github.com/berekuk/ubic;
mailing list is at http://groups.google.com/group/ubic-perl.
If you are using ubic or have any questions, let me know. It motivates me SO much :)
In my previous post I showed most trivial example of ubic service.
Now it's time to talk about more interesting stuff :)
Every ubic service is simply a perl object which implements start/stop/status methods.
It is very similar to /etc/init.d shell scripts, but since perl is a real programming language, you don't have to copy-paste tons of boilerplate code every time (Dave Rolsky, if you're reading this - your silki init script is good, but "apache2-backend" line in comments means it was copy-pasted too; is there anyone who can actually write proper init script by hands?).
So, instead of copy-pasting, you can reuse any existing Ubic::Service::* module, or implement your own if it's necessary.
For example, there is the Ubic-Service-Plack distribution on CPAN which you can use to run any PSGI applications.
Quoting Ubic::Service::Plack synopsis:
use Ubic::Service::Plack;
return Ubic::Service::Plack->new({
server => "FCGI",
server_args => { listen => "/tmp/app.sock",
nproc => 5 },
app => "/var/www/app.psgi",
app_name => 'app',
status => sub { ... },
port => 4444,
ubic_log => '/var/log/app/ubic.log',
stdout => '/var/log/app/stdout.log',
stderr => '/var/log/app/stderr.log',
user => "ppb",
});
Notice the 'status' coderef. It is optional, but if provided, it can be used as custom status check:
...
status => sub {
use IO::Socket::UNIX;
use IO::Select;
use Ubic::Result qw(result);
my $socket = IO::Socket::UNIX->new(
Peer => $self->{socket},
Timeout => 5,
) or return 'broken';
$socket->send(pack("C*", 1,9,0,0 ,0,0,8,0 , 0,0,0,0, 0,0,0,0));
my $content = "";
my $select = IO::Select->new($socket);
unless ($select->can_read(5)) {
return result('broken', 'socket timeout');
}
$socket->recv($content, 8);
my @bytes = unpack("C8", $content);
if ($bytes[1] == 10) {
return 'running';
}
else {
return 'broken';
}
},
...
This check will be called several times with increasing time intervals until status will become 'running'. Time intervals and number of checks until ubic will give up are configurable too (actually, they are implemented in Ubic::Service::Common class, which can be base class for most ubic services unless you want to do something really complex).
Now, if you have tons of PSGI applications which you run as fastcgi servers, you can implement Ubic::Service::Plack::FCGI on top of Ubic::Service::Plack.
Or maybe you are running tons of HTTP servers which all implement http://.../ping http check.
Or you may wish to restart your service on some external conditions, like Steven Haryanto wants to.
All these things can be implemented using custom status checks and then encapsulated via ineritance or delegation.
Of course, 'start', 'stop' and 'reload' methods can be replaced as well.
One last bit of information (it should have been in first post but i forgot to mention it): any ubic service can be turned into SysV init script.
Simply put this line in /etc/init.d/ and you are done:
# cat >/etc/init.d/any-ubic-service
#!/usr/bin/perl
use Ubic::Run;
Next time I'll talk about multiservices and custom commands.
Also, there are Ubic::Service::Memcached and Ubic::Service::Lighttpd I still have to upload.
Don't forget to join our mailing list if you are interested in ubic and service managing stuff!
I had a hidden agenda when I asked on this blog about how do you run your daemons.
There is a tool I recently opensourced. It can be compared to classic SysV init script system or to daemontools, runit or upstart, but it already is better and more flexible at least in some of its properties (otherwise I wouldn't bother to implement it :) ).
It is called Ubic and it is a toolkit for describing daemons in pure perl.
Ubic service is simply a perl object of some class inherited from base Ubic::Service class.
Ubic loads service descriptions from /etc/ubic/service/ directory.
To create your first service, you have to write something like this:
# cat >/etc/ubic/service/test
use Ubic::Service::SimpleDaemon;
return Ubic::Service::SimpleDaemon->new({ bin => "sleep 1000" });
"ubic status" command can show you status of all or some services on host:
# ubic status
test off
ubic-ping running
# ubic status test
test off
Now, as you probably already guessed, to start service you simply have to say "ubic start":
# ubic start test
Starting test... started
# ubic status
test running
What are you getting for free with this simple example?
# wget -q -O - 'http://localhost:12345/status/service/test'
ok
By this point in my story:
- init.d people should already think that ubic is cool ;)
- daemontools/runit users will probably require more persuading, so I'm going to continue tomorrow.
PS: Installation note: please read "installation" section in README.md file before installing.
PPS: There is a mailing list, don't hesitate to join and ask your questions.
PPPS: This is my first opensource module which i'm trying to attract attention to, so any meta-advices from guys with tons of popular CPAN distributions about what i'm doing right/wrong or how else can I persuade anyone to try ubic are very welcome too :)
How do you start and daemonize your programs (PSGI apps, fastcgi apps, gearmand, several memcached instances, custom handcoded daemons)?
Do you write custom init scripts?
Use daemontools or upstart?
Do you simply invoke plackup in terminal? :)
Maybe you spawn your fcgi apps directly from your webserver?
Do you need watchdogs? Do you actually write them?
Ubic is a toolkit for describing services on Linux in perl. It's been implemented in-house at Yandex as a better way to run daemons than writing ugly SysV-style /etc/init.d scripts, and i've put it on github and CPAN recently.
I also gave a lightning talk about it at YAPC::Russia::2010 in Kiev several days ago, it should explain most of its traits:
So... any feedback? :)
PS: Several Ubic::Service::* modules are not in public domain yet - i'll have to do some cleanup first, hopefully in next week or two.
I wrote Ubic. I worked at Yandex for many years, and now i'm building my own startup questhub.io (formerly PlayPerl). I'm also working on Flux, streaming data processing framework.
CPAN ID: MMCLERIC.
