Threads in Perl, Erlang style
Adam Kennedy's recent post on threads in Padre reminded me to post about an experiment of mine. Last year I learned some Erlang. I really liked their model of multi-threading: many threads that share no data at all and communicate through message queues. A lot of other things where really annoying though, specially their crappy support for strings and lack of libraries in general. I kept thinking I want perl with these threads, so I started implementing it. And thus threads::lite was born.
The main difference between threads::lite and threads.pm is that t::l starts an entirely new interpreter instead of cloning the existing one. If you've loaded a lot of modules, that can be significantly quicker and leaner than cloning. As an optimization, it supports cloning itself right after module loading, so you can quickly start a large number of identical threads. Threads can be monitored, so that on thread death their send an exit code to their listeners.
Every thread has it's own message queue. Every thread can send messages to any thread whose thread-id it knows. Any kind of data that can be serialized using Storable can be send, including coderefs.
Receiving messages is done on basis of pattern matching (based on smart matching). This can range from a simple null-pattern (matches anything, so returns the front message on the queue) to complex tables of patterns.
I've started building some high level abstractions on top of it, most notably a implementation of a parallel map and grep. I'd like to do a map-reduce, but that's for a later stage.
This is very much an experimental module. I'm still not sure perl is really suitable for true erlang style threading: I suspect perl interpreters are to heavy to have a 100000 of them in one process, but it would be interesting to try…
Cool.
I've got nothing else to say right now, but I'm just commenting to let you know I've read your post with much interest. So you know you are not alone in a Perl desert... :)
So, like Coro / Coro::MP: channels, message passing, separate perl interpreters, except with ithreads?
Exactly how does the matching work? I took a look at the docs & source, but it wasn't clear to me (disclaimer: I have far too much blood in my caffeine stream at the moment, so this may not be a problem with the docs....)
@James: no, not like Coro at all, Coro shares one interpreter between many cooperative threads, threads::lite is using one interpreter per preemptive thread.
It does share a lot of idea's with Coro::MP/AnyEvent::MP, specially the message passing and channels. They don't have pattern matching though.
@dstar: Pattern matching is currently rather underdocumented (hey, it's an experimental module), specially if you're not familiar with Erlang. It's hard to describe it in words, though I think examples work pretty well. t/10-basics.t and lib/threads/lite/list.pm contain a few examples. I'll add a paragraph about it to the documentation in the next version.
This is really interesting -- just a couple months ago, I started thinking about the ease and power offered by software built on the message-passing model, and began working on my own design for a Perl 5 messaging layer. Since then, I've found that three or four other people are also diving into the space, everybody coming at it from a slightly different point of view.
I think this is pretty awesome all around, and it makes me want to work really fast, but I'm making myself actually write out a protocol spec and try to handle as many issues as I can on paper before I start coding, shooting for a balance between BDUF and making decisions on the fly.
Looks familiar:
Thread::Apartment
Tho it was based on regular Perl ithreads, it used msg passing. But relying on Thread::Queue and esp Storable turns out to be very expensive. Some related stuff:
DBIx::Threaded HTTP::Daemon::Threaded
fwiw, I tested using a regular new thread and starting a new interpreter...the performance benefit vs. starting a threadpool upfront in ithreads wasn't that great. The bigger issue is passing the msgs around wo/ having a major lock on the universe (which I was trying to solve w/ Thread::Sociable, until I tired of pushing a big rock up Perl's hill).
@renodino: Thread::Apartment is a much more high-level module. It may be an interesting candidate to convert from threads.pm to threads::lite, as I think it's the kind of use case where threads::lite would be much more suitable.
Storable has its costs, but it gives back a lot of features. I'm considering to use a simple
pack("(N/a)*")instead for messages that only contain simple scalars, those should be the majority anyway.IMO, perl is not really ready or even feasible to be used for erlang style threadining. I believe it can be but it's far from ready. We tested this out a few times with my development team and had errors come up like clockwork with complex tables.
Bonnie Smith COO/Director FXP http://www.forexpulse.com