Perl dropped and Go adopted due to concurrency issues in baconbird
A little part of my Perl soul died the other day when I read a tweet by Mark Jason Dominus:
Perl didn't work for this guy, here's why. It was a really good article, should be required reading for #perl folks: synflood.at/blog/index.php...
When Andreas Krennmair announced a new Twitter client called baconbird in late 2010 he sounded very enthusiastic about Perl, saying
The code itself is about 850 SLOCs of Perl code, the user interface is based on STFL, the Twitter backend uses Net::Twitter, and all the OO glue code in between uses the Moose object system. All in all it took me less than 2 weeks of my free time to develop the first version of it.
I was excited to read this post because I really like his newsbeuter feed reader (I've chatted once or twice with Andreas over IRC about it and helped package it), and it was interesting to see him choose Perl over C++ for this new project. I pulled in the baconbird dependencies and started using it right away.
Some months later Andreas wrote a blog post entitled "I don't want my programming languages to be hip" in which he praised the Perl community, saying
I just want a quiet, hype-free, pragmatic, down-to-earth knowledgable community. . . Exactly the things that I learned to despise about the culture revolving around Ruby and Rails are something that I haven't experienced in the Perl community at all so far. Libraries, framework, documentation and people had more time to mature, probably, while still being down-to-earth. I can expect good documentation, functioning software, friendly people and most importantly accurate words in announcements and discussions.
Everything seemed peachy. We had a happy Perl programmer on our hands.
Last week, however, within a year of baconbird's release, Andreas blogged about the problems he has been having with Perl:
But first things first: my first poor choice was using Perl. Sure, Perl provided me with all the tools to come to meaningful results really quickly, through the vast amount of ready-to-use modules in the CPAN archive. And that's what's really great about it. What's not so great about Perl are the ideas of concurrency. You basically have to resort to event-based, async-I/O programming, which is not only painful by itself, but also extremely painful to integrate with my widget set of choice, STFL. And threads in Perl... don't do it, that's what everyone says. That meant that I couldn't do any of the Twitter communication asynchronously in the background, but had to do it in my primary thread of execution, essentially make it part of the subsequent calls of the input loop, if I wanted to retain sanity and a certain level of productivity. And that made baconbird really slow to use, and when Twitter's API was offline, baconbird practically hung. That made baconbird just a PITA to use, and I had to change something about it.
That post is entitled "Rebooting the baconbird project" but his new Twitter client is a complete rewrite and has a new name: gockel. It's written in Go, which has a particular focus on concurrency, as I understand it.
So far I've been disappointed by the comments on Andreas's post. I'm not skilled enough in Perl or concurrency to say anything about how to solve his problems, but maybe someone else will be inspired to post an nice comment with some suggestions. As I quoted above, he isn't using an event-based model (so I take it AnyEvent is a non-starter) and he's really into STFL (newsbeuter uses this well), which seems to complicate matters. Maybe Go is simply a better tool in his situation. Maybe threads::tbb could help. I don't know.
I blog about Perl.
It looks like STFL is his real issue. I cannot compile it on Windows, so i cannot confirm, but if i read things right it takes over process execution until user input happens when run is called on it.
Of course this is absolutely terrible when you're trying to use a C library in Perl and can only be worked around with threads, which utterly suck in Perl.
And to provide a counterpoint, i once wrote a Perl script to render a 3d world in opengl, using data read from another process in realtime and continually updated; and it was fluid. As such Perl's certainly capable of decent amounts of concurrency.
Source: https://github.com/akrennmair/baconbird
After having chatted with him on his blog entry the situation is now more clear:
- he hates event-based programming, which would be exactly the correct solution for this sort of thing and which perl makes easy
- the next closest thing is using threads, which perl makes hard
So he opted for Go because it allows him to put in zero effort, throw up a bunch of threads and have the network stuff be in its own thread. It still operates blocking in there, which means that if one request gets jammed, all other requests jam too; but that's "good enough", since the UI thread is separate and remains responsive instead of locking up.
What is the technical reasons why perl threads are so bad? We've been saying they are bad for more than a decade now. over two different architectures. But what is the key technical reason why we should avoid perl threads that specifically makes it worse than threads in some other language?
In most cases you don't need threading. See nginx or ligthttpd... But since Java and Windows, a lot of people think only by this way... It's a pity, event programing is elegant and very efficient in terms of memory AND speed.
Most of the time, your application does nothing, just waiting for data from the kernel. No need to do threading to handle that.
It seems that the author don't like event programing. OK. Perhaps he never tried AnyEvent...
I'll write a blog entry on that over the weekend.
Perl’s so-called threads aren’t threads. They are an emulation of
fork, a clumsy one at that.I'm curious to see what you write. If you don't mind, this Reddit commenter had some concerns about the event programming approach (The discussion was regarding Node.js and the commenter's event programming experience is in C++). If it turns out to be relevant to what you end up writing, you might address these concerns.
http://www.reddit.com/r/programming/comments/ky6uc/nodejs_is_cancer/#c2obr7p
RE: STFL, is just a curses user and Curses has similar problems with blocking. Wouldn't a POE::Wheel::Curses-like workaround of checking if STDIN has data to be read and running $stfl->run(0) if it does, $stfl->run(-1) otherwise, work?
This old post of mine was just referenced in a new post by Mario Roy called "STFL Terminal UI - Concurrency Demonstrations" at http://www.perlmonks.org/?node_id=1200923 . Enjoy!