#! /usr/bin/env perl
use strict;
use warnings;
use Benchmark 'cmpthese';
use File::Slurp 'read_file';
my $filename = shift or die "No argument given";
my $count = shift || 10;
cmpthese($count, {
'Unix' => sub { open my $fh, '<:unix', $filename or die "Couldn't open $filename: $!"; read $fh, my $buffer, -s $fh or die "Couldn't read $filename: $!" },
'Slurp' => sub { read_file($filename, buffer_ref => \my $buffer, binmode => ':raw') },
});
For large files, it's just as fast as File::Slurp is:
Rate Slurp Unix
Slurp 2.28/s -- -0%
Unix 2.29/s 0% --
For small files, it's actually significantly faster:
Rate Slurp Unix
Slurp 51020/s -- -66%
Unix 151515/s 197% --
So why use File::Slurp, when a two-liner will actually perform better?
]]>A short week later I flew to Germany, for the Perl Reunification Summit in Perl. Like Schwern I arrived a day earlier than most, so I had a calm start of the meetup. It was mostly a gathering of familiar to me faces, though a significant number I hadn't really spoken to before, specially the Perl 6 guys, -Ofun attracts awesome people. I spent most of the PRS talking to people, and doing a little coding (both related and unrelated). It was a very enlightening meetup.
Lastly, there was YAPC::EU. Despite the sometimes unbearable heat, it was awesome. At some points it seemed a bit less organized than my previous YAPCs, but that may also be me noticing more of what's going on. I spent most of my time in the hallway track, which extended into the pub track, and I spent enough time discussing (and occasionally ranting) that it's a miracle that I still have voice left. In between I found enough time to attend some talks, interestingly I attended most of them on the day I gave one myself. After doing threads last year I could only top it with signals this year. I'll have a challenge to come up with a crazier, I think I'll have to look in a vastly different direction (I have ideas already). After a full week of conferencing, I was relieved to be going home though.
So in all I met Mark Keating in 3 different places in 2 weeks time, I'd almost accuse him of stalking me!
]]>I think we've all seen code like this example from perlipc:
my $ALARM_EXCEPTION = "alarm clock restart";
eval {
local $SIG{ALRM} = sub { die $ALARM_EXCEPTION };
alarm 10;
flock(FH, 2) || die "cannot flock: $!";
alarm 0;
};
alarm 0;
if ($@ && $@ !~ quotemeta($ALARM_EXCEPTION)) { die }
Here, signals are used to put a time limit on some action. However sometimes this doesn't work as wanted. In particular, some C libraries used in XS modules don't honor the deferred signaling resulting in it being ignored until the C function has finished, which is unlikely to be what you want.
Therefore, people resort to unsafe signals
use Sys::SigAction qw( set_sig_handler );
my $ALARM_EXCEPTION = "alarm clock restart";
my $h;
eval {
$h = set_sig_handler('ALRM', sub { die $ALARM_EXCEPTION }, { });
alarm 10;
flock $fh, 2 or die "cannot flock: $!";
alarm 0;
};
alarm 0;
$SIG{ALRM} = $h;
if ($@ && $@ !~ quotemeta($ALARM_EXCEPTION)) { die }
This works as expected, mostly, but there is a serious problem with doing this; serious enough to have an explicit and specific high severity advise against it in CERT's secure coding guide (and it also happens to violate most other secure coding advises regarding signaling).
Signal handlers (or at least the real, unsafe ones) have a highly restricted set of operations they can safely to perform, doing anything that's not allowed means risking segfaults and data loss. This is why we needed "safe" signaling in the first place. By longjumping out of the unsafe/real signal hander (which is what die does), those restrictions are continued into the rest of the program. That means that anything from that point on can (and at some point probably will) cause segfaults and other bugs.
Ouch!
That's the harsh part, sometimes there isn't any easy way out. If a piece of C code doesn't have it's own timeout support, there may be no alternative. The real solution is to write blocking/computationally intensive software in such a way that it can handle this more graciously, for example by using an event loop, but often one has to deal with the tools one has.
So, I'm not saying everyone is wrong for using unsafe signal timeouts, but you should be aware of and accept the risks that come with it.
]]>I want to fix his only module on CPAN (POSIX::RT::MQ), but his public email address no longer exists. If I can not reach him I will ask the PAUSE admins for permission to take over this module.
]]>Let's be honest, many (most) prominent new features of 5.10 are failures:
given/when is even worse as it's almost impossible to predict if it will use smartmatching or not.$_? Mostly a new source of bugs, and the _ prototype is merely a hack to work around lexical $_ issues.A lot of others are not flawed, but are so uncommon that I haven't seen them being used in any code. UNITCHECK, stacked filetests, no VERSION, the list goes on…
In the end, there are only two new features of 5.10 that I end up using all the time: say and defined-or. These two features have one thing in common: they are small and simple features that make daily programming easier. Likewise my favorite new feature in 5.14 is the /r modifier on s/// and tr///. I don't know how we managed to do without that, it makes so much code so much simpler. I want more of those features.
On the other hand, there's another group of new features that is just as important, but not nearly as visible. It's the under-the-hood or right on top of it. Few people know how the $^H and %^H changed in 5.10, but if you're writing pragmas you'll appreciate them. These are expert features that few people will use, but those few use them to write the modules on CPAN that everyone else uses. These features are just as important, if not more so: they lay the foundation for progress by enabling (competitive) evolution on CPAN.
It is important for this progress to happen not in core but on CPAN. Because if modules screw up they can be discarded and we can try to come up with something better (which you can't with the core). Because modules easily allow an allow an ecosystem of TIMTOWTDI. But most of all because the Perl community is awesome at creating modules.
Few people outside the echo chamber follow know Steven's work on a perl MOP, but that may become the most important development in Perl OO since the arrival of his Moose (also written by him). Likewise few people are using Zefram's awesome keywords API, but that is what will allow us to do Devel::Declare kind of stuff in a sane way, and may one day open up doors to macros. Few people know of the Unicode improvements by Karl Williamson and others that make Perl hands down the best language for Unicode processing.
We can't always know end-user's requirement in advance. That's why big core features should be open ended. Maybe that doesn't make for spectacular perldeltas, but it does lead to a better end result.
]]>$< = 1000;
$> = 1000;
$> = 1000;
$< = 1000;
$< = $> = 1000
$> = $< = 1000
when @array can have 5 rather different meanings depending on the value of $_, the two most important being «each element in @$_ matches the corresponding element in @array» and «$_ matches any of @array». This is madness. I believe there's a lot of value in being able to explicitly differentiate when list(@array) and when any(@array).
Hence Smart::Match. It's a module that does exactly that and more. Not only does it allow you to disambiguate smartmatches, it also allows you to use powerful new matches by using higher order matchers. For example it doesn't just allow you to match against string_length(4), but also string_length(even) or string_length(range(13,42)). This makes smartmatching a lot easier and saner than doing it raw.
The core idea of Module::Build, that of a module builder written in Perl itself is a really good one in my opinion. Module::Build paved the way to do that, but this idea is not tied to Module::Build. End users and CPAN clients don't really care what's behind that Build.PL command, as long as it behaves according to the interface that was defined by Module::Build. That interface is currently in the process of being specified. The first experiment is Module::Build::Tiny, a proof-of-concept that does the simplest thing possible for pure Perl modules.
Since 5.10.1 the toolchain supports configure_requires. This means that the end user no longer has to have the install tool installed before installing the module, the toolchain will take care of this. This means that now the module building is just another program for most purposes.
Dependencies
Module::Build does not have a proper dependency graph. This decision is in many ways at the core of it's design. Writing a proper dependency graph has a serious upfront costs and makes a lot of simple things more complicated, however it also makes a lot of complicated things more simple. It also makes it much more simple to debug various issues, if only because you can serialize a graph.
Abstraction
ExtUtils::MakeMaker uses make, which does have a dependency graph. However there is no convenient way to manipulate it. It boils down to either appending some make commands to the string containing the Makefile or even manipulating it using regexps. This makes it way too difficult to customize the build process. CPAN authors should be able to specify what they want done, not how.
Extendability
Like I said in my previous post, Dist::Zilla showed how easy it can be to extend a program. My imaginary module builder should be just as extendable. However that doesn't mean it shouldn't have the right defaults. The default configuration that handles modules that don't do anything special (which is probably 95% of them right now) should be as simple as:
[@Basic]
The Build.PL wouldn't need to contain any state. It would probably be as simple as:
use Dist::Build;
Build_PL(@ARGV);
More simply isn't necessary. All data is either in the the meta file, configuration file or is generated by plugins.
The extensions could be anything, though I would obviously expect most to affect the build stage like editing or serializing the graph (perhaps even to a makefile), but it could also be in configuration (e.g. meta information), testing and installing.
The vast majority of modules, probably something like 95%, don't need anything special. They are just some Perl files, maybe with some XS: their build process is entirely standard. That means that for 95% of all modules out there it doesn't really matter what you use, as long as it doesn't break. Module::Build, Module::Install and ExtUtils::MakeMaker all work fine. Boring, but boring is a good thing in a toolchain. Module::Build is a frequently used option, but lets be honest, it hasn't exactly eclipsed ExtUtils::MakeMaker and Module::Install. In fact, the latter seems to be the usual choice among the "cool modules" (Moose, Catalyst, etc…).
The other 5% is where things gets interesting. There are basically two approaches to them:
Obviously, the second is to be preferred whenever possible. That scenario should have been a homerun for Module::Build. Reality is different, there are hardly any extensions for it on CPAN. In fact Module::Install seems to have more of them. What the hell went wrong?
Module::Build is very extendable, but those extensions are not reusable. IMO there are two design decisions here to blame:
Extending Module::Build is done by subclassing it. This works fine for simple examples, but has a major problem: it only works with well single inheritance. Because of that you can not combine Module::Build::PDL with Module::Build::WithXSpp for example. This greatly limits the usefulness of such extensions. This problem should be obvious to anyone who's familiar with roles. When you compare it with how easy it is to extend Dist::Zilla for example, the problem of this design becomes glaring obvious.
The problem mentioned above is aggravated by the fact that almost all behavior is mashed together in one class. Granted, it delegates a few platform specific behaviors to platform specific subclasses, but almost all other behavior is implemented in Module::Build::Base. It has more than 5600 lines and 342 methods (269 public, 73 protected) including 84 accessors. It's massive. You can't naively add roles to that and expect it to work out.
Well, that's the topic of my next post… ;-)
]]>I had already written POSIX::RT::SharedMem before, but even though POSIX IPC is a great idea (for being remarkably sane to use) it isn't widely implemented (Linux, Solaris and recent versions of OS X support it out of the box, FreeBSD does too but you have to enable it explicitly). Therefor I ended up writing SysV::SharedMem, which does pretty much the same thing, accessing shared memory as a string, but implemented using a different backend. It's largely derived from File::Map, unlike POSIX::RT::SharedMem that delegates most of the work to File::Map. Writing it has made my conviction that SysV IPC is an incredably crappy API even stronger, but fortunately that's largely hidden from the end-user.
I wrote a number of building blocks, mostly small and simple ones, though one ended up being a bit larger and more complicated than initially planned.
They cover a number of different things, but there is a line through them: they are all the pieces needed to write a modern, fast and efficient event loop for Linux. There are many event-loops written in C that use these functions, but in Perl they were either unavailable(thread specific signal masks, special filehandles) or had modules that were unsatisfying to me (epoll). I'm not sure that event-loop needs to be written, but I find experimenting with it highly educational.
Other than that, an increasing amount of my time has gone to maintainance of existing modules. I'm barely past 20 and it already feel bogged by them, I can't imagine what it must be like to have 200 modules on CPAN.
]]>
I loathe the Python community’s snobbish and hostile attitude to newcomers and outsiders. Maybe I’m biased by having to counter FUD in people who have been exposed to the Python community, but I can’t call it anything else. I wouldn’t be the first to suggest that’s the main reason they never took over even when they were otherwise in a much better shape than we were. They threw bricks into their own windows and I am somewhat doubtful they’ll get a second chance.
PHP did the exact opposite, and behold: they’re more popular than sliced bread. That seemed to be a double edged knife though: it seems to attract more people who have no idea how little they know than any other language, resulting in the PHP community being completely dominated by amateurs. Don’t get me wrong - I don’t dislike newbies, in fact I have more friends in this category than any of the others I’m discussing here - but there has to be a certain ratio of capable versus incapable programmers for a community to remain healthy. Perl once was in that same position and we’re still recovering from it a decade later. Despite its current popularity I don’t envy the position they’re in. Maybe they’ll have their own renaissance ten years from now.
I’m hearing some early signs Ruby (or Rails in particular) is going to get the newbie carrot next. Maybe I’m being cynical here, but if I were a member of the Ruby community I’d worry about how to handle them. Growth is nice but how do you keep it sustainable? That may prove to be an interesting challenge for them.
Meanwhile, I’m comfortable and warm in the Perl community. It’s healthier than most and that’s more important than being the coolest or the most popular kid on the block.
]]>Readonly hasn’t been updated in 6 years. It has collected quite a list of bugs. Not all of them can be fixed, but most that can be are either trivial or already have patches. Fact is it hasn’t been done and I have my doubts on whether it will be done.
Readonly has a number of design issues. It uses ties for its implementation. This is not only slow, but also rather buggy. There are lots of subtle corner cases to ties, and Readonly has probably run into a lot of them, though I suspect there may be more waiting to be discovered. Most of its bugs are related to this. There are some more issues but they are relatively minor compared to this. Your variable shouldn’t have unexpected behavior simply because it’s readonly, that’s madness.
The sad part is that ties aren’t really necessary here. Perl 5 has always supported real readonly variables, and though this functionality wasn’t exposed for Perl code until 5.8. It makes a lot more sense to use the real thing. The optional Readonly::XS tries to do that for scalars, but in the most common use case of Readonly (using sub Readonly on a scalar) it isn’t actually used (this is probably a bug).
So I’ve written my own module, Const::Fast. It does not use ties or XS. It has one function const, that simply makes a scalar, array or hash recursively readonly. Without the speed penalty of ties and without the bizarre class of bugs that they bring.
Scalar uvar magic isn’t actually evil, it’s just fairly useless. It’s a way to add get and set hooks to scalars. It may have been a useful interface back when it was introduced in 1993, but with the introduction of custom magic vtables in 5.8 it stopped particularly useful. The new interface is more powerful in many ways and in typical cases it that takes less code to use. There’s absolutely no reason why anyone should still be using scalar uvar magic unless you have to maintain compatibility with 5.6.
I blogged before about what hash uvar magic is and how to use it. Hash uvar magic is an abomination. It is the single most horrible interface in the whole of the perl API and that is an achievement. p5p must have been asleep when they allowed it in.
You may wonder, why is it so evil? It abuses the relatively sane interfaces of scalar uvar magic and uses for some completely different purposes. Parameters and even properties of the parameters get completely new functions. It’s a case of fitting a square peg in a round hole.
The callback doesn’t even receive it’s most important parameter (the key value) as an argument! You have to use mg_find to extract it out of one of the arguments (the hash). Equally bad, there’s absolutely no way to add data to the magic (other than adding more magic) because all possible fields are already in use: uvar itself uses mg_ptr and mg_len. The hash magic clobbers both mg_obj and uf_index.
The saddest thing is that all of this wasn’t necessary. It should have been easy to define some new kind of magic that passes it’s data in a more sane way quite easily. Maybe it’s not too late yet and the current API can be deprecated in favor of something sane (it’s not like people are using it a lot so far).
]]>Since 5.10, perl supports uvar magic on hashes, though I called it magic hash key transformations because that gives at least tries to describe what it does. This was implemented to make fieldhashes possible. It’s an interface that does only one thing: when a hash element is accessed in any way the callback is called. The callback can’t change the semantics of the operation (unless it dies), all it can do is change the value of the key looked up. So it can transform $hash{A} into $hash{B}.
Nothing more, nothing less.
AFAIK, there are only 4 modules on CPAN that use it
Why aren’t more modules using this? Probably because it’s fairly limited in what useful things it can do. I’m sure there are more uses of it, but I can’t think of any (that’s not true, for my previous post I was considering writing Acme::Keys::Rot13). It’s an interface with a limited purpose, though it can be useful sometimes.
In my next post, more about the darker side uvar magic.
]]>