In addition, you may join us before or after the meeting for small-group video chats at: https://gather.town/HAWHVP3b6eywWEV3/chicago.pm
In case you are not familiar with gather.town, after you join the conversation, you will have a small avatar on a 2d map and can walk around. When you are close to somebody or a group of people, you can video chat with them over video. Perlmongers is supposed to be a social gathering, and we are experimenting with this venue to see if it'll make that possible!
We are still finalizing our speaker for this meeting. If you are interested in giving a talk at Chicago.pm in August or a future month, reply here or find me on irc.perl.org/#chicago.pm
]]>On a closely related note, if/when it comes time to run a heavy calculation and I want to speed things up, is there a way to turn assertions into no-ops?
]]>I am a bit torn to learn about your work here. Performance is a big deal to me, so much so that I've started my own project to solve it. It seems like there is quite a bit of prior work in this arena and I worry you haven't reached out to other project leaders. Your work seems most closely aligned with rperl, but also addresses an issue that I've been trying to fix with C::Blocks. Could you perhaps comment how your project is different from both of those projects?
]]>I expect that these kinds of computing devices are going to get more important over the next five to ten years. Specifically, I think that physics departments will begin using them for hybrid electronics/programming courses. At my institution we have an electronics course on the books, and I have toyed with the idea of using an Arduino for some of the digital material. If my department could agree on adding programming to our curriculum, then we might move to RPi (or Beaglebones) as they would support something besides C++.
It remains to be seen just how the popularity of the individual boards will evolve. Apart from the BBBs, every board that I encountered was built around a system-on-a-chip taken or derived from cell-phone CPUs. RPi has certainly done well getting brand recognition, but when the it comes to spending money, I wonder how many folks will go for cheaper (C.H.I.P.) or more CPU-powerful (Intel Edison) alternatives. Does most of the RPi code you're producing work with other single-board systems?
]]>Yesterday I used C::Blocks to play around with Perl's C API and mess with keywords. Today I will focus on a couple of neat C tricks that can help clean up the C-end of your library API.
]]> One of the chief goals of C::Blocks is to make it easy to share C code with others. It's one thing to share code, but quite another to share useful code, code with an API that is easy to use and easy to read. Part of the reason this is difficult is because good API design is inherently difficult, but part of the reason is that C only provides one way to pass arguments to functions. Fortunately, there are two preprocessor tricks we can use to alleviate this problem.The first trick is quite common in Perl's own C API: hiding arguments. Consider the simple sv_setiv
, which sets a scalar to the given integer value. For example:
my $var;
cblock {
sv_setiv($var, 5);
}
print "\$var = $var\n"; # prints '$var = 5'
Notice that I use this like a function with two arguments, but it's actually a macro that wraps the real function called Perl_sv_setiv
:
#define sv_setiv(a, b) Perl_sv_setiv(aTHX_ a, b)
This makes for a good example for two reasons. First, if you are working with a Perl compiled with MULTIPLICITY
(the case if your Perl is threaded), then aTHX
is the current Perl interpreter. You are always going to call this function with the current Perl interpreter, so sv_setiv
can save us some keystrokes by always adding that as our first argument to the real function. Second, if your Perl is not compiled with MULTIPLICITY
, then the Perl interpreter is maintained as a global variable, in which case aTHX
is a macro that gets replaced with spaces! In other words, the actual signature for Perl_sv_setiv
depends on how you compiled Perl! The complete details for how this works are discussed in perlguts, but you can use sv_setiv
in blissful ignorance of these details because the macro wrapper takes care of everything for us. This is good API design.
A simpler but arguably more useful example would be a situation where you want to give useful feedback when your function croaks. In that case you can write a function that takes the current line number and file name, in addition to the rest of its arguments. The macro wrapper would supply those automatically, like so:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
clex {
#define munge_input(input) munge_input_(__LINE__, __FILE__, input)
int munge_input_ (int line, char * file, char * input) {
/* make sure input is non-null */
if (input == 0) {
croak("In %s line %d, munge_input called with null input\n",
file, line);
}
printf("munge_input not yet implemented...\n");
}
}
cblock {
munge_input("to be munged");
munge_input(0);
}
When I run that, I get this output:
$ perl test.pl
munge_input not yet implemented...
In test.pl line 21, munge_input called with null input
Just like Perl's Carp
provides useful die
ing behavior, you can provide useful exceptions by wrapping function calls like this. If you have many public functions and a handful of private ones, your public functions can call the private functions explicitly, sending the values of line
and file
it received when it was called. This way, somebody using your C API will get useful error reporting regardless of how you internally implement your code.
The Tiny C Compiler is a nearly compliant C99 compiler, which means we can use macro tricks to emulate named arguments. This uses compound literals and variadic macros, both C99 features. The Tiny C Compiler's compound literal handling wasn't quite right for this task until it was fixed very recently (i.e. within the last month). The update is currently only available through a developer's release of Alien::TinyCCx
, but should be in the next point release. Take this as a sign of things to come.
First, I'd like to show you how this works. Suppose I wanted to have a GUI command that draws a label somewhere on a canvas. It could have many optional arguments, like padding width, border width, border color, etc. Using a single struct, macro, and function declaration, I can call my function like this:
draw_label(canvas, 1, 5, "X marks the spot");
draw_label(canvas, .y = 5, .x = 1,
.label = "X marks the spot");
draw_label(canvas, 1, 5,
.label = "X marks the spot", .x = 3);
This almost looks like Perl's key/value pair calling convention, except that the keys are prefaced with a period. So, how does this work?
The original idea breaks a normal function declaration into three pieces. First, define a preprocessor macro with the actual name that you are going to use in your public API. For example:
#define draw_label(c, ...) draw_label_(c, (struct draw_label_args_){ __VA_ARGS__ })
Notice that this is a variadic macro, and it simply dumps the contents of the ...
into a so-called compound literal declaration. Before we can understand that, we need to look at the struct layout:
struct draw_label_args_ {
const float x;
const float y;
const char * label;
};
Now look carefully at the different ways we called the function. Judging from those examples, the following would all be valid ways of initializing a draw_label_args_
struct:
struct draw_label_args_ my_args
= { 1, 5, "X marks the spot" };
struct draw_label_args_ my_args
= { .y = 5, .x = 1, .label = "X marks the spot" };
struct draw_label_args_ my_args
= { 1, 5, .label = "X marks the spot", .x = 3 };
The first assignment is a classic struct assignment, the sort of thing you'd see in C89 code. (In C99, any field that is not mentioned is initialized to zero.) In the second case, all labels are mentioned explicitly, and can be out of order! In the third case, we see that we mix sequential positional values and named fields. In fact, named fields override previous identical named fields, or even positional values!
What sort of function do we need? We need a function that accepts a canvas as its first argument and the struct as its second:
static void draw_label_(Canvas * c, struct draw_label_args_ args) {
if (args.label == 0) args.label = "(none)";
// default x,y of 0 is OK
...
}
Within the body of the function, the label is accessed as args.label
. Likewise the x- and y-positions are accessible via args.x
and args.y
. If any of those were not specified, they will default to zero, a situation that is fine and suitable for x and y, but not for the label. This can be easily detected and fixed for the label.
We can take the previous example one step further. The default value for any uninitialized member is zero. A position of zero is valid for x and y, but what if we want to explicitly indicate an unspecified position? Or, what if we want to provide a different default?
We can specify defaults that are different from zero, but there are multiple ways to do it, with varying trade-offs. The key is to remember that later statements in a struct initialization override previous ones. So, one simple approach for specifying nonzero defaults is to revise the macro to something more like this:
#define draw_label(c, ...) \
draw_label_(c, \
(struct draw_label_args_){ \
.x = 100, \
.y = 100, \
__VA_ARGS__ \
})
(Note that I split the macro definition across multiple lines by ending the line with a backslash.) When this is later used by somebody calling the function, they can override the defaults:
draw_label(canvas, .x = 50);
Unfortunately, the naive implementation here breaks positional arguments, i.e. the following would no longer work:
draw_label(canvas, 1, 5, "X marks the spot");
After naming a field, you can continue to list values in succession. Since we ended our defaults at .y
, the next valid unnamed field would be the label, not x
. To fix this properly, we need to add an additional item to the beginning of our arg struct, something that the user will not need to override. In this case, by moving the canvas into the arg struct, we can get the behavior we want:
struct draw_label_args_ {
Canvas * c_,
const float x;
const float y;
const char * label;
};
#define draw_label(c, ...) \
draw_label_(c, \
(struct draw_label_args_){ \
.x = 100, \
.y = 100, \
.label = "(none)", \
.c_ = c, \
__VA_ARGS__ \
})
static void draw_label_(struct draw_label_args_ args) {
// canvas is args.c_...
}
Perhaps more interestingly, we can add a couple of extra members to our argument struct for the calling line and file. This would let us produce an error message naming the calling context. I've mostly been illustrating with snippets of code, so here I'll provide a full working example with C::Blocks:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
clex {
#define salutations(...) salutations_( \
(struct salutations_args_){ \
.message = "Hello", \
.calling_line = __LINE__, \
.calling_file = __FILE__, \
__VA_ARGS__ })
struct salutations_args_
{
int calling_line;
char * calling_file;
char * name;
char * message;
int is_exclamation;
};
void salutations_(struct salutations_args_ args)
{
/* Croak if no name given. */
if (!args.name) {
croak("salutations called without specifying a name at %s:%d\n",
args.calling_file, args.calling_line);
}
printf("%s %s%s\n", args.message, args.name,
args.is_exclamation ? "!" : ".");
}
}
cblock {
salutations("David");
salutations("David", .is_exclamation = 1);
salutations("David", "Merry Christmas");
salutations("David", "Merry Christmas", 1);
salutations("David", "Merry Christmas", .is_exclamation = 1);
salutations(.message = "Merry Christmas",
.name = "David");
// runtime error:
salutations(.message = "Merry Christmas");
}
When run, I get this output:
$ perl test.pl
Hello David.
Hello David!
Merry Christmas David.
Merry Christmas David!
Merry Christmas David!
Merry Christmas David.
salutations called without specifying a name at test.pl:44
When I began learning about the distinctions between C and C++, I remember that named arguments and argument defaults were two big niceties in C++ that were missing in C. I did not discover this trick until much later. On the one hand, you need to write a lot of boiler-plate for named arguments with defaults in C. On the other hand, the code shown in the cblock
looks really good. In some respects it is even more flexible than key/value pairs in Perl functions because you are not required to specify keys for your arguments if you don't want to. And of course, we're not talking about strict C: we're talking about C::Blocks, which is capable of automating this kind of code with interpolation blocks or source filters.
Before wrapping things up I want to point out two important aspects of these sorts of tricks. Using macros to reduce the argument count is an old and reliable technique. Named arguments and defaults is much newer. I am not sure if these are universally implemented by modern compilers, specifically Microsoft's Visual C. An eventual goal of C::Blocks is to provide an optimizing compiler back-end, in addition to the TCC back-end. If that ever becomes a reality, then named arguments may be a tripping point. (It sorta looks like MS may have gotten compound literals working, but I haven't found a definitive answer on the subject.) But that is probably some ways off, and for now I'd say if you like them, you should use them!
Also, I am new to these sorts of function machinations. I think they're great, but please do not take my code as examples of best practice. They are merely my musings, with the hope that folks will find them illuminating and exciting.
Today I showed how to use macros in C::Blocks to create C APIs that are versatile and easy to use. Named arguments and defaults rely on newly added behavior in tcc, but it should roll out onto the CPAN soon.
]]>I must admit that some of yesterday's results have me a bit depressed. I've put a lot of work into this library, and I am really surprised and worried about the performance cliffs I illustrated yesterday. Today, though, we're going to have some fun.
]]> With C::Blocks it is easy and fairly cheap to get access to the entire Perl API. One aspect of the C API with no built-in Perl hook are the keyword parsing hooks. There are modules for handling this, including Keyword::API (to "original"), Keyword::Simple, and most recently Keyword::Declare, a Damian module that is itself built on Keyword::Simple. Damian mentions that "you either have to write them in XS (shiver!) or use" the other modules I mentioned. Well, today we will shiver! Grab a coat and some hot chocolate because I'm going to use C::Blocks to write a keyword hook in C, just to show how simple it is to grab a random part of the Perl C API and play with it.There isn't a lot of documentation on writing keyword plugins with C. The best you'll get is the discussion of PL_keyword_plugin
from perlapi, which is not bad, but doesn't exactly hold your hand, either. There are two basic things we need to do: (1) install our keyword plugin, and (2) make it do something interesting.
To get started, here is a skeleton C::Blocks script that installs a nearly useless keyword plugin:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
# The simplest keyword hook. It just prints what it
# is given and hands off to the next handler.
clex {
/* This holds on to the previously installed hook,
* which we'll eventually call. */
int (*next_keyword_plugin)(pTHX_ char *, STRLEN, OP **);
/* Here's our dumb hook */
int my_keyword_plugin(pTHX_ char *keyword_ptr, STRLEN keyword_len, OP **op_ptr) {
printf(" -> given keyword [%*s]\n", keyword_len, keyword_ptr);
return next_keyword_plugin(aTHX_ keyword_ptr, keyword_len, op_ptr);
}
}
# Install the keyword hook
BEGIN {
cblock {
printf("Hooking parser\n");
next_keyword_plugin = PL_keyword_plugin;
PL_keyword_plugin = my_keyword_plugin;
}
print "Hook installed\n";
}
# From here onward, all keyword-looking things will be printed
print "Hello!\n";
use constant N => 10;
for (my $i = 0; $i < N; $i++) {
printf "Step $i of %d\n", N;
}
BEGIN {
print "When is this seen?\n";
print "And when is THIS seen?\n";
}
# invalid function:
keyword1();
print "All done!\n";
The keyword hook is a C function, so we have to declare it in a clex
block. We also declare next_keyword_plugin
, which holds the function that used to be at PL_keyword_plugin
. Since Larry has already written a keyword handler that does something useful, we'll just call this at the end of our useless keyword handler.
Just declaring a keyword handler doesn't do anything. I have to install it, which is what the cblock
inside the BEGIN
block does for me. All it has to do is backup PL_keyword_plugin
and assign our new handler to it. That's it! All the code that follows is just there to illustrate what gets passed to the keyword handler.
It may be useless, but I find the output to be fascinating:
$ perl test.pl
Hooking parser
Hook installed
-> given keyword [print]
-> given keyword [use]
-> given keyword [for]
-> given keyword [my]
-> given keyword [N]
-> given keyword [printf]
-> given keyword [N]
-> given keyword [BEGIN]
-> given keyword [print]
-> given keyword [print]
When is this seen?
And when is THIS seen?
-> given keyword [keyword1]
-> given keyword [print]
Hello!
Step 0 of 10
Step 1 of 10
Step 2 of 10
Step 3 of 10
Step 4 of 10
Step 5 of 10
Step 6 of 10
Step 7 of 10
Step 8 of 10
Step 9 of 10
Undefined subroutine &main::keyword1 called at test.pl line 42.
The first thing that caught my attention was that they keyword hook was passed some important built-ins: print
, my
, use
, and even BEGIN
! It is conventional to think of the keyword API as a means for adding keywords, but apparently you can use it to modify the behavior of existing ones! Could we completely disable these? I'll try that shortly.
The next thing that caught my attention was the way the hook interacted with BEGIN
blocks. The second BEGIN
block makes it clear that the parser fully parses the contents of the BEGIN
block before running it; this is not a line-by-line parse/execute mode. You probably knew that already, but I feel it's nice to get explicit evidence. This then explains why the hook didn't trigger on the print
statement in the first BEGIN
block, the one that installed the hook in the first place: that print
had already been parsed. This behavior is also quite different from a source filter, which only hooks when you invoke it with a use
statement. Interestingly, this means you can potentially use BEGIN
blocks to interact with your keyword hook.
True to its name, the keyword hook does not get called on variables, operators, literals, or comments. This is a Good Thing, and is one of the big annoyances of source filters. Smart::Comments are cool, but implementing a source filter properly is hard. Keyword hooks only hook on keywords, but they hook every single time. This means you can keep the scope of your changes minimal.
Finally, we can see that the keywords get fully processed before the script runs. This is not a big surprise since keywords are really a parser hook.
I mentioned earlier that the hook got called with fundamental keywords. Can we do anything with them? According to what I've seen, Perl has already "consumed" they keyword, so there's no way to completely remove it. You can, however, nullify it. We can do this by inserting a null op and returning a parse result (rather than returning whatever next_keyword_plugin
might have returned).
To see this in action, replace my_keyword_plugin
with this:
int my_keyword_plugin(pTHX_ char *keyword_ptr, STRLEN keyword_len, OP **op_ptr) {
printf(" -> given keyword [%*s]\n", keyword_len, keyword_ptr);
/* disable BEGIN blocks */
if (keyword_len == 5 && strEQ(keyword_ptr, "BEGIN")) {
*op_ptr = newOP(OP_NULL, 0);
return KEYWORD_PLUGIN_STMT;
}
/* everything else as normal */
return next_keyword_plugin(aTHX_ keyword_ptr, keyword_len, op_ptr);
}
To disable a keyword of any sort, we have to do two things. First we need to return an op. Perl has already parsed the keyword and knows something is there, so you are obligated to give it an op of some sort. In this case, since we want to disable BEGIN
blocks, we return a null op. Second we need to indicate that the this keyword should be parsed as a complete statement, rather than as an expression. Here I picked a statement because expressions need a terminating semicolon while statements do not.
If I put this function in place of my previous keyword plugin, the output verifies that I can turn BEGIN
into a no-op:
$ perl test.pl
Hooking parser
...
-> given keyword [N]
-> given keyword [BEGIN]
-> given keyword [print]
-> given keyword [print]
-> given keyword [keyword1]
-> given keyword [print]
Hello!
Step 0 of 10
...
Step 9 of 10
When is this seen?
And when is THIS seen?
Undefined subroutine &main::keyword1 called at test.pl line 51.
You can do many things with a C-based keyword hook. The Perl-level keyword hooks all focus on text replacement: modifying the contents of the source code buffer before it gets sent to the Perl parser. This makes them rather like specialized source filters. You could do the same thing with a C-based keyword hook together with the lexer interface. If that's your aim, you would be better served using the Perl wrappers. Perl really is the best tool at your disposal for string manipulation.
However, with the C interface, you could build your own sequence of ops. Using the lexer interface, you could actually extract some of the contents of the ensuing source code, parse it, build your own op sequence, and let Perl pick back up where you leave off. For example, you could write a keyword hook called jsblock
which parses everything between enclosed braces and builds a set of ops corresponding to the meaning of the javascript code. This is completely different from how a source filter works. Here you have an enormous wealth of information about the current state of the parser, including things like variable types. You have access to the entire state of the parser, and can add new lexical PAD entries, for example. In contrast to a source filter, a C-based keyword hook is incredibly powerful and rich.
This is the heart of how C::Blocks itself works. When it sees a recognized keyword, it pulls out all the code until it finds the matching closing brace. If it is dealing with a cblock
, it adds an op that executes the code within the block. Otherwise it simply adds a null op and stores the compiled code and symbol table in a safe place for later retrieval. During the extraction process, when it encounters a sigiled variable name, it can query the Perl parser for any type information for that variable name. This kind of rich information is availabe to keyword hooks, but not to source filters.
I would like to show you one of the useful parts of Perl's C API that can enrich a potential C-based keyword. This is the process for retrieving a lexical variable's type. That is, if we have a variable declared as
my Class::Foo $bar;
how would a keyword figure out that $bar
has a type of Class::Foo
? First it needs to find the location of $bar
in the current lexical pad:
int var_offset = (int)pad_findmy_pv(varname_with_sigil, 0);
if (var_offset != NOT_IN_PAD) {
...
}
Second it needs to get the stash associated with the type. Replace those ...
with
HV * stash = PAD_COMPNAME_TYPE(var_offset);
if (stash) {
...
}
You can think of the stash
as the hash underlying the Perl package. If we knew the name of the package, we would have gotten this stash with something like:
HV * stash = gv_stashpvs("Package::Name", 0);
We're going in the reverse direction: PAD_COMPNAME_TYPE
gives us the stash directly, and we want its name. Alternatively, you might want to retrieve a method from this package, which is what C::Blocks does with type information. In that case you would use gv_fetchmeth_autoload
or a similar function. Since we want to just get the name of the package, we would replace the ...
in the previous if
statement with something involving HvENAME(stash)
, like:
printf("Variable %s is of type %s\n", varname_with_sigil, HvENAME(stash));
To make this more useful, though, I'll actually have my keyword, typeof
, insert an OP that returns the type as a Perl scalar. This way, the typeof
keyword can be used in expressions. Putting this all together, I have this doozy:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
use C::Blocks::Types qw(uint);
# Create the "typeof" keyword
clex {
/* This holds on to the previously installed hook,
* which we'll eventually call. */
int (*next_keyword_plugin)(pTHX_ char *, STRLEN, OP **);
/* returns true if character is part of a valid variable name
* (I've omitted a couple of corner cases for brefity) */
int _is_id_cont (char to_check) {
if('_' == to_check || ('0' <= to_check && to_check <= '9')
|| ('A' <= to_check && to_check <= 'Z')
|| ('a' <= to_check && to_check <= 'z')) return 1;
return 0;
}
/* isolates the variable name following the keyword. Returns the
* location of the character just past the varname. */
char * identify_varname_end (pTHX) {
/* clear out whitespace and prime the pump */
lex_read_space(0);
if (PL_parser->bufptr == PL_parser->bufend) lex_next_chunk(0);
/* make sure keyword is followed by a variable name */
char * curr = PL_parser->bufptr;
if (*curr != '$' && *curr != '@' && *curr != '%') {
croak("typeof expects variable");
}
/* find where the variable name ends */
while (1) {
curr++;
if (curr == PL_parser->bufend) {
int offset = curr - PL_parser->bufptr;
lex_next_chunk(LEX_KEEP_PREVIOUS);
curr = PL_parser->bufptr + offset;
}
if(!_is_id_cont(*curr)) return curr;
}
}
/* Here's our actual hook for "typeof" */
int my_keyword_plugin(pTHX_ char *keyword_ptr, STRLEN keyword_len, OP **op_ptr) {
if (keyword_len == 6 && strEQ(keyword_ptr, "typeof")) {
/* get end of varname and set it to null so we can use the
* buffer directly in pad_findmy_pv */
char * end = identify_varname_end(aTHX);
char backup = *end;
*end = '\0';
/* get the name and store it in a constant op */
SV * to_return = &PL_sv_undef; /* default to undef */
int var_offset = (int)pad_findmy_pv(PL_parser->bufptr, 0);
if (var_offset != NOT_IN_PAD) {
HV * stash = PAD_COMPNAME_TYPE(var_offset);
if (stash) {
char * name = HvENAME(stash);
if (name) {
to_return = newSVpv(name, 0);
}
}
}
*op_ptr = newSVOP(OP_CONST, 0, to_return);
/* restore the end character and discard up to it */
*end = backup;
lex_unstuff(end);
return KEYWORD_PLUGIN_EXPR;
}
/* everything else as normal */
return next_keyword_plugin(aTHX_ keyword_ptr, keyword_len, op_ptr);
}
}
# Install the keyword hook
BEGIN {
cblock {
next_keyword_plugin = PL_keyword_plugin;
PL_keyword_plugin = my_keyword_plugin;
}
}
# Use an explicit package name
my C::Blocks::Type::NV $fraction;
my uint $counter;
my $basic_var;
print "\$fraction is of type ", typeof $fraction, "\n";
print "\$counter is of type ", typeof $counter, "\n";
if (defined typeof $basic_var) {
print "\$basic_var is of type ", typeof $basic_var, "???\n";
}
else {
print "\$basic_var's type is not defined\n";
}
my C::Blocks @foo; # goofy!
print "Type of \@foo is ", typeof @foo, "\n";
# print "typeof junk is ", typeof junk, "\n";
This is enormous. Let me break it down into its chunks. my_keyword_plugin
looks for typeof
keywords. The variables that follow are then prompted for their type, which is stored in and ultimately returned by an op. _is_id_cont
is a utility function for identifying characters that can be part of a variable name. identify_varname_end
works with the lexer, first discarding any whitespace, then expanding the buffer until it contains the first character after the variable name.
The last 15 lines are where we get to see it in action. The result of this script is:
$ perl test.pl
$fraction is of type C::Blocks::Type::NV
$counter is of type C::Blocks::Type::uint
$basic_var's type is not defined
Type of @foo is C::Blocks
If you uncomment the last line, you would instead see this:
$ perl test.pl
typeof expects variable at test.pl line 105.
So there you have it. This keyword even dies when you use it improperly!
As you can see, string manipulation in C is extremely verbose, which is why Damian shivered at the thought of it. I'm not going to disagree: it's annoying. But, if we put this effort into writing our keyword parser, we are rewarded at the end by being able to (1) get at a variable's type info and (2) build a constant op that returns this type info, making it accessible at runtime. This might seem useless: you would think you would know your variable's type, just like you know your current package's name and should not need __PACKAGE__
. However, this is not actually always true, as illustrated with typeof $counter
. Here we use a short name, uint
, which actually represents the longer C::Blocks::Type::uint
. This could also be useful if you ever wanted to use a pattern in which you declare a variable, then create a new copy of it:
my Class::Name $var;
$var = (typeof $var)->new;
To the best of my knowledge, it's not possible to get a variable's type information any other way, and this just made it accessible at runtime. I don't know about you, but I think that's pretty neat!
Today I showed how to use C::Blocks to play around with Perl's C API, in particular, how to write your own keyword hook. I focused on the keyword API because you can do some pretty crazy and neat things with it. However, you could use this to explore any facet of Perl's C API that amuses you, all within the context of a humble Perl script. In this way, C::Blocks makes it possible for mere mortals to begin playing around with Perl's C API.
]]>-run
option, so saying something like tcc -run my-code.c
will immediately compile and run your code. This means tcc sorta turns C into a scripting language all by itself. But if we're talking about Perl, the first distribution to work with tcc was C::TCC. This was little more than a glorified wrapper around tcc's own run function, except that you could assemble your C code into a string rather than storing it in a file. The first wrappers to focus on serious interactions between Perl and C were my C::TinyCompiler and Steffen Mueller's XS::TCC. Steffen's module focused on making it easy to write C code that could be called from Perl; my distribution sorta attempted that, but mostly tried to provide a framework for writing C libraries that could be used from Perl. Graham Ollis' FFI::TinyCC is a more recent edition and leverages the amazing capabilities of FFI::Platypus. His library is similar in spirit to Steffen's, but creates its xsubs using a foreign-function interface instead of the usual XS mechanism. All of these present tradeoffs of different sorts. If you find yourself needing some of these capabilities, which library should you use?
Just like with my last post on benchmarks, this post has been delayed even more than usual. The reason for the delay is a growing body of data on spectacular slow-downs for incremental changes to functions created using C::Blocks. Much of my spare time over the last few days has focused on chasing down the root cause of these slow-downs and discussing them on the tcc mailing list. As far as I can tell, the slowdowns that I can produce arise not from tcc itself, but from unfortunate processor cache evictions that arise from interactions between tcc-compiled code and the Perl interpreter. As far as I can tell, XS modules do not suffer from these sorts of problems, but more testing is needed to be sure. Regardless, now is a great time to mention that C::Blocks is still in PRE-beta. I'll discuss what that means in a future treat, but for now know that performance optimizations are not a key goal yet. I want these issues sorted out before v1.0. Now, on to the comparisons and discussion.
I do not recommend C::TCC or C::TinyCompiler for anything. The first of these does not provide any facilities for passing Perl data to C (or back), and the latter of these (mine) is not well designed for its purpose. Anything you can do with C::TinyCompiler can be done better with one of the others.
If you simply need to call C functions from Perl from a pre-compiled C library, and if you can properly form your arguments, you should just use FFI::Platypus. It is the most mature option for doing this sort of thing. The only reason for using C::Blocks in this situation is philosophical, i.e. if you'd rather think of you C code in terms of blocks rather than function calls.
If you want to share C constructs across multiple compiler contexts and even multiple modules, then C::Blocks is really the way to go. I know I'm a bit biased here, but C::Blocks' support for sharing struct, function, enum, typedef, and preprocessor declarations is unmatched. This sort of sharing was one of the chief design goals of the distribution, and was based on things I learned from doing C::TinyCompiler poorly. FFI::TinyCC and XS::TCCdo even less than C::TinyCompiler in this realm (though they do what they do much better than C::TinyCompiler). If you're building a large codebase in which you want to freely mix C and Perl, you should use C::Blocks.
If you just want to include in your Perl script C code that exchanges data with Perl, then read on. Inline::C is an option, but if you want to do this with jit then you should consider XS::TCC, FFI::TinyCC, and C::Blocks. Let's drill down a bit.
The simplest meaningful example I could come up with was one we've already seen: the random number generator. We saw this on Day 4 as an example of how to declare functions using clex
. On Day 5 I compared the performance of the C::Blocks and Inline::C implementations. On Day 8 I demonstrated how easy it is to share code across modules using cshare
. We'll use it this time to compare how it is implemented using these various approaches.
I've written a number of implementations. The complete collection can be found as a gist on github. They are all fairly similar in layout so I will only explain the key aspects. All of them except for rand-clex.pl
create a Perl function KISS_rand
that have a C-implemented random number generator. Of course, rand-perl.pl
implements its random number generator with pure Perl.
I provide three implementations using C::Blocks. The first example uses csub
, differing from the others because it has to explicitly push the return value onto the stack. The second example embeds a cblock
in a regular Perl sub
. It uses an idiom in which a variable is declared, then modified by the cblock
, and finally returned at the end. The third implementation uses clex
to declare the function and cblock
s to execute it. This is awkward because a cblock
is not considered an expression, and so cannot be combined with a postfix foreach loop. None of these implementations are very fluid, evidence that this specific problem was not a major design target.
The implementation for rand-ffi.pl
and rand-xs-tcc.pl
are nearly identical, differing only in the library that provides tcc_inline
. The Inline::C
implementation is mostly the same. All three of these extract the function signature and wrap the function call with essentially no work needed by the programmer, making this particular task much easier than C::Blocks.
The only remaining implementation is rand-perl.pl
, which implements the generator in pure Perl. This one is almost the same as the others except that all integer operations must be truncated at 32 bits by &
ing the results with 0xffffffff
. If I don't do this, the random number generator produces numbers that quickly blow up. Apparently my perl interpreter was built with 64-bit integers, and Perl quickly switches to a floating-point representation of a number when it encounters integer overflow.
An important quality metric is ease of writing. The pure-Perl and csub
implementations were the hardest to write; the former because it needed debugging to truncate at 32-bits, and the latter because I had to look up how to return values on the Perl stack. The others are all pretty easy to write, with the cblock
example losing an edge due to the boilerplate inherent in the declare-modify-return idiom. Ultimately, if I were given this algorithm from a C programmer and asked to implement a Perl-visible one, the only implementation that was tricky was the pure-Perl one. All the others were basically of equal difficulty, I would say.
Another important quality metric is speed of execution. To check the speed, I timed these on my computer using the time
bash built-in, like so:
$ time perl rand-inline.pl 10000000
Random number #10000000 is 292847080
real 0m0.751s
user 0m0.744s
sys 0m0.004s
The times below are rough averages of the "real" times as reported above, but here listed in milliseconds, and are sorted according to execution time for ten million iterations, the longest test.
N = 1,000 10,000 100,000 1,000,000 10,000,000 LOC
clex 60 60 65 95 400 24
inline 55 60 65 125 730 23
xs-tcc 100 100 100 200 1000 23
csub 60 60 70 250 1900 24
perl 10 30 80 490 4500 18
cblock 60 65 120 550 5500 26
clex** 60 60 120 650 5650 24
ffi 50 60 130 800 8000 23
Before doing anything else, I want to discuss the two entries for clex
. If you look at the gist, you'll see that rand-clex.pl
has a line with a trivial operation commented out. The first set of results in the above table are for that situation. Uncomment that line and on my machine you get the results shown for clex**
. In light of these huge swings, it's hard to draw deep conclusions from these timings. All this having been said, here is what I conclude from these:
The first take-away is that Perl really is fast! Since the pure-Perl implementation did not need to load any modules, it easily beat out all the other options for as many as 10,000 iterations. The speed improvements for compiled code only become evident after enough iterations have caught up with the cost of loading the module or compiling the code in the first place. In other words, don't over-estimate the speed gains of a C implementation if you're only doing a few lines of math.
The next take-away is that if you don't plan on calling this kind of function more than 100,000 times, the implementation does not matter. They all take roughly the same amount of time, so if you are already using one of these tools and you find yourself in this situation, you should keep using your tool.
The third take-away is that XS::TCC is hindered a bit by its approach. XS::TCC is the only distribution that instructs tcc
to #include
the Perl header files like perl.h
. C::Blocks uses a cached symbol table produced from perl.h
whereas FFI::TinyCC packs its arguments in a way that does not rely on perl.h
at all. Other tests not shown here indicate that XS::TCC suffers dramatically on repeated string compilation, which can only be explained by the direct loading and parsing of perl.h
and friends.
The fourth caveat-laden take-away is that a cblock
calling a function defined in a clex
can be faster than a gcc-compiled xsub! "Wait," you say, "what about what you said on Day 5, about gcc-compiled xsubs being faster than C::Blocks???" It's important to emphasize that the for-loop for all of today's examples happen in Perl code, not C code. This means that each and every call to KISS_rand
in the Inline::C code has to be invoked via the xsub calling mechanism; the C::Blocks clex
code, on the other hand, calls it using just one or two ops in the op-tree. The cost of the argument stack setup and teardown for an xsub ends up dominating the computation cost, which is no surprise in light of the brevity of the function being called.
Just about everything else I might want to conclude would seem unjustified in light of the effects of cache conflicts. I suspect that results could vary significantly if these benchmarks were run on other machines. It appears that FFI::TinyCC suffers from the same sort of cache problems as clex**
, though it fares worse, for some reason. As such, it seems that the best-case performance for FFI::TinyCC is probably comparable to, or maybe just a little bit slower than, C::Blocks.
So, what are we to make of all of this? Here are my take-aways:
clex
and cblock
. But, be sure to benchmark the heck out of it to be sure. Even better: implement your code using XS; add a keyword hook that injects an op that calls your function when it sees your keyword.All of this having been said, the case under consideration is a little constrained. I'm assuming you never want to share the C code implementing your function with other modules. Most Perl programmers have learned to live with this as a reality of using XS, but I would like to challenge that. C::Blocks makes it really easy to share your C code across modules.
Also, we haven't considered the case of multiple return values, which necessitates manipulating Perl's stack. As far as I can tell, that case knocks out FFI::TinyCC since it doesn't have any stack manipulation facilities. XS::TCC and Inline::C have those capabilities, but I'm not sure that they can efficiently handle the Perl stack. They do not pass the Perl interpreter into the function where you do your work, so they would need to retrieve it from thread-local storage. I'm told that's a costly operation. At any rate, the additional code for this sort of thing levels the aesthetic playing field, so to speak.
Today I compared implementations of the random number generator using a number of different tcc-based libraries. If you find yourself needing to implement a single function to run quickly, there is no obvious choice: you'll have to resort to benchmarks of the various libraries to find out what works best. Writing these benchmarks have been illuminating for me; I hope that today's treat has been illuminating for you as well. I also hope that you consider opening up your C code and putting it on par with your Perl code as something worth sharing.
]]>use strict;
use warnings;
use threads;
for my $tid (1 .. 10) {
async sub {
sleep 1;
print "Hello from thread $tid\n";
threads->exit;
};
}
print "Back in Perl\n";
$_->join foreach threads->list;
print "All done!\n";
When run we get:
$ perl test.pl
Back in Perl
Hello from thread 1
Hello from thread 2
Hello from thread 3
Hello from thread 4
Hello from thread 5
Hello from thread 6
Hello from thread 7
Hello from thread 8
Hello from thread 9
Hello from thread 10
All done!
Basically, this kicks off 10 threads that sleep for a second and print a message. The messages print in order, but there's no guarantee that they would do this if the content of their execution times varied. The simplest way to test C::Blocks is to replace the print
statement in the async
block with a bit of C code. For example:
use strict;
use warnings;
use C::Blocks;
use threads;
for my $tid (1 .. 10) {
async sub {
sleep 1;
cblock {
printf("Hello from thread %d\n", SvIV($tid));
}
threads->exit;
};
}
print "Back in Perl\n";
$_->join foreach threads->list;
print "All done!\n";
which produces...
$ perl test.pl
Back in Perl
Hello from thread 1
Hello from thread 2
Hello from thread 3
Hello from thread 4
Hello from thread 5
Hello from thread 6
Hello from thread 7
Hello from thread 8
Hello from thread 9
Hello from thread 10
All done!
As you can see, the cblock
is able to compile without a problem, and the code within it runs in each thread without a problem. Everything Just Works. This is something of a surprise since the Tiny C Compiler used to compile C::Blocks code with is not thread-safe.
Why does this work? Recall that the code in cblock
, clex
, and all the other blocks get compiled during script compile time. This is almost always a single-threaded process: multiple threads tend to be spawned at runtime, not compile time. This means that the Tiny C Compiler is used in a single-threaded fashion, producing object code and stuffing function pointers in the op tree. Then when threads get spawned, the function pointer addresses for cblock
s get copied with the op-tree, so each thread calls the exact same function in each thread. Threads in C have access to the memory in other threads, so calling a function whose object code is stored in another thread is not a problem.
When might things go wrong? The major means for seriously breaking things is to use C::Blocks in a string eval that is evaluated in many threads simultaneously. Tiny C Compiler's lack of thread safety will probably lead to catastrophic behavior. Even if the compiler does not croak, it'll probably produce invalid machine code. Presumably this could be protected by guarding use of the TCC with a parallel memory lock, but this has not yet been implemented.
Using C::Blocks we can bend the rules of data sharing a bit. In this rather silly bit of code, each thread sets one of the characters in a string that is not shared across threads at the Perl level:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(Int char_array);
use threads;
# Store a pointer to the SV's PV slot
clex { char * message_workspace; }
my $message = "Hello, world!!!";
cblock {
message_workspace = SvPVX($message);
}
print "Message is '$message'\n";
# Overwrite the contents of $message; each thread handles a letter
my @to_replace = ( qw(H a p p y), ' ', qw(h o l i d a y s !) );
for my Int $tid (1 .. length($message)) {
async sub {
my char_array $new_letter = $to_replace[$tid-1];
cblock {
message_workspace[$tid-1] = *$new_letter;
}
threads->exit;
};
}
$_->join foreach threads->list;
print "Message ended up as '$message'\n";
This works as expected, but the careful observer may be a bit troubled at this approach for two reasons. First, the thread that owns $message
must not alter the scalar using regular Perl operations like assignment. If the main thread altered $message
, it could invalidate the memory slot pointed to by message_workspace
, and lead to invalid memory access in the cblock
code. Second, the thread that owns $message
must not exit before the other working threads. In this case the primary thread outlives all the async
threads and this is not a problem, but in other situations it might be tempting to assign message_workspace
to a memory slot that will become invalid before all threads finish. This approach works only because I could ensure that the variable lives long enough for each thread to do its work, and ensure that the variable is not modified in a way that would change the allocated memory slot.
One way to side-step both of the problems just mentioned is to allocate special memory for the task. Allocated memory that is not tied to a Perl variable cannot be accidentally clobbered by an innocent bit of Perl code. Furthermore, Perl's C API provides functions to allocate and free memory that will survive even if the original thread terminates. To do that, use one of the saveshared*
family of functions and deallocate with the usual Safefree
:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(Int char_array);
use threads;
# Copy the contents of an SV to shared memory
clex { char * message_workspace; }
my $message = "Hello, world!!!";
cblock {
message_workspace = savesharedsvpv($message);
}
print "Message is '$message'\n";
# Overwrite the contents of the message; each thread handles a character
my @to_replace = ( qw(H a p p y), ' ', qw(h o l i d a y s !) );
for my Int $tid (1 .. length($message)) {
async sub {
my char_array $new_letter = $to_replace[$tid-1];
cblock {
message_workspace[$tid-1] = *$new_letter;
}
threads->exit;
};
}
$_->join foreach threads->list;
# Copy the updated message back to $message and free shared memory
cblock {
sv_setpv($message, message_workspace);
Safefree(message_workspace);
}
print "Message ended up as '$message'\n";
Compared to the previous script, this one merely adds copying the data into shared memory and back. This is unnecessary caution for this script since we can easily verify that the main thread does not modify $message
and since $message
outlives all of the working threads. However, this approach in a larger codebase would make it more robust against accidents.
Today I showed that C::Blocks works perfectly fine when working with multiple threads, so long as you avoid simultaneous string evals using C::Blocks code in multiple threads. Operating on shared memory also works fairly well, and judicious use of the saveshared*
family of functions can help avoid accidental segfaults or data corruption.
C::Blocks::Object::Magic
while writing a simple class that had APIs in both Perl and C. Today I dig into one of the keys of yesterday's example: writing a type that can be used with C::Blocks.
]]>
When Perl sees code like this:
my Type::Package $some_variable;
it makes a note that the variable name $some_variable
has a "type" with information available in Type::Package
. I must emphasize that this type information is associated with the variable name itself: nothing special is done to the underlying scalar. If Type::Package
set up a few fields with the fields
pragma, then Perl will check the (spelling of) keys of hash dereferencing at compile time. C::Blocks uses this type information in an orthogonal way: to produce custom code for marshalling your data between Perl and C.
To get an idea of how this works, consider the short script from yesterday which used the KISS library. If I use C::Blocks::Filter;
before the cblock
, then I end up with this script:
use strict;
use warnings;
use C::Blocks;
use KISS;
use C::Blocks::Filter;
my KISS $rng = KISS->new;
print "rng's first value is ", $rng->rand, "\n";
cblock {
printf("rng's second value is %u\n", KISS::rand($rng));
}
which produces output like the following when run (I have added whitespace for clarity):
$ perl test.pl
##################################################
void op_func(C_BLOCKS_THX_DECL) {
SV * SV__PERL_SCALAR_rng = (SV*)PAD_SV(1);
struct KISS__state * _PERL_SCALAR_rng
= xs_object_magic_get_struct_rv(aTHX_ SV__PERL_SCALAR_rng);
printf("rng's second value is %u\n", KISS__rand(_PERL_SCALAR_rng));
}
##################################################
rng's first value is 2079675107
rng's second value is 4185567647
The line containing the printf
demonstrates things I've discussed previously. KISS::rand
becomes KISS__rand
and $rng
becomes _PERL_SCALAR_rng
.
The interesting bit is the thing that comes before. I said on day 6 that C::Blocks detects when you use a sigiled variable in your cblock
and injects code to transform that to the SV*
(or AV*
or HV*
) underlying the variable. (It's worth pointing out that C::Blocks does not allow sigiled variables in clex
, cshare
, or csub
blocks because there is no way for it to know which PAD to work with. This only works with cblock
blocks.) If the variable is typed, and if the type's package contains c_blocks_init_cleanup
, C::Blocks will call that method to get the code to use for the transformation.
The C end of the KISS
library is built around a struct pointer. The one method, KISS::rand
, expects a pointer to a struct. Furthermore, you could directly read or set the state by accessing the x
, y
, z
, and c
members of the struct. It makes sense, then, that the type should produce code that would map $rng
to the pointer to the struct. This is what it does. First it gets the SV*
for $rng
from the current PAD
, but instead of putting it in _PERL_SCALAR_rng
as it would for untyped variables, it puts it in SV__PERL_SCALAR_rng
. The variable _PERL_SCALAR_rng
is used for the struct KISS__state
pointer, which is unpacked with xs_object_magic_get_struct_rv
. By the time we reach the printf
line, our Perl $rng
has been unpacked, and $rng
gets transformed into the pointer to the KISS struct, as expected.
Let's look again at the c_blocks_init_cleanup
code from KISS.pm
:
sub c_blocks_init_cleanup {
my ($package, $C_name, $sigil_type, $pad_offset) = @_;
my $init_code = "$sigil_type * SV_$C_name = ($sigil_type*)PAD_SV($pad_offset); "
. "struct KISS::state * $C_name = xs_object_magic_get_struct_rv(aTHX_ SV_$C_name); ";
return $init_code;
}
The method is called with four arguments: the package ("KISS
"), the gently mangled C variable name ("_PERL_SCALAR_rng
"), the sigil type ("SV
"), and the pad offset (1). It returns a single string with the initialization code, utilizing string interpolation throughout.
Here is the init/cleanup code for double arrays, from C::Blocks::Types
:
package C::Blocks::Type::double_array;
sub data_type { 'double' }
sub c_blocks_init_cleanup {
my ($package, $C_name, $sigil_type, $pad_offset) = @_;
my $data_type = $package->data_type;
my $init_code = join(";\n",
"$sigil_type * SV_$C_name = ($sigil_type*)PAD_SV($pad_offset)",
"STRLEN length_$C_name",
"$data_type * $C_name = ($data_type*)SvPVbyte(SV_$C_name, length_$C_name)",
"length_$C_name /= sizeof($data_type)",
'',
);
return $init_code;
}
Unlike my example from KISS
, this method is written in such a way that it can be used by other type packages, such as float_array
and char_array
. These packages simply inherit from this package and implement an alternative data_type
method. To see an example of this, try:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(char_array);
my char_array $string = "Hello!";
cblock {
printf("From C, %s\n", $string);
}
When run with -MC::Blocks::Filter
, I get
##################################################
void op_func(C_BLOCKS_THX_DECL) {SV * SV__PERL_SCALAR_string = (SV*)PAD_SV(1);
STRLEN length__PERL_SCALAR_string;
char * _PERL_SCALAR_string = (char*)SvPVbyte(SV__PERL_SCALAR_string, length__PERL_SCALAR_string);
length__PERL_SCALAR_string /= sizeof(char);
printf("From C, %s\n", _PERL_SCALAR_string);
}
##################################################
From C, Hello!
In this case, quite a bit gets unpacked when using this variable. The original SV*
is SV__PERL_SCALAR_string
, the character array is _PERL_SCALAR_string
, and the length is available as length__PERL_SCALAR_string
. In particular, these special variables can be utilized in our cblock
code like this:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(char_array);
my char_array $string = "Hello!";
cblock {
printf("The string '%s' is %d characters long\n", $string, length_$string);
}
Notice how length_$string
gives the length! For most string operations the length is not crucial because the string ends in a null character. This is not the case for the numerical types: the length is a crucial piece of information needed to process the full contents of the array:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(double_array);
my double_array $data = pack('d*', 1 .. 10);
cblock {
double sum = 0;
for (int i = 0; i < length_$data; i++) {
sum += $data[i];
}
printf("The sum is %f\n", sum);
}
which produces:
The sum is 55.000000
The idea that length_$variable
would resolve to a variable with useful information is an unplanned but very useful side-effect of how the code extractor works. For lack of a better name, I've taken to calling these extra bits of information "prefix macros" because the code extractor only properly resolves them when you add on letters prior to the variable name, not after it.
It turns out that the code generator expects either one or two return values from c_blocks_init_cleanup
. The first return value is always the initialization code; the optional second return argument is any cleanup code. This is useful for basic types, which have to call sv_setiv
or similar to ensure that any changes you've made are propagated back to the original SV*
. Everything we've seen up to this point have involved pointers to things. Modifying those things would lead to the desired side effects, so no cleanup was necessary.
Finally, there is one more trick worth knowing about type handling. Whenever C::Blocks sees a sigiled variable in a cblock
it will replace it with the gently mangled name, as we have seen. What if simply using a variable is insufficient? In that case you can resort to using macros. For example, when using C::Blocks::Types::Pointers
, you can take the address of a pointer to get something that works (caveats aside). Here is the relevant bit of code from day 7
my double_LL $head = 0;
my double_LLp $tail_p = 0;
cblock {
$tail_p = &$head;
}
If $head
resolved to a local variable, this would lead to an local address, which would become invalid as soon as we left the block. To get around that, C::Blocks::Types::Pointers
actually creates a pointer to the desired pointer type called POINTER_TO_$C_name
, pointing to the address of the underlying IV
slot in the SV*
. It then defines a C macro: #define $C_name (*POINTER_TO_$C_name)
. This means that whenever you see $head
in the cblock
, it is ultimately replaced with a pointer de-reference.
Today I explained how to create your own types with C::Blocks. When your library provides both a Perl and C interface, types make it possible to flow back and forth between Perl and C code and have your variables resolve to the "right" thing. This lets you concentrate on writing actionable code instead of extracting your data from a Perl SV*
.
C::Blocks::Object::Magic
.
]]>
My example code yesterday was heavy on C pointers, which will come as no surprise to anyone who has programmed in C. With C::Blocks::Types::Pointers
, managing these pointers was painless, even easy. The cblock
line $tail_p = &$head
is particularly smooth.
However, that line should also sound off alarm bells to anyone who has worked with both Perl and C. The idea of carrying around pointer values in Perl scalars is not the real problem (while it's not the safest option, XS programmers have been doing that for decades with the T_PTR
typemap). The problem is in taking the address of $head
. Where does $tail_p
actually point? It points to the address of the IV slot of $head
. Things could quickly go downhill if we cause $head
to upgrade its internal memory representation. This is easier to do in Perl than you might realize.
One way to change a variable's internal representation is to use it in a string context, such as printing it. This example shows exactly that:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types::Pointers
void_p => 'void*',
void_pp => 'void**';
my void_p $address = 0;
my void_pp $ref_to_address = 0;
cblock {
$ref_to_address = &$address;
printf("From C, after assignment, address of address is %p; ref_to_address is %p\n",
&$address, $ref_to_address);
}
print "From Perl, address is $address\n";
cblock {
printf("From C again, address of address is %p; ref_to_address is %p\n",
&$address, $ref_to_address);
}
An example of output for this on my machine is:
$ perl test.pl
From C, after assignment, address of address is 0x12fb8e0; ref_to_address is 0x12fb8e0
From Perl, address is 0
From C again, address of address is 0x12f8090; ref_to_address is 0x12fb8e0
The agreement in the first line shows that ref_to_address
has the correct value. By the third line they disagree. I should reiterate that the problem is not with pointers stored in Perl scalars: these are fine and their values persist correctly. The problem is when I try to use C::Blocks::Types::Pointers
to manage a pointer to a pointer, and then accidentally upgrade the SV*
holding the original pointer. My pointer-to-a-pointer will point to a newly invalid slot in memory that was just returned to the memory pool.
(Note: if I revise the Perl print
to be a printf
instead, it would not upgrade the underlying scalar. If you find yourself regularly using C::Blocks::Types::Pointers
, you should make a standard practice of using printf
instead of print
when printing pointer values.)
While there are many other ways to store pointers, the most elegant solution I've seen is XS::Object::Magic
. I liked it so much that I ported it to C::Blocks as C::Blocks::Object::Magic
. This approach uses Perl Magic (literally) to store pointers. Magic is a mechanism for overriding core behaviors of an individual scalar, array, or hash (such as assignment). It is orthogonal to the object system and does not rely on blessing. Attaching a bit of magic to a Perl variable requires a struct with applicable methods, and an optional pointer to additional information. XS::Object::Magic
(and therefore C::Blocks::Object::Magic
) store the pointer by adding magic with no methods (a struct filled with null pointers) and using the pointer slot associated with this null magic to store the desired pointer. Using this approach, the pointer is only accessible from C code, and pointers can be attached to a scalar, an array, or a hash. The last option is particularly nice since it means I can write a hashref-based object with C data safely tucked away.
The next three code snippets comprise KISS.pm
. It combines a number of concepts I've brought up thus far, so I've broken the code into chunks to illustrate each idea. I start with
# KISS.pm
package KISS;
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(uint);
use C::Blocks::Object::Magic;
# The KISS random number generator C-side implementation
cshare {
struct KISS::state {
unsigned int x, y, z, c;
};
/* force xs_object_magic_get_struct_rv to be included in this symbol
* table, so that imports of KISS get this symbol. */
void * KISS::ignore_me = &xs_object_magic_get_struct_rv;
unsigned int KISS::rand(struct KISS::state * s) {
unsigned long long t, a = 698769069ULL;
s->x = 69069*s->x+12345;
s->y ^= (s->y<<13); s->y ^= (s->y>>17); s->y ^= (s->y<<5);
t = a*s->z+s->c; s->c = (t>>32);
return s->x+s->y+(s->z=t);
}
}
Because this use
s C::Blocks and contains a cshare
block, this module will provide C code to the lexical contexts where it is used. I use double-colons in my struct and function names so to minimize the likelihood of name clashes with other libraries. Also notice the bit about KISS::ignore_me
. I have this line to force C::Blocks to copy the symbol xs_object_magic_get_struct_rv
into this symbol table. This ensures that any code that use
s this one will be able to call that function. I'll cover more about symbol table tricks like this in a later post.
# Also make it possible to use KISS as a cblock type
sub c_blocks_init_cleanup {
my ($package, $C_name, $sigil_type, $pad_offset) = @_;
my $init_code = "$sigil_type * SV_$C_name = ($sigil_type*)PAD_SV($pad_offset); "
. "struct KISS::state * $C_name = xs_object_magic_get_struct_rv(aTHX_ SV_$C_name); ";
return $init_code;
}
By implementing a function called c_blocks_init_cleanup
, KISS
can be used as a type for C::Blocks. This means that I can type my KISS $rng
, and this type conversion code will be used. In fact, all code written after this function can use the KISS
type, even code in the same module. Obviously there's a lot going on in this that is beyond the scope of this treat: I'll cover how to write a type library soon.
# Perl-side constructor. Build an empty hash and attach the
# rng state struct to it.
sub new {
my $class = shift;
my $self = bless {}, $class;
cblock {
struct KISS::state * state;
Newx(state, 1, struct KISS::state);
*state = (struct KISS::state){123456789, 362436000, 521288629, 7654321};
xs_object_magic_attach_struct(aTHX_ SvRV($self), state);
}
return $self;
}
sub DESTROY {
my KISS $self = shift;
cblock {
Safefree($self);
}
}
# Perl-side method for calling the rng
sub rand {
my KISS $self = shift;
my uint $to_return = 0;
cblock {
$to_return = KISS::rand($self);
}
return $to_return;
}
1;
Finally I get to the Perl code: the new
method builds an object including its hidden state struct, DESTROY
frees up the allocated memory, and rand
gets the next random number. The module use
s C::Blocks::Object::Magic
and illustrates how to use xs_object_magic_attach_struct
to attach the struct to the object. It also uses xs_object_magic_get_struct_rv
to get the struct, though you probably missed it because it's buried in the type definition.
And now I can write a script that uses this module:
use strict;
use warnings;
use C::Blocks;
use KISS;
my KISS $rng = KISS->new;
print "rng's first value is ", $rng->rand, "\n";
cblock {
printf("rng's second value is %u\n", KISS::rand($rng));
}
When run, that script prints:
$ perl test.pl
rng's first value is 2079675107
rng's second value is 4185567647
This is a short script, and on its surface it looks pretty simple: I create a new KISS
random number generator and I use it to produce two random numbers. The remarkable aspect of this script is that I use the same random number generator---even the same variable name $rand
---in both Perl and C code. The object underlying $rng
fluidly moves between the two contexts because the KISS
package provides type information. The Perl-side rand
method ultimately invokes KISS::rand
, which means that I can generate random numbers with my object in whichever context is more convenient. To accomplish this feat, I wrote a module that provides both a Perl and a C interface to a struct, but even the module was not terribly hard to write.
The short script above does not actually show off the utility of using C::Blocks::Object::Magic
. To see that, I need to utilize the fact that the object is a blessed hash:
use strict;
use warnings;
use C::Blocks;
use KISS;
my KISS $rng = KISS->new;
$rng->{name} = 'Gerry';
print "$rng->{name}'s first value is ", $rng->rand, "\n";
cblock {
printf("rng's second value is %u\n", KISS::rand($rng));
}
Running this produces the following output:
$ perl test.pl
Gerry's first value is 2079675107
rng's second value is 4185567647
If I decide while developing a library that I need certain information to be available from C then I add it to the struct, but if the information only needs to be available from Perl then I can simply store it in the hash. Having worked with Prima, which uses a different scheme to support hashref-based objects with a core C struct underneath, I have found this to be particularly useful for storing data used by custom handlers. Of course, subclassing would be a more systematic way to achieve the same goal, but either way the hashref is indispensable for storing the data.
A critique of this approach is that the module author must write distinct C and Perl methods (usually with one calling the other). This sort of code duplication will be cumbersome for anything but the smallest of projects. The proper solution to this problem is an object system, something that simultaneously builds both C and Perl methods from a common declaration. Such a system is not yet available. However, C::Blocks is up to the task and in the coming days I will provide a number of treats that go through the capabilities needed to write a proper object system.
Today I gave an example of a Perl class that provides both a C and a Perl interface. In particular, I showed how C::Blocks::Object::Magic
makes it easy to have a hashref-based object while safely storing a pointer to an underlying struct for the C-visible state. I glossed over how to write a C::Blocks type, an important detail that I will discuss soon. A proper object system for C::Blocks will require some way to implement inheritance in C. How is this accomplished? These details will be some of the forthcoming treats this Advent.
I will begin with the basic C code for this task. The next few chunks of code would go into a clex
or cshare
block. First, we would declare the linked list structure:
typedef struct double_LL double_LL;
struct double_LL {
double_LL * next;
double value;
};
Each element of our linked list will hold a pointer to the next element of the list (which is null if this is the last item in the list), and the actual value of interest. Memory for these will be obtained with Perl's Newx
, so we need to Safefree
the linked list when we're done. We could implement this using recursion, but I prefer to write it using a loop:
void LL_free(double_LL * curr) {
while(curr) {
double_LL * next = curr->next;
Safefree(curr);
curr = next;
}
}
Finally we get to the interesting part, or parts. The insert
function needs to take a node in the linked list and a value, and then insert this new value in a new node that follows the current one. It would be nice if this were generic enough that it could handle insertions not at the tail of the list. It would be even nicer if it could allocate a fresh node when passed the null pointer, thus doubling as the list initial allocator. Here is one not-so-good implementation:
double_LL * LL_insert(double_LL * curr, double new_value) {
double_LL * new_node;
Newx(new_node, 1, double_LL);
new_node->value = new_value;
if (curr == NULL) {
new_node->next = NULL;
return new_node;
}
new_node->next = curr->next;
curr->next = new_node;
return new_node;
}
This function is not ideal because it has to handle an edge condition. It seems like it would almost be better to have a separate constructor, rather than trying to have the insertion method do double-duty. Also, its use is a bit odd:
double_LL * head = NULL;
double_LL * tail = NULL;
/* first entry has to be special cased */
tail = head = LL_insert(NULL, first_value);
/* all others use this insertion idiom */
tail = LL_insert(tail, second_value);
tail = LL_insert(tail, third_value);
...
We could instead use a shorter and more consistent function. Instead of taking (and returning) a double_LL*
, this would work with pointers to double_LL*
. The extra level of abstraction actually makes things much more consistent:
double_LL ** LL_insert(double_LL ** next_addr, double new_value) {
double_LL * old_next = *next_addr;
Newx(*next_addr, 1, double_LL);
(*next_addr)->value = new_value;
(*next_addr)->next = old_next;
return &((*next_addr)->next);
}
To use this, we would set up a tail
pointer that initially points to the address of head
:
double_LL * head = NULL;
double_LL ** tail = &head;
tail = LL_insert(tail, first_value);
tail = LL_insert(tail, second_value);
tail = LL_insert(tail, third_value);
...
This is really nice for a number of reasons. First, because this insertion function takes a double pointer, the first insertion will update head
to point to the first element of the list, but none of the others will change it. Second, head
will be NULL
if we happen to make zero insertions. Third, if the insertions are performed in the body of a loop, our loop will be more concise because it does not need to watch out for or do anything special for the first entry. Although it is a bit harder to grok, I would argue that this insertion function is better than the previous one.
To get an idea of how I might use this linked list, here is a script that parses my kernel log collecting the times at which kernel events were logged. Notice that I use C::Blocks::Types::Pointers
to declare pointer types to facilitate this work. This module is not yet distributed on CPAN, but is in the github repository.
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI; # explicit for Newx and Safefree
use C::Blocks::Types qw(double uint);
use C::Blocks::Types::Pointers
double_LL => 'double_LL*',
double_LLp => 'double_LL**';
# Basic linked list.
clex {
... struct and functions from above ...
}
# Set up Perl variables to hold the pointers to the linked list head and tail.
my double_LL $head = 0;
my double_LLp $tail_p = 0;
cblock {
$tail_p = &$head;
}
# scan through kernel log and calculate the time between events. I'll assume
# that the path is given in @ARGV so I can just use the diamond operator.
while (my $line = <>) {
if ($line =~ /kernel: \[(\d+\.\d+)\]/) {
my double $time = $1;
cblock {
$tail_p = LL_insert($tail_p, $time);
}
}
}
# Calculate the average time between log entries
my double $avg = 0;
my uint $N_skips = 0;
cblock {
if ($head != 0) {
double sum = 0;
unsigned int N = 0;
double prev_time = $head->value;
double_LL * curr = $head->next;
while (curr != NULL) {
double diff = curr->value - prev_time;
if (diff < 100) {
sum += diff;
N++;
}
else {
$N_skips++;
}
prev_time = curr->value;
curr = curr->next;
}
$avg = sum / N;
}
}
print "Average time between close log entries is $avg seconds; skipped $N_skips separated entries\n";
# All done, free up the linked list
cblock {
LL_free($head);
}
While that's a lot of code, notice that after the clex
block it breaks basically into four units: set up $head
and $tail_p
, scan through the log file, calculate and print the statistics, and free up the memory. Thanks to C::Blocks::Types::Pointers
, I was able to set up double_LL
as a pointer to the struct of the same name, and double_LLp
as a pointer to that pointer. This makes it possible to dereference $head->value
without getting warnings, or to call LL_insert
without warnings.
I hope you can see my point through the silliness of the example. Obviously it's not hard to write a pure Perl script that performs the same calculation with about half as many lines of code. If you do so, you'll see it runs 1x to 2x faster than this C example. My point isn't that you'd use this linked list to perform this calculation. Rather this illustrates how C::Blocks makes it possible to define C data structures. Typing facilities like C::Blocks::Types and C::Blocks::Types::Pointers minimize the boiler plate you would need to perform operations on those data structures. All of these lower the barrier to entry to using C for complex algorithms and data structures.
EDIT: After some thought, I made a slight revision to how C::Blocks::Types::Pointers
works so that you can now get the address of a pointer by simply saying &$thing
.
C::Blocks::Filter
, does not manipulate the code at all but prints out the contents of each C::Blocks block just before compiling it. It can be invoked from the command-line when something is not working and serves as a useful debugging aid. For example, this script:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
cblock {
printf("Merry Christmas from C::Blocks!\n");
}
Can be run with C::Blocks::Filter
, producing:
$ perl -MC::Blocks::Filter test.pl
##################################################
void op_func(C_BLOCKS_THX_DECL) {
printf("Merry Christmas from C::Blocks!\n");
}
##################################################
Merry Christmas from C::Blocks!
This lets us peak under the hood to see what's really going on. Between the many #
symbols, you can see that the code for our cblock
was wrapped into a function called op_func
. After C::Blocks compiles this little snippet of C code, it gets the pointer to this function and stores it in the op-tree. This is later invoked when the Perl interpreter encounters the location in your code where the cblock
was positioned.
This gives us a useful tool to see exactly how C::Blocks does its magic. When applied to the first examples from Day 2 we get:
$ perl -MC::Blocks::Filter test.pl
##################################################
void op_func(C_BLOCKS_THX_DECL) {SV * _PERL_SCALAR_message = (SV*)PAD_SV(1);
char * message = SvPVbyte_nolen(_PERL_SCALAR_message);
printf("%s from C::Blocks\n", message);
sv_setpv(_PERL_SCALAR_message, "Feliz Navidad!");
}
##################################################
Merry Christmas! from C::Blocks
After the cblock, the message is Feliz Navidad!
In this example I had used the variable $message
directly in my cblock
. Now we see how that works. At compile time, C::Blocks gets the location in the pad where $message
lives (offset 1 in this case) and injects code to retrieve the SV*
from that pad location. Through the rest of the code, $message
is replaced with _PERL_SCALAR_message
.
And how about types? When applied to the first examples from Day 3 we get:
$ perl -MC::Blocks::Filter test.pl
##################################################
void op_func(C_BLOCKS_THX_DECL) {SV * SV__PERL_SCALAR_limit = (SV*)PAD_SV(1); int _PERL_SCALAR_limit = SvIV(SV__PERL_SCALAR_limit);
for (int i = 0; i < _PERL_SCALAR_limit; i++) {
printf("Ho ho ho! ");
}
printf("\n");
sv_setiv(SV__PERL_SCALAR_limit, _PERL_SCALAR_limit);}
##################################################
Ho ho ho! Ho ho ho! ...
Here we see that C::Blocks puts the SV*
for $limit
in a slightly different C variable, SV__PERL_SCALAR_limit
. As before, the variable name $limit
is replaced with _PERL_SCALAR_limit
, but that is not of type int
. Also, there is an additional line at the end which sets the Perl variable's value to whatever is in _PERL_SCALAR_limit
. This way, if we modify _PERL_SCALAR_limit
, the effect is visible after the block has run.
However, source filters are generally meant for modifying code, not printing debug output. With a source filter we can easily add new quasi-keywords, such as loop
in this example:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
# loop->while filter
sub loop_to_while {
s/loop/while(1)/g;
}
use C::Blocks::Filter qw(&loop_to_while);
cblock {
int i = 0;
loop {
printf("This is number %d\n", i++);
if (i == 10) break;
}
}
and here's what we get:
$ perl test.pl
This is number 0
This is number 1
This is number 2
This is number 3
This is number 4
This is number 5
This is number 6
This is number 7
This is number 8
This is number 9
In C, the break
keyword behaves like the last
keyword in Perl, so when i
reaches 10, we exit the loop. To get an idea of how this really works, let's add use C::Blocks::Filter
both before and after the custom filter:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
# loop->while filter
sub loop_to_while {
print "*** converting loop to while(1)...\n";
s/loop/while(1)/g;
}
use C::Blocks::Filter;
use C::Blocks::Filter qw(&loop_to_while);
use C::Blocks::Filter;
cblock {
int i = 0;
loop {
printf("This is number %d\n", i++);
if (i == 10) break;
}
}
Running that produces:
##################################################
void op_func(C_BLOCKS_THX_DECL) {
int i = 0;
loop {
printf("This is number %d\n", i++);
if (i == 10) break;
}
}
##################################################
*** converting loop to while(1)...
##################################################
void op_func(C_BLOCKS_THX_DECL) {
int i = 0;
while(1) {
printf("This is number %d\n", i++);
if (i == 10) break;
}
}
##################################################
This is number 0
This is number 1
...
The first C::Blocks::Filter
, being the first in the series, gets the code as extracted by the C::Blocks code extractor, including the loop
. The second C::Blocks::Filter
is called after loop_to_while
has been applied, letting us see its effect. It has replaced loop
with while(1)
.
Applying a filter by naming the function is useful, but repeatedly using the same filter in this way can lead to lots of typing. Since this is Perl, there is always More Than One Way to Do It. In particular, filters can also be implemented as modules. Getting their effect is as simple as including use My::Filter::Module
in your script.
Like so many aspects of C::Blocks, the code to which any filter gets applied is limited by its lexical scope. This lets you apply filters with high levels of granularity, if you so wish.
Interpolation blocks are another tool for generating code: you can write snippets of Perl code that generate C code within your C::Blocks block. These are the philosophical counterpoint of source filters. Source filters are meant to modify your code, effectively enhancing the language, but a source filters have the tendency of over-reaching, modifying bits of the code they shouldn't. This is why source filters are generally discouraged in the Perl community. Interpolation blocks are different: they cannot modify the contents of a block, but can generate code at a specific location. They are blocks of Perl code enclosed as ${ ... Perl code here ...}
. The code is run as soon as it is extracted, and the return value is injected into the C code in its place. For example, you could programatically build up the entries in a C struct as follows:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
our @struct_fields;
# add points for x and y
BEGIN { push @struct_fields, 'float x', 'float y' }
# ...
# add a color
BEGIN { push @struct_fields, 'int color_idx' }
# ...
# add a name
BEGIN { push @struct_fields, 'char * name' }
# ...
clex {
typedef struct My::Labeled::Point {
${ join ('', map "$_;\n", @struct_fields) }
} My::Labeled::Point;
}
# ...
cblock {
My::Labeled::Point test;
test.x = 1.05;
test.y = -3.4;
test.color_idx = 27;
test.name = "Frank";
printf("test's position is (%f, %f)\n", test.x, test.y);
}
The interpolation block is the mess of code within the struct declaration, ${ join ('', map "$_;\n", @struct_fields) }
. This produces a string of valid C code with the different fields. To see this in action, it's probably best to run this with C::Blocks::Filter
like so:
$ perl -MC::Blocks::Filter test.pl
##################################################
typedef struct My__Labeled__Point {
float x;
float y;
int color_idx;
char * name;
} My__Labeled__Point;
##################################################
##################################################
void op_func(C_BLOCKS_THX_DECL) {
My__Labeled__Point test;
test.x = 1.05;
test.y = -3.4;
test.color_idx = 27;
test.name = "Frank";
printf("test's position is (%f, %f)\n", test.x, test.y);
}
##################################################
test's position is (1.050000, -3.400000)
As promised, the contents of the struct
declaration are filled with the types that were assembled: float x
, int color_idx
, etc. The cblock
then uses this struct type and assigns to the various fields in the struct.
You probably also noticed that I used double-colons: My::Labeled::Point
, and these were replaced with double-underscores during the code extraction stage, producing My__Labeled__Point
. There is nothing fancy going on here, it just replaces the double-colons (invalid C syntax) with double-underscores (valid C syntax) as a notational convenience.
These mechanisms for generating code are not without their warts. Keeping track of line numbers is currently shaky with C::Blocks, and a number of line counting issues have yet to be fully resolved with these approaches. Those kinds of issues are a major target for work once C::Blocks reaches Beta.
The third method for generating code on the fly is the humble string eval. There is a very clever use of string evals, but this Advent entry is already too long, so I will not demonstrate it yet.
Today I explained the many facilities for generating or modifying C code with C::Blocks. It is possible to modify C code with a source filter, and to generate code at a specific point using interpolation blocks. Incidentally, it's possible to put these together. Interpolation blocks inject code as soon as their closing bracket is detected, and this generated code is part of what gets sent to the filters. (That's why using C::Blocks::Filter was able to print it out.) Both of these mechanisms were implemented with specific, distinct tasks in mind, and I'm sure they will be used for a variety of purposes as the distribution matures and sees new and unexpected uses.
]]>Sorry that this Fifth Advent entry is being posted on the Seventh day of December! I got some really weird benchmark results and it took a while to really dig through them and figure out what was going on. I’ll have to work double-time for the next few days to get caught back up!
]]> Let’s begin by measuring how long it takes to loadC::Blocks
. As a point of comparison, a script that does nothing except load strict
and warnings
would be this:
use strict;
use warnings;
On my machine the bash built-in time
usually reports a duration of 5-10ms:
$ time perl test.pl
real 0m0.009s
user 0m0.008s
sys 0m0.000s
If we simply include C::Blocks, the time jumps up a little:
# test.pl
use strict;
use warnings;
use C::Blocks;
$ time perl test.pl
real 0m0.043s
user 0m0.040s
sys 0m0.000s
Simply loading C::Blocks costs about 30ms. Finally, if we load PerlAPI, Types, and include a cblock
that modifies a variable, we get some idea for the minimal cost of really using C::Blocks. Here’s the test script:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::PerlAPI;
use C::Blocks::Types qw(uint);
my uint $foo = 0;
cblock {
$foo = 5;
}
Notice it doesn’t print anything because I don’t want the I/O system to interfere with the timing. Running this I get a running time of no more than 60ms:
$ time perl test.pl
real 0m0.060s
user 0m0.048s
sys 0m0.008s
Thus, the minimal additional cost of doing something useful with C::Blocks (on my machine) should be at least 50ms, five times what it takes to merely load the Perl interpreter.
With these as our baseline, let’s do some more useful benchmarks. Perl is supposed to have perfomant I/O and string manipulation. What is the tradeoff between pure-Perl and C for munging the contents of a text file? We expect the C code to be lengthier, but will it lead to any appreciable performance increase? To find out, I generated text files with lines of random text, some of which included Fred. The program should create a new file with the same contents except Fred is replaced by Barney. Here is the Perl program:
use strict;
use warnings;
open my $out_fh, '>', 'perlout.txt';
while (<>) {
s/Fred/Barney/g;
print $out_fh $_;
}
Here is the C::Blocks program:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(char_array);
my char_array $filename = $ARGV[0];
cblock {
FILE * in_fh = fopen($filename, "r");
FILE * out_fh = fopen("cout.txt", "w");
char * original = "Fre";
int match_length = 0;
int curr_char = fgetc(in_fh);
while (curr_char != EOF) {
if (curr_char == original[match_length]) {
/* found character in sequence */
match_length++;
}
else if (match_length == 3 && curr_char == 'd') {
/* found full name! print and reset */
fprintf(out_fh, "Barney");
match_length = 0;
}
else {
/* incomplete match, print what we've skipped */
if (match_length) fprintf(out_fh, "%.*s", match_length, original);
/* just in case we have FFred or FreFred */
if (curr_char == 'F') match_length = 1;
else {
match_length = 0;
fputc(curr_char, out_fh);
}
}
curr_char = fgetc(in_fh);
}
fclose(in_fh);
fclose(out_fh);
}
Here’s a breakdown of the difference between these two programs. For the Perl script, we have:
As for the C script, we have:
original
First the caveats. The execution times are obviously specific to my machine, but the relationship between them is likely to remain the same on other platforms. If this code were compiled using gcc, it would likely run faster because the Tiny C Compiler underlying C::Blocks does not produce lightning fast C code. (We’ll get to that in a second.) Also, I am pretty sure that a more succinct C version could be written, and there’s probably another C implementation of this process that would run faster. Finally, I didn’t really hit noticeable performance issues until I processed 100,000 lines of text. Anything shorter than that was hardly any different from the startup times.
All of those caveats having been said, I doubt that the C::Blocks version can be made to run significantly faster. As such, we are left with the basic conclusion that the Perl version is easier to write, easier to understand, fewer lines of code, and comparable speed. If you are trying to write code that performs lots of I/O and string manipulation, just work with pure Perl.
The conclusion from the previous benchmark is not a surprise because I was playing to Perl’s strengths. One of Perl’s weak spots is fast numerics, like the sort of random number generator work I introduced yesterday. Is C::Blocks significantly faster than Perl? And, does it deliver performance that is comparable to what we would get with an XS module? To find out, I ran the script located at the bottom of this post. The script implements the random number generator in pure Perl, C::Blocks, and Inline::C. In these results, the N = 10
line indicates the number of iterations run through the random number generator in a single function call, so we actually have a sequence of results for a wide range of iteration count:
--- For N = 10 ---
Rate Perl CBlocks Inline
Perl 275691/s -- -96% -97%
CBlocks 7126245/s 2485% -- -24%
Inline 9331219/s 3285% 31% --
--- For N = 31 ---
Rate Perl CBlocks Inline
Perl 91167/s -- -97% -98%
CBlocks 3243566/s 3458% -- -44%
Inline 5782587/s 6243% 78% --
--- For N = 100 ---
Rate Perl CBlocks Inline
Perl 28981/s -- -98% -99%
CBlocks 1214700/s 4091% -- -56%
Inline 2752511/s 9398% 127% --
--- For N = 316 ---
Rate Perl CBlocks Inline
Perl 9221/s -- -98% -99%
CBlocks 413537/s 4385% -- -59%
Inline 1017940/s 10940% 146% --
--- For N = 1000 ---
Rate Perl CBlocks Inline
Perl 2901/s -- -98% -99%
CBlocks 132740/s 4476% -- -61%
Inline 337317/s 11528% 154% --
--- For N = 3162 ---
Rate Perl CBlocks Inline
Perl 922/s -- -98% -99%
CBlocks 42309/s 4487% -- -62%
Inline 110277/s 11856% 161% --
--- For N = 10000 ---
Rate Perl CBlocks Inline
Perl 293/s -- -98% -99%
CBlocks 13525/s 4512% -- -62%
Inline 35223/s 11911% 160% --
Above N = 10,000, the relative results were essentially unchanged.
The good news is that the C::Blocks implementation consistently beats out the Perl implementation by a significant margin. Looking at the rate, we see the C::Blocks version producing between 25x and 45x more random numbers per second than the Perl implementation. What’s more, the line counts for these two implementations are pretty similar. Switching to a C::Blocks implementation would produce a huge speedup in this portion of code at essentially the same code complexity.
On the other hand, the Inline::C implementation is clearly the fastest implementation. For 10,000 iterations and higher it will produce 2.6 random numbers with each round of the C::Blocks implementation. For some, that factor of 2.6 is crucial, and an XS-based solution is the right tool.
Real code does not boil down to benchmarks like this. The execution of real code is often governed by a number of slow points whose effect is cumulative. Speeding up the worst offender will speed up your program, but then you’ve only removed one of many slow points. While the others remain, making any satisfactory code blazingly fast is a waste of time. If you are considering using using C to speed up some critical piece of code, C::Blocks won’t give you the fastest implementation, but it’ll probably be fast enough.
Today I looked into the performance of C::Blocks for two important circumstances: a task dominated by I/O and text processing, and a task dominated by computation. For text processing Perl is clearly the right tool for the job. For numerical computation, Inline::C is clearly faster, but if the computation is only one bottleneck in a complicated program, the speedup from C::Blocks is likely good enough to move the pain-point to some other part of the code base.
C::Blocks Advent Day 1 2 3 4 5 6 7 8 9 10 11 12 13
P. S. Why was this entry so late? Apart from being pressed for time, I got benchmark results that varied wildly. I’m OK presenting benchmark results that I don’t like: I’m a bit annoyed by that factor of 2.6, but that’s an fair representation of things. I’m not OK with the kind of crazy results I was getting, sometimes showing that the C::Blocks implementation was competitive with Inline::C, and other times showing that it ran at the same speed as the Perl one. After a wide variety of implementations, I came to the conclusion that it was a CPU memory page alignment issue. The latest version on github fixes this and I will release another pre-Beta version soon (but for now, the bed is calling).
# rng.pl
use strict;
use warnings;
use C::Blocks;
use Inline 'C';
use C::Blocks::Types qw(uint);
use Benchmark qw(:hireswallclock cmpthese);
my uint $N;
my $a = 698769069;
my ($x, $y, $z, $c) = (123456789, 362436000, 521288629, 7654321);
my $reps = 10;
for my $log_n (1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5) {
$N = int(10**$log_n);
print "--- For N = $N ---\n";
cmpthese(-1, { Inline => \&Inline_rng, CBlocks => \&c_blocks_rng,
Perl => \&Perl_rng});
}
sub Perl_rng {
my $rand;
for (1 .. $N) {
my $t;
$x = 69069*$x+12345;
$y ^= ($y<<13); $y ^= ($y>>17); $y ^= ($y<<5);
$t = $a*$z+$c; $c = ($t>>32);
$z = $t;
$rand = $x+$y+$z;
}
return $rand;
}
clex {
/* Note: y must never be set to zero;
* z and c must not be simultaneously zero */
unsigned int x = 123456789,y = 362436000,
z = 521288629,c = 7654321; /* State variables */
unsigned int KISS() {
unsigned long long t, a = 698769069ULL;
x = 69069*x+12345;
y ^= (y<<13); y ^= (y>>17); y ^= (y<<5);
t = a*z+c; c = (t>>32);
return x+y+(z=t);
}
}
sub c_blocks_rng {
my uint $to_return = 0;
cblock {
for (int i = 0; i < $N; i++) $to_return = KISS();
}
return $to_return;
}
sub Inline_rng {
inl_rng($N);
}
__END__
__C__
/* Note: y must never be set to zero;
* z and c must not be simultaneously zero */
static unsigned int x = 123456789,y = 362436000,
z = 521288629,c = 7654321; /* State variables */
unsigned int inline_KISS() {
unsigned long long t, a = 698769069ULL;
x = 69069*x+12345;
y ^= (y<<13); y ^= (y>>17); y ^= (y<<5);
t = a*z+c; c = (t>>32);
return x+y+(z=t);
}
unsigned int inl_rng(unsigned int N) {
int i;
unsigned int to_return;
for (i = 0; i < N; i++) to_return = inline_KISS();
return to_return;
}
]]>
use
d throughout your code.
]]>
I will begin with a specific implementation of a random number generator. This is a decent candidate for a C::Blocks implementation when you anticipate needing to produce many random numbers, such as for Monte Carlo simulations. I came across a nice little writeup about random number generators by David Jones, and the author provides a decent starting point called KISS
. How would that library be implemented using C::Blocks? Place the global variables and the function in a clex
block, then call the function in your cblock
s, like so:
use strict;
use warnings;
use C::Blocks;
# Implement KISS random number generator, copy-and-pasted from
# http://www0.cs.ucl.ac.uk/staff/d.jones/GoodPracticeRNG.pdf
clex {
/* Note: y must never be set to zero;
* z and c must not be simultaneously zero */
static unsigned int x = 123456789,y = 362436000,
z = 521288629,c = 7654321; /* State variables */
unsigned int KISS() {
unsigned long long t, a = 698769069ULL;
x = 69069*x+12345;
y ^= (y<<13); y ^= (y>>17); y ^= (y<<5);
t = a*z+c; c = (t>>32);
return x+y+(z=t);
}
}
use C::Blocks::Types qw(uint);
# Get a random number from the generator
my uint $random = 0;
cblock { $random = KISS(); }
print "Random number is $random\n";
When run, this produces
$ perl test.pl
Random number is 2079675107
As you can see, the clex
block contains a function and some global variable declarations which are available to the function and to any cblock
s later in the same lexical scope. Indeed, as the name suggests, the visibility of the declarations in clex
blocks are lexically scoped. As you know, you cannot use a Perl variable outside of its lexical scope. Likewise, the Perl block that surrounds a clex
block defines the scope of the declarations in that block: a cblock
or clex
block outside of that scope will not be able to see the declarations.
David Jones makes a number of suggestions for improving the generator, but the aim of this Advent treat is not really to implement a random number generator. I simply want to use the idea of such a generator to motivate some of the features of C::Blocks.
A random number generator is meant to be reused. I could of course create a Perl function with the cblock
in it, making it reusable from Perl. But how do I make my C functions sharable to other modules and scripts? The answer is simple. Place the clex
block in a module with a package name matching the module name, and change clex
to cshare
. For example, here is a module that provides the same random number generator, called MyRNG.pm
:
# MyRNG.pm
package MyRNG;
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(uint);
# Implement KISS random number generator, copy-and-pasted from
# http://www0.cs.ucl.ac.uk/staff/d.jones/GoodPracticeRNG.pdf
cshare {
unsigned int x, y, z, c;
unsigned int KISS() {
unsigned long long t, a = 698769069ULL;
x = 69069*x+12345;
y ^= (y<<13); y ^= (y>>17); y ^= (y<<5);
t = a*z+c; c = (t>>32);
return x+y+(z=t);
}
}
# Set up state
my uint ($x, $y, $z, $c) = map 1 + int(rand(4294967294)), 1 .. 4;
cblock {
x = $x;
y = $y;
z = $z;
c = $c;
}
1;
Notice that this module initializes the state to random numbers via a cblock
. For cblock
s written within the module itself, there is no difference between clex
and cshare
. The difference comes in which blocks have their symbol tables shared when the module is use
d. Here I illustrate that I can use a local cblock
to initialize global variables declared in a cshare
block.
An example of a script that uses this module:
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(uint);
use MyRNG; # import KISS()
# Produce a random number
my uint $rand = 0;
cblock {
$rand = KISS();
}
print "Random number is $rand\n";
Take a moment to appreciate the following two facts: (1) Writing the module was absurdly easy, basically copying a clex
block into a module file and renaming the block to cshare
. (2) Importing and using the function in the module was absurdly easy, just a use
statement. Inline::C has made the first of these fairly simple, and ExtUtils::Depends sorta provides a system for the second of these. C::Blocks was designed to simultaneously solve both, and I believe it provides an easier-to-use approach.
Finally, here is an illustration of the lexical scoping in action, to give you an illustration of how it really works. The symbol tables imported when you use
a module are lexically scoped to the block into which you use
them (just like the effect of use warnings
is lexically scoped).
use strict;
use warnings;
use C::Blocks;
use C::Blocks::Types qw(uint);
my uint $rand = 0;
# Create an inner lexical scope
{
use MyRNG;
cblock {
$rand = KISS();
}
print "First random number is $rand\n";
}
cblock {
$rand = KISS();
}
print "Second random number is $rand\n";
Upon running this script, I got an error:
$ perl test.pl
C::Blocks compiler error:
test.pl:18: error: undeclared function 'KISS'
If you comment out that line, it runs just fine.
Today I explained how C::Blocks makes it easy to write functions and other declarations using clex
blocks. I also explained how to write a module that provides function and other declarations for other modules by using cshare
blocks. This module-based C code sharing was the driving impetus for this project all along, and getting it right has been central to my work on it. Even though C::Blocks makes all of this easy, does that mean you should use it? Perl is much more expressive and you may be able to articulate your algorithmic thoughts in 1/3 the number of lines compared to a C implementation. Choosing the right tool---the right language---for the job shall be the topic of a forthcoming treat.