Finally, with our new website, we're starting a project to help teach some topics with more depth, to help fill in the gaps between the beginner's materials and writing useful, maintainable code using the best CPAN has to offer. We've started the Chicago.PM Github Organization, where we will be developing presentations that can be used freely by anyone wishing to learn more about Perl, or wanting to teach others about Perl. The presentations and the code inside is licensed under the same terms as Perl itself, so if you can use Perl, you can use and edit the presentations to suit your needs.
The first presentation we're releasing is about building command-line applications in Perl. I've given this presentation twice now, to excellent reviews. There is still some content that can be added to make it a more robust presentation (especially in the http://metacpan.org/release/App-Cmd section), but it's a good talk on going from "Just read my first Perl book and my first Linux book" to "Wielding Perl to build command-line applications."
We hope to build more presentations of this type in the future, as well as other learning materials as time goes on. Let us know if you want to help!
]]>For those who have added Ack to their regular arsenal, the 2.0 release brings some minor breaking changes (removing the -a option for "search all files" and changing the default file type filter to "all text files") designed to make Ack more Do-What-I-Mean and more compatible with Grep (when it makes sense, ack is made for humans). The result being that using Ack should not take you out of The Zone by requiring you to check the manual every time (something I must confess to doing often with Grep and Find).
I must also confess to using only the most basic Ack command-line options, but Andy explored a lot of functionality I was not aware of, both in Ack and Grep:
# Search for full words
ack -w
grep -w
# List the files that matched instead of the matching line
ack -l
grep -l
# Invert the match ("does not match X")
ack -v
grep -v
# Override the default output with capture groups
ack 'use (\S+);' --output='$1' -h | sort -u
# Search the file path for matches (Beyond Find)
ack -f
ack -g
# Accept a list of files to search on STDIN (Built-in Xargs)
ack -x
The last two features are great for chaining together Ack commands:
# Find all perl files not in the 'release' repository
ack -g -v 'release' --perl |
# ... search them for the word '2.46.3207.2'
ack -x '2.46.3207.2' -w -l |
# ... and if they match, edit them with sed
xargs sed -i 's/2.46.3207.2/2.49.3333.12/'
# This little snippet saved me some hours of manual work upgrading version numbers
There was also a lot of discussion on other tools to help programmers be more efficient. The ones I was most interested in were ctags, which I don't use nearly enough (I use ack instead), and cscope, which I had never heard about.
The problem with adding new things to your toolkit is integrating them into your flow (the flow that can be consciously experienced is not the true flow). Whenever you add a new thing, especially if it replaces an existing thing, you're losing efficiency in the short-term in order to add potential efficiency in the long-term. Despite this, a lot of things are worth it, Ack more-so than most.
Sometimes you don't know you're doing something inefficiently until someone comes along and tells you so, or shows you a solution to a problem you never knew you were having. Ack is a solution to a grep problem: Grep is harder to use in the most-common cases. That said, Grep still has its uses, just uncommon ones.
A big thanks to Andy for giving a presentation on such short notice! Our next meeting is on May 23, so check out the Chicago.pm website and join us!
]]>use DateTime;
my $date = DateTime->new( year => 2013, month => 4, day => 15 );
$date->set_time_zone("Australia/Sydney");
print $date->today;'
This code gives the error Can't locate object method "_normalize_nanoseconds" via package "2013-04-15T00:00:00" at /usr2/local/perlbrew/perls/perl-5.16.3/lib/site_perl/5.16.3/x86_64-linux-thread-multi/DateTime.pm line 252.
The package "2013-04-15T00:00:00" is the curious part: It looks like a stringified DateTime, but who could possibly be stringifying a DateTime object and then using that as a package name?
]]> It turns out the problem is at$date->today. today() is a constructor, constructors are class methods, constructors inevitably call bless on the class they are invoked with. But, we called this constructor with an object instance (a blessed reference), not a class.
In an object without overloads, this fails (as expected) with an error message: Attempt to bless into a reference. But DateTime overloads stringification. So instead of trying to use an object as a package name, the object gets stringified and that string gets used as a package name.
This problem could be mitigated by checking for refs in the constructor, dying as a result. I'm not sure if it would be a good idea to disallow stringification during a call to bless, for historical reasons (it's not a bug if it's working as expected, it is a bug if a change breaks code).
Here's some sample code to play around with the problem. Enable/disable the overload (comment out the use overload (...)) and see what changes:
use strict;
use warnings;
package Foo;
use overload (
q{""} => sub { return "Bar" },
);
sub new {
my ( $class ) = @_;
return bless {}, $class;
}
sub greet {
print "Hello, World!\n";
}
package main;
my $obj = Foo->new;
$obj->greet;
my $clone = $obj->new;
$clone->greet;
What follows is an account of this ordeal, provided for entertainment value only. There will be a future post that explains some of the things I did to make this task surmountable.
]]>my $arg_name = $args{arg_name}; return $self->method( arg_name => $arg_name ) should never happen!Total time elapsed: 3.25+3.5+2.25 = 9 hours. Not bad for 130 commits to migrate 122 modules.
]]>[EDIT: The presentation doesn't appear to work on mobile devices. I'm trying deck.js, and I'm not sure I like it.]
]]> When I started my current job, I was introduced to the custom Dependency Injection framework they built. At the time, I was dismissive: "It's programming via config file, which is always limiting (and so, a Bad Idea)." "It's something that Java needs, but not Perl."Slowly, and with great reluctance, I learned where dependency injection fits into a large framework: It removes the work of creating objects. Any object. Anywhere.
For an idea of how powerful this is, do a quick search through your libraries and scripts (except for tests) for /->new/. Hopefully, the results will only be defaults (otherwise you've got tight coupling, another problem DI can fix). But every time you call an object constructor is another place where dependency injection can reduce the amount of code you write and maintain.
So, since the bureaucracy means I can't simply open-source the DI framework we have at $dayjob, I decided to make my own.
Beam::Wire is available on CPAN. There are a lot of features planned for it, so stay tuned! If you're interested, up-vote Beam::Wire on play-perl.
]]>Best of all, the talk itself was written in Mojolicious! As Joel was talking, he was able to edit the code and display the results, showing off various features of Mojolicious like:
There are quite a few interesting parts of Mojolicious that make it worth giving a look to, and I hope to be able to do so with some web projects that have been sitting in my queue for a while (I wrote a nice ticket tracker with AngularJS, but the backend is Python, I'd like to fix that glaring mistake).
]]>Since the XS module was intended to be used by other, larger systems, I decided to help those larger systems test their dependency on my module: I included a Test::MockObject that mocked my module's interface. By using my test module, the tests can try some data and see if their code works.
But the hardest part to test is always the failures. How do they test if the news service goes down in the middle of a data pull? How about if it goes down between data pulls but still inside the same process? How do they test if the user has input an invalid ID for data?
To help them write good error-checking and recovery, I added specific Test::MockObjects that return failure conditions. Want to know what happens if an ID is invalid? Use the Test::Feed::InvalidId mock API. Want to know what happens if the feed goes down in the middle of a process? Use the Test::Feed::RandomDisconnect mock API.
By providing these mock objects, users can write more robust code more simply. Then, in the eventuality that the service fails, they know exactly what their code won't do: Wake them up at 3 in the morning because of an unexpected, unhandled error. These mock objects don't even need an external connection to operate, so they can be as fast as possible.
Unfortunately, this is a limited solution that covers only what I know about. If the service changes, or has an error I didn't cover, the code still fails. For this reason, for the web services I build, I build in explicit ways to get errors. When the service itself gives the exact error output it gives in exceptional circumstances, it can be tested and handled. Instead of a mock service, it's the real service (perhaps on a dev system), with real input, returning a real (but desired) error message.
By providing as much help as possible to the users of my code, I can make sure they can create robust applications that make their own users happy. By helping users respond to and deal with the error conditions of my code, I can cut down on the support requests and bug reports. Being proactive about testing helps everyone write better code.
]]>At Double Cluepon, I use it to create the release packages based on tagged commits, so that releasing our software is exactly: git tag vX.X.X && git push --tags. A Perl script builds every package and then pushes them to our update server, where the game will check for a new release.
At Bank of America, we use it to combine our 20-30 Perl distributions into a single release. Using git submodules, we have a "release repository" that holds references to all the modules for each team's releases (some are team-specific, others are shared between teams). A Perl script manages the submodules, determines when the submodule refs need to be updated, tags and branches for each release, and finally builds and installs our modules using Module::Build and local::lib.
All this Git stuff gave me some ideas for possible useful code I can release, perhaps leading to me finally recovering my CPAN ID.
The slides for my Scripting Git With Perl talk
The code for the script that automatically builds releases tagged like "vX.X.X"
]]>package FakeRepository;
use Moose;
use TimeSeries;
has timeseries => (
is => 'rw',
isa => 'TimeSeries',
required => 1,
);
package TimeSeries;
use Moose;
has dates => (
is => 'rw',
isa => 'HashRef[Number]', # Date => Value pairs
default => sub { {} },
);
Here we have a FakeRepository that requires a TimeSeries object. Certainly, this is where a Dependency Injection framework could step in and inject the required TimeSeries. The drawback is the indirection: The two configured objects are completely separate, joined only by the reference, like so:
# dependency.yml
- service:
name: 'test_repo'
class: 'FakeRepository'
constructor_args:
timeseries: { ref: test_data }
- service:
name: 'test_data'
class: 'TimeSeries'
constructor_args:
dates:
2012-01-01: 1.56
2012-01-02: 1.69
2012-01-03: 1.45
So here, we define two services (objects), test_repo and test_data, and test_repo uses test_data to fill its timeseries attribute. test_data fills in its dates attribute directly from the configuration file.
This works great if test_data is needed by more than just test_repo. It also works fine as-is, the dependency injection framework does the work. But what if we wanted to specify the timeseries value directly, instead of indirectly?
Moose's typing system allows us to do just that. By creating a custom type with a coercion from the data structure in our configuration file, we can create the dependency that our test_repo needs.
package My::Types;
use MooseX::Types qw( HashRef Number );
use TimeSeries;
# declare our TimeSeries class as a type
class_type TimeSeriesType;
# coerce a TimeSeries from a hash of date => value pairs
coerce TimeSeriesType, from HashRef[Number], via sub {
# coercions put the value to be coerce in $_
return TimeSeries->new( data => $_ );
};
Once we have our new custom types, we must use them in our package.
package FakeRepository;
use Moose;
use My::Types qw( TimeSeriesType );
has timeseries => qw(
is => 'rw',
isa => TimeSeriesType, # Not quoted!
coerce => 1, # Activate coercion powers!
required => 1,
);
Now, we can configure our TimeSeries directly from our configuration file, without indirection.
# dependency.yml
- service:
name: 'test_repo'
class: 'FakeRepository'
constructor_args:
timeseries:
2012-01-01: 1.56
2012-01-02: 1.69
2012-01-03: 1.45
Moose will create the object for us using our defined coercion.
There are other ways to solve this: Enhance the Dependency Injection framework to allow anonymous objects (instead of providing a ref: to an object, provide a full object definition with class: and constructor_args:), but having these coercions in place also helps when writing test code:
use Test::More;
use FakeRepository;
my $repo = FakeRepository->new(
timeseries => {
'2012-01-01' => 1.56,
'2012-01-02' => 1.69,
'2012-01-03' => 1.45,
},
);
No need to increase the apparent coverage of the test by including the TimeSeries class, we never have to refer to it at all. No need to lock the interface to a specific TimeSeries class (if that's a desired goal of the project), the coercion takes care of creating the actual object used.
Coercions are a powerful feature. I've used them to build complicated trees from arrays of arrays (more on that later), and I've used them to simply force-uppercase a string so that Log4perl would do its job. Coercions are one more very useful tool in a robust toolbox.
]]>I learned some interesting things about DynaLoader that helped cement the idea that it's an interface on to the system's dynamic linker. I've never dealt with DynaLoader from the Perl side of things, except when it broke, and I would solve those problems in system-specific ways. I learned that DynaLoader could solve those problems in a cross-platform way.
I started writing my own Alien::Base module for libusb, but unfortunately ran out of time. We had some interesting side-discussion on human interface devices and augmented reality.
In two weeks, I'll be giving a talk on scripting Git with Perl. Come on down and check it out!
]]>Moose is built on top of Class::MOP. MOP stands for Meta-Object Protocol. A meta-object is an object that describes an object. So, each attribute and method in your class has a corresponding entry in the meta-object describing it. The meta-object is where you can find out what type constraints are on an attribute, or what methods a class has available.
Since the meta-object is a Plain Old Perl Object, we can call methods on it at runtime. Using those meta-object methods to add an attribute would modify our object, adding that attribute to the object. Using Class::MOP, we can compose classes at runtime!
I have recently used this to great effect in a custom dependency injection and configuration framework we have at Bank of America. By adding a "with" key to the configuration YAML file, the DI will create a new, anonymous class that composes the roles specified.
{
name: "Repository",
class: "Bank::HistoricalData::DailyRepository",
with: [ "Bank::Role::FlattenIntraday", "Bank::Role::CalculateHighLow" ],
constructor_args: { }
}
So, when I ask the DI for the "Repository" object, it will get the meta-object for Bank::HistoricalData::DailyRepository, create an anonymous class that extends Bank::HistoricalData::DailyRepository, and then compose the two roles into the new class.
The code to do all this is extremely short:
my $class = $conf->{class};
my $meta = Moose::Meta::Class->create_anon_class(
superclasses => [ $class ],
roles => $conf->{with},
);
$meta->make_immutable; # for performance
my $object = $meta->name->new( %{ $conf->{constructor_args} } );
return $object;
If a lot of objects end up composing the same roles, I can create a concrete class to get a bit of a performance boost. Since I create a new, anonymous meta-class, I don't have to worry about the class I'm extending being modified, or being made mutable again ($class->meta->make_immutable speeds up a lot of things, but doesn't allow us to add attributes or methods).
With this, I can be a lot more flexible about my dependencies, adding whatever role I want to change their behavior whenever I need.
]]>Along the way, we discussed logging practices (most of us are using Log::Log4perl), and the best way to get a Perl module ready for CPAN (I suggested using Module::Build directly, but Dist::Zilla outvoted me).
Sean mentioned in passing the Salt stack for executing commands across multiple machines, which looks like a very interesting alternative to more detailed tools like Puppet or Chef. Salt seems to be just a simple way to execute commands on multiple machines. Those commands could be administrative (restart httpd), or they could be the application.
One other very interesting thing from App::Services was providing a role for the larger system to use. So one creates App::Service::Log::Log4perl and App::Service::Log::Role, and in other services you consume the App::Service::Log::Role and then can use the Log4perl service. What's interesting about this is that the role is provided alongside the logging service, it isn't something that you have to create yourself. Doing it this way improves interoperability and simplifies adoption by giving you the tedious bits already. It's the difference between "Here's a service, figure out how to use it best" and "Here's a service, and here's a good way to use it" (I've been doing something similar by providing Test::MockObject implementations of my modules for testing purposes in the same distribution as the module itself. An "official" mock object).
In two weeks we have another project night, which I'm hoping is as productive as the last. If you're in the Chicago area, check us out on the Chicago Perl Mongers meetup page.
]]>We developers are a lazy lot, it's one of our greatest strengths. But like all virtues, it is a double-edged sword that must be wielded responsibly. Laziness is defined as maximum gain for minimal effort. For laziness's sake, I write automated tests to avoid manual testing, I build a development server that mimics production closely so I don't get woken up at 2:00 am when my code blows up, I write documentation that explains the design of the software so I don't have to trace through layers of code to figure it out, and anything else I can do to make sure I don't have to work as hard solving the same problem in the future.
Automated tests are the programmer's best friend. How do I know my code works? By running it. How do I know it works every time I change something? By running it again. Automated tests are simply running the code, controlling the input and checking the output. Since I write tests once according to expected input and output, I can refactor the entire program and be absolutely sure it works for that set of input. I don't need to test manually, clicking everywhere or preparing input on-the-fly, which saves me a lot of time in the long run (because this code will live forever).
A development server is another necessity for laziness. At a former employer I instituted a policy: All custom client projects included renting a development/staging server, no exceptions. This staging server could be completely destroyed and rebuilt with zero consequences. Our deployment processes were automated (more laziness) to make it easy to recover from mistakes. Once our automated tests passed, the clients had a place they could verify the expected behavior and see their ideas in action before releasing the code on to an important, production system. We instituted this policy from learning our lesson: Our laptops were never the same as the production server, so we needed some place that was. Deploying code directly to production is only rarely without unforeseen consequences.
I've been called a hero a couple times now. I don't believe or understand why, but I've noticed it's always after I mention how much I enjoy writing documentation. Writing documentation is like explaining my code to someone else, usually future me: It forces me to think critically about the code, how it works, and what side-effects and edge-cases it has. Every time I go back to write documentation I've found bugs in my code: incorrect assumptions, inelegant algorithms, or undesired side-effects. Usually, the first other person who reads them finds even more: things I didn't think of, or things I thought would never be a problem (famous last words). Writing the documentation gets me out of the code mindset and into the design mindset, and well-designed code is code that will live forever.
There is nothing more boring than a solved problem. By making sure I have automated tests, a sandbox environment, good documentation, I can make maintenance easier, which is the true definition of laziness: Maximum gain for minimum effort.
]]>There were some small ideas to make the program more user-friendly, and some larger ideas like an IRC client and guided tutorials based on the same format that http://perltuts.com uses.
I'm hoping that if I keep saying this, I'll be embarrassed into doing it: I would really love to see the Ruby Koans translated into Perl (in spirit, if not in actual content). I've started writing down ideas for chapters, but there is a lot of content to cover.
I added a -I flag and a -M flag to the prima-repl command-line launcher that work as close to perl's flags as I could get. This is one of the things I love about prove and plackup: Where it makes sense, they work like perl does. So now the prima-repl can have subs and modules imported on the command-line.
Altogether it was a wonderfully productive evening and I'm looking forward to the next one.
]]>My tests failed, but only sometimes. I was building an XS module to interface
with a C wrapper around a C++ library (wrapper unnecessary? probably). make
test was failing with exit code 11. Some quick searching revealed that I had
an intermittent segfault. Calling a function as_xml would fail with a SEGV
in strlen(). This only happened in perl after as_xml when perl was making
a SV out of the return value. This also only mainly happened during make test.
Doing prove myself would succeed 19 times out of 20, where make test would
fail 19 times out of 20. Worse, my C test program would never fail at all.
I changed everything I could think of: Using a debugging perl and keeping debug symbols in my C library and XS module made the failures happen less frequently (making debugging ever more frustrating). perlbrew was a big help here, letting me switch between different versions of perl, debugging perl, threaded perl, and combinations thereof.
After playing with GDB (once again succeeding 19 times out of 20), I gave up and searched the web. I found the same mailing list thread multiple times, and read it multiple times, not coming up with anything that was relevant to my situation.
Until I read the thread again after another frustrating half-hour with GDB: I had forgotten to put a prototype in the .h file, causing the char* pointer being returned to be treated as an int. On my 64-bit system, this causes segfaults. The compiler was warning me of this, "warning: initialization makes pointer from integer without a cast", but I didn't understand the warning (and the web was not helpful on that one).
Adding the function prototype to the C wrapper header and recompiling fixed the problem.
And that is why I need to learn a lot more about C. Function prototypes? Header files? Why are those necessary (I'm asking rhetorically, of course)? Heap and stack? Might as well be herp and derp.
Lesson reinforced: Depth of knowledge does not equal breadth of knowledge.
Also, having IRC at work might have saved me a few hours of hassle.
]]>