I wanted to know how difficult a setup of a Pinto server could become. The requirement behind was to access a single cpan-like repository for deploying server machines. The repository should contain company-provided distributions optionally combined with a collection of cpan-available distributions.
]]> Installing perl-modules on a server should be as simple ascpanm --mirror http://myrepo-server.de:port --installdeps .
If we had a cpan-like repository (possibly created using Pinto) somewhere we could run a simple PSGI application on the repo server
plackup -M Plack::App::File \
-e 'Plack::App::File->new(root => "/path/to/repo")->to_app'
and we are ready to go, if we had the repository filled.
Filling a repository sitting in a given directory on a certain machine using remote shell commands is not very comfortable but this is where pintod comes into play. Accessing our repository for reading without authentication is OK, but adding distributions to the repository should be password-protected. Not really a problem.
First, a repository has to get created on the remote machine.
pinto -r /path/to/repo init
Then, we need a way to authenticate. I chose the option to use simple password authentication. The preparation of passwords could look like this:
htpasswd -c /path/to/htpasswd user
Now we are ready to launch the pinto daemon (for demonstration purpose not deamonized)
pintod --root=/path/to/repo --port 5555 --auth backend=Passwd --auth path=/path/to/htpasswd
Back on the developer's machine, we can now query or fill the remote repository. Setting the PINTOREPOSITORYROOT environment makes things easier, otherwise the "-r" option is required at every pinto command.
export PINTO_REPOSITORY_ROOT=http://user:secret@myrepo-server.de:5555
pinto list
pinto add --author SOMEBODY /path/to/dist-0.01.tar.gz
Dear admin at $work, I would like to get a VM for our pinto-server tomorrow :-)
]]>singleton() method like this:
sub instance {
my $class = shift;
state $self = $class->new(@_);
return $self;
}
Just to get me right. This is not a complain to Moo*X::Singleton modules. The snippet above is very primitive, does not provide a separate initialization and has no way of clearing an instance. But sometimes this simplicity might do its job.
]]>As a simple aid, we peeked into pod2pdf and wrote a little wrapper around it in order to handle many files and create the outline. Well, this solution is not rock solid, as we rely on its internals but at least it is short and usually creating PDFs is not mission-critical.
Maybe some parts can still be improved, but for people interested, here is my first try: https://gist.github.com/2277444. Feedback welcome!
]]>
The main part of the module generates an HTTP-Header using this construct ($c is the mocked catalyst context, expire_in is a method containing the nr of seconds to expire in):
$c->response->headers->expires(time() + $self->expire_in)
if ....some_condition...
Well, adding a number of seconds to an epoch value cannot hurt. Can it? The test looked like this:
my $expected_date =
(DateTime->now
+ DateTime::Duration->new(seconds => $controller->{expire_in}
)->strftime('%a, %d %b %Y %H:%M:%S GMT');
is $c->response->header('expires'), $expected_date,
'expired header date is OK';
For the test case having an expiry of 3 years, the test failed with a difference of exactly one second:
# Failed test 'expired header date is OK' # at t/5-expire.t line 55. # got: 'Thu, 19 Feb 2015 18:27:59 GMT' # expected: 'Thu, 19 Feb 2015 18:27:58 GMT' not ok 4 - expired header date is OK
One second? Do we have a rounding problem? No. After a while, I remembered that DateTime mentioned a leap-second in its Change-Log some time ago. This could explain our difference.
Doing time-calculations by ourselves is wrong. Replacing the header generation with a more complicated but correct construct worked. Using DateTime before 0.71 kept the old date arithmetic-tests working, but actually the tests should have failed, because the leap-second was ignored.
$c->response->headers->expires(
DateTime->now
->add(seconds => $self->expire_in)
->epoch
)
My personal conclusion: Date arithmetic is dangerous. Be more strict to any kind of date arithmetic. Never ever do date-math again. Never. Instead: keep all date manipulating modules at a current version and trust in them. Their authors known what they are doing, I don't. I didn't :-(
]]>Some days ago, I was experimenting with the Redis noSQL Database in order to find an alternative to file based session storage for Catalyst web applications. Redis comes bundled with a benchmark tool redis-benchmark which reveals the speed of this database. On my Macbook I get these counts when benchmarking the database with single concurrency:
$ redis-benchmark -c 1 -n 10000 [...] ====== SET ====== 10000 requests completed in 0.50 seconds 1 parallel clients 3 bytes payload keep alive: 1 100.00% <= 0 milliseconds 20202.02 requests per second ====== GET ====== 10000 requests completed in 0.49 seconds 1 parallel clients 3 bytes payload keep alive: 1 100.00% <= 1 milliseconds 20408.16 requests per second [...]
I wondered what the speed would be using Redis from Perl. I first checked the speed of Redis.pm, a very cool Perl binding. It offers lots of features, fully implements the protocol and is fast while still being a pure-perl library.
How fast could I go when sacrificing everything, Redis.pm does by simply doing network writes and reads on my own? Can I reach the C benchmark's speed? Well, Perl is even faster -- and I also use JSON::XS to encode a hash :-) At least my benchmark is telling me...
Benchmark: timing 10000 iterations of get_native, get_redis, set_native, set_redis...
get_native: 0 wallclock secs ( 0.23 usr + 0.13 sys = 0.36 CPU) @ 27777.78/s (n=10000)
(warning: too few iterations for a reliable count)
get_redis: 2 wallclock secs ( 0.95 usr + 0.20 sys = 1.15 CPU) @ 8695.65/s (n=10000)
set_native: 1 wallclock secs ( 0.20 usr + 0.14 sys = 0.34 CPU) @ 29411.76/s (n=10000)
(warning: too few iterations for a reliable count)
set_redis: 1 wallclock secs ( 0.82 usr + 0.21 sys = 1.03 CPU) @ 9708.74/s (n=10000)
Rate get_redis set_redis get_native set_native
get_redis 8696/s -- -10% -69% -70%
set_redis 9709/s 12% -- -65% -67%
get_native 27778/s 219% 186% -- -6%
set_native 29412/s 238% 203% 6% --
The full benchmark, also comparing JSON::XS versus YAML::XS and pure writes without any serialization can be found here: https://gist.github.com/1697936
]]>
Improved version:
I hope it is not too bad. Otherwise I will have to wait until I reach a reasonable age :-)
]]>In order to get a feeling on how to use Marpa, I decided to try it for parsing CSS. Thankfully, the official grammar for CSS is available. So I tried to become a friend of Marpa::XS. I was used to work with Parse::RecDescent in the past, so I could not imagine to get into trouble. However, using both modules is different. Parse::RecDescent handles the scanning process by itself and allows various shortcuts for repeating elements inside a grammar. Marpa on the other hand needs a separate scanner to feed it with tokens and only allows very simple grammar rules. Parse::RecDescent combines parser code and grammar rules in one string. Marpa keeps the grammar clean and calls action methods based on rules defined inside the grammar.
After figuring out how to write grammars and having coded a simple tokenizer I finally got a parser working. Maybe, some rules might need expansion because of erroneous white-space handling or other flaws, but the result impressed me. Defining the grammar as a data structure makes it very good readable. Getting action methods called for processing the parse results keeps code separate from the grammar which keeps the entire code readable.
What are the next steps? Maybe a compressor or a sass-like processor? Maybe.
My quick and dirty try can be seen here.
]]>Denis Banovic started with a live demo demonstrating development and deployment of a dancer app in the "Cloud". It looked very easy to do.
Karl Gaissmaier was the next speaker explaining a lot of details about Captive::Portal, a hostspot software his university developed.
Stefan Hornburg talked about various advanced features of and his experiences with Dancer.
After lunch, Rolf Langsdorf held a very interesting talk comparing Perl and JavaScript. After that, Herbert Breunung had two talks: Thoughts about better documentation and a comparison of Hg and git.
And the last two speakers were Richard Lippmann discussing the usage of the right language for programming projects and Renée Bäcker with tips and tricks for doing talks.
All to gether, this was a great workshop. I will definitely attend next year's workshop which is in Erlangen from March 5 to March 7, 2012.
]]>Gerhard Schwanz presenting Mapweaver was the next speaker, directly followed by Denis Banovic presenting the simplicity of Dancer. For me, this talk was the best of today.
Steffen Ulrich gave an excellent introduction into TLS and explained some recently happening issues and how they could occur.
As a replacement for an omitted talk, Heiner Kuhlmann presented a tool to display some software quality metric values graphically. Nice to see Perl-Critic in colored boxes :-)
Herbert Breunung talked about some features of Perl 6. Whenever I hear him talking, I wish Perl 6 was already production-ready. All the new features are really awesome!
The last two speakers before the lightning talk session were Rolf Langsdorf showing a way of using macros to expand Perl 5 and Lars Dieckow who gave another SSL talk. The Macro-Talk was very impressive but I have to take a look into the sources in order to fully understand what happens under the hood.
As usual, the lightning talks went over too fast. The things I remember most were a new search engine operating under Perl: http://duckduckgo.com and the reminders to next year's German Perl Workshop in March and the YAPC Europen in August.
]]>Renée Bäcker demonstrated the use of Role::Basic for non-Moose environments. Looks very useful.
Rolf Langsdorf talked about possibilities to extend Perl's syntax using syntactic sugar and function prototypes. He tries to avoid source-filters and parsers in order to avoid breakage of external tools.
Uwe Voelker talked about HTML5::Sanitizer, a module he wrote to clean uploaded HTML code.
Max Maischein demonstrated Web Scraping techniques using FireFox to allow scraping websites that depend on JavaScript.
Looking forward to tomorrow's talks :-)
]]>Test::DBIx::Class. Scanning through the documentation was impossible -- it knows far too many options to get captured within a few minutes. But carefully reading the brilliant manual is worth the time spent. Test::DBIx::Class has a lot of options and test-functions tailored especially to the needs of DB-testing.
My primitive schema knows 3 tables: Sensors, Measures and AlarmConditions. Inside my t/ directory, I needed to create only one supplementary file to allow the tests to get coded: a config file t/etc/schema.pl
{
schema_class => 'Statistics::EasySchema',
force_drop_table => 1,
resultsets => [ qw(AlarmCondition Measure Sensor) ],
connect_info => ['DBI:Pg:dbname=statistics_test', 'user', 'secret'],
}
My tests started using a blank database which was OK for me in this particular situation with a primitive database like this.
My test-script partially looked like that:
use strict;
use warnings;
use Test::More;
use Test::DBIx::Class;
# ensure DB is empty
is Sensor->count, 0, 'no records in sensor table';
# add a sensor
ok my $sensor = Sensor->find_or_create({name => 'erlangen/keller/temperatur'}),
'find or create sensor';
is_result $sensor;
is_fields [qw(sensor_id name)], $sensor, [1, 'erlangen/keller/temperatur'],
'sensor fields look good';
is Sensor->count, 1, '1 record in sensor table';
# more test...
done_testing;
How easy things can be by simply using the right module :-)
]]>Everything looked obviously right, but after a while of debugging, we found out that the populate method of DBIx::Class::ResultSet behaves differently depending on the context used. In Scalar or List context, the records are inserted using the INSERT statement, while in void context, a bulk insert method is used.
From a performance standpoint, this absolutely makes sense. The documentation does explain this very well.
However, some optimizations have been made under the hood that are not obvious unless you know about -- I did not know that before.
My code looked like this:
MyApp->model('DB::Element')->populate([
{element_id => 113, name => 'f4'},
{element_id => 114, name => 'f5', usage_starts => '2000-04-01'},
]);
And the statement sent to the database was:
INSERT INTO element( element_id, name ) VALUES( ?, ? ) : '__BULK_INSERT__'
Every column which was not mentioned in the first record got lost and simply was ignored.
Conclusion? A bulk insert can only operate on a well known set of columns. The logic behind DBIx::Class retrieves the set of rows from the keys of the first hashref. This is important to keep in mind when using this syntax.
Solutions? TIMTOWTDI :-) If speed is not important, a populate method call could easily put into list context forcing the usage of single insert statements that do what you expect.
# force list context, ugly but usable :-)
() = MyApp->model('DB::Element')->populate([
# ...
]);
]]>
Especially when teaching newcomers how to use the CPAN this leaves a bad impression.
Experienced people quickly learn to skip those things but taking the archive more serious would be nice.
]]>