Building a Geo-Coding Database and Website

OK, so this is my first ever public blog, so be patient with me.

I've written a few Perl related Genealogy programs including gedcom (https://github.com/nigelhorne/gedcom) and ged2site (https://github.com/nigelhorne/ged2site). One of the things that these do is to check the validity of your family tree, and one of those tasks is to verify place-names. Of course places do change names and spelling becomes more consistent over the years, but the vast majority remain the same. Enough of a majority to computerise the verification. Unfortunately all of the on-line services have one problem or another - most either charge for large number of access, or throttle the number of look-ups. Even my modest tree, just over 2000 people, reaches those limits.

Why Programmers Use the Test Hierarchy Antipattern

The first part of this series described Test Hierarchy, a hierarchy of test classes that mirrors the classes under test, and explained why it’s an antipattern. Part two explored what makes a good unit test and why Test Hierarchy does not. This third and final post reflects on why programmers use Test Hierarchy and why these reasons aren’t persuasive.

Baby Moose struts his stuff

Well is still container day here in the Moose-pen.

So now that I have the 'execute_array' all fixed up and working I think I will carry on the same past and get the container to work with an array of 'classes'. In that end I created this little class;

package Test::User;
use Moose;
has username => ( is  => 'rw',
               isa => 'Str',);
has address => ( is  => 'rw',
               isa => 'Str',);
1;
Now the next thing I needed to do was make a test out of what I fixed yesterday;

$container = [{username=>'Bill',address =>'ABC'},
              {username=>'Jane',address =>'DEF'},
              {username=>'John',address =>'HIJ'},
              {username=>'Joe',address =>'KLM'},
              ];
ok($user->create( $utils->connect(),$container),"Execute Array add 4");
unless($user->result()->is_error) {
  ok(scalar(@{$user->result()->set}) == 4,"Four recodes added");
}else{
   fail("Execute Array failed");
}
Now I add in the code to create all my classes;

use Test::User;
my $class_container = [];
foreach my $item (@{$container}){
   push(@{$class_container},Test::User->new($item ));
}

What is a Bool?

Perl allows pretty much any value to be evaluated in a boolean context:

if ($something) {
   ...
}

No matter what $something is, it will safely evaluate to either true or false. (With the exceptions of a few edge cases like blessed objects which are overloaded to throw an error when evaluated as booleans.)

So when a Moose class does something like this, what does it mean?

has something => (
   is  => 'ro',
   isa => 'Bool',
);

Yes Baby Moose had a Bug.

Well it is look at another module day here in the Moose-Pen.

So yesterday I ran into this problem;
DBD::DBM::st execute_array failed: 4 bind values supplied but 2 expected [for Statement "INSERT INTO user ( user.address, user.username ) VALUES( ?, ? )"] at …
Which rather peeved me as I tried the same code on two other DBs and the both ran. So it it just me? I took a look at example in the DBI POD;

$dbh->{RaiseError} = 1;        # save having to check each method call
$sth = $dbh->prepare("INSERT INTO staff (first_name, last_name, dept) VALUES(?, ?, ?)");
$sth->bind_param_array(1, [ 'John', 'Mary', 'Tim' ]);
$sth->bind_param_array(2, [ 'Booth', 'Todd', 'Robinson' ]);
$sth->bind_param_array(3, "SALES"); # scalar will be reused for each row
$sth->execute_array( { ArrayTupleStatus => \my @tuple_status } );
unfortunately with DBD::DBM I only have two fields so I had to adapted the above code to this;

Testing Insights from B::DeparseTree

rockyb’s recent post about B::DeparseTree contained several insights on testability and writing good tests. Here are my takeaways.

Why is parsing considered solved?

"Why is parsing considered solved?" is the newest entry on my Ocean of Awareness blog.

It is often said that parsing is a "solved problem". Given the level of frustration with the state of the art, the underuse of the very powerful technique of Language-Oriented Programming due to problematic tools, and the vast superiority of human parsing ability over computers, this requires explanation.
On what grounds would someone say that parsing is "solved"? To understand this, we need to look at the history of Parsing Theory. In fact, we'll have to start decades before computer Parsing Theory exists, with a now nearly-extinct school of linguistics, and its desire to put the field on strictly scientific basis.

Update to XPath Sandbox

As part of my project to create a tutorial for XML::LibXML, I created an XPath Sandbox tool that allows you to try out different XPath expressions directly in your browser. I've recently enhanced that tool to add a couple of useful features:

  • 'Upload'* your own XML files, and query them
  • Namespace support, including registering your own prefix mappings

When working with the built-in sample files, URL parameters can be used to: select a file, specify an XPath expression, override the default namespace prefix mappings. Here's an example link that does all three!

* I used the term upload in 'scare quotes' because it's a client-side app, nothing actually gets sent to the server.

Baby Moose Has a Bug?

Well it get lucky postette day here in the Moose-Pen.

Well I left off yesterday a little worried that I was only testing half of the execute_array as I never actually got down into the DBI code of things. So I decided lets give it a try on that pre-installed 'DBD::DBM' that I used in my '10_crud_basic.t' test case as according to the DBI docs there shold be a defult implimentsion of 'execute_array''

Well I opened that sucker up and added this in

$container = [{username=>'Bill',address =>'ABC'},
              {username=>'Jane',address =>'DEF'},
              {username=>'John',address =>'HIJ'},
              {username=>'Joe',address =>'KLM'},
              ];
$user->create( $utils->connect(),$container);
and then got

DBD::DBM::st execute_array failed: 4 bind values supplied but 2 expected [for Statement "INSERT INTO user ( user.address, user.username ) VALUES( ?, ? )"] at D:\GitHub\database-accessor-driver-dbi\lib/Database/Accessor/Driver/DBI.pm line 99.
hmm perhaps things will work out as it look like I get into my DBI code and the SQL;

An Array of Baby Moose

Its a little harder day here in the Moose-Pen.

Carrying on with adding to the '32_params.t' case I though today I would tackle one of the harder parts of prams and that is passing an array of params or in my case containers into a query statement.

So in my test I could have this;

$container = [{first_name=>'Bill',last_name =>'Bloggings'},
              {first_name=>'Jane',last_name =>'Doe'},
              {first_name=>'John',last_name =>'Doe'},
              {first_name=>'Joe',last_name =>'Blow'},
              ];
and what I wold expect to happen on a create would be that all a new recoded would be added for each. In DBI has the 'execute_array' which works with 'bind_param_array' to effect an SQL action on such a set of tuples. DBI works on the above by either the default implimetaion of simple iteration over each other DBDs such ad DBD::Oracle have built in API for such actions.

As MongoDB (and other Key-pair Dbs) has a similar function 'updateMany' adding support for this is well with-in the scope of my project.

Next the test;

The Perl Conference 2018 Newsletter: 06/02/2018

In This Issue:

Why May 2018 is so special?

As we entered the sixth month of the year 2018. So what have I achieved in May 2018? In short, plenty. Let me share the details.

May 2018 has been the best month so far in the year 2018. In this month, I submitted 60 Pull Requests. Only two occasions in the past where I had better number than May 2018. It was 77 Pull Requests in December 2016 and 63 Pull Requests in January 2017.

As of today, 2nd June 2018, I have submitted 858 Pull Requests. Of those 575 Pull Requests have been merged successfully. I am hoping to get to the magic number 1000 before the next London Perl Workshop, which is the 3rd Nov 2018. I am keeping my fingers crossed.

Rewriting B:Deparse and Reintroducing B::DeparseTree and (part 1)

I will be giving a talk on B::DeparseTree and its use in a debugger Devel::Trepan at the upcoming YAPC 2018 in Glasgow. As a result, I have been completely refactoring B::DeparseTree and have a number of thoughts on it and B::Deparse.

Here are some of them. This first part focuses more about B::Deparse and how it could be (or in a sense is) being rewritten. The second part if I get around to writing it will be about the cool features of B::DeparseTree from an application.

Introduction

As someone who make a lot of mistakes, I’ve long wanted to improve the precision of debugging and error reporting. Through Perlmonks, I was directed upon the idea of using the OP address as means to get more detailed information of where the program is. I was also pointed to B::Deparse.

Baby Moose Makes a Stand

Its fix typo and a little programming day here in the Moose-Pen.

Before I get back on track and start working on test case '32_params.t' but before that I had to clean-up in my code. Seems I had 'parenthes' all over the place when it should really be 'parentheses' so the last thin I did last night was clean all those up. Thank goodness for Padre and it search and replace functions.

Anyway on to '32_params.t' and the fist thing I did was cut out some tests from other places and put them in there as it made a little more logical sense. From 20_where_basic.t I striped out all the tests that check params and ended with these tests;

Baby Moose Skips Along

Its think of something, look it up and skip it day here in the Moost-pen.

Yesterday I added in the first option for Accessor.pm 'only_elements' and that got me thinking, never good ever comes out of that, I should enforce the rule that 'only_element' option has to be a hash-ref.

So I quickly looked-up a MooseX that I know will let me do that 'MooseX::Params::Validate' and reading a little from Dave's POD I should skip this module and have a look at MooseX::Method::Signatures or MooseX::Declare. Now looking at both of these I see a great big '(DEPRICATED)' on each and this waring in MooseX::Declare;

Warning: MooseX::Declare is based on Devel::Declare, a giant bag of crack originally implemented by mst with the goal of upsetting the perl core developers so much by its very existence that they implemented proper keyword handling in the core.

Baby Moose Does More

Its wrap up a test day here in the Moose-pen;

I am going to finish off '15_alias.t' today by adding in a few more test looking at the field 'alias' and here are the tests;

$in_hash->{elements}->[0]->{alias} = 'last';
$in_hash->{elements}->[1]->{alias} = 'first';
$da  = Database::Accessor->new($in_hash);
$da->create( $utils->connect(),$container);
ok($da->result()->query() eq "INSERT INTO people sys_users ( sys_users.first_name ) VALUES( ? )","create SQL correct");
$da->retrieve( $utils->connect());
ok($da->result()->query() eq "SELECT users.last_name AS last, sys_users.first_name AS first FROM people sys_users","retrieve SQL correct");
$da->update( $utils->connect(),$container);
ok($da->result()->query() eq "UPDATE people sys_users SET sys_users.first_name = ?","update SQL correct");
and surprise surprise they all run correctly. So this will could be a very short post today. Well maybe not.

Me thinks I will work on doing a little planing of what I will test next. The various fields/elements items look like a good candidate and I could load them all into the '30_fields.t' case.

Test Hierarchy Produces Poor Unit Tests

The first part of this series described Test Hierarchy, a hierarchy of test classes that mirrors the classes under test, and explained why it’s an antipattern. For how common it is, this practice doesn’t even produce good unit tests.

New release of Perl::Build

Perl::Build is a Perl Builder created by tokuhirom. You can build and install Perl by:

$ curl -L https://raw.githubusercontent.com/tokuhirom/Perl-Build/master/perl-build | perl - 5.26.2 /opt/perl-5.26/

Also, it is the backend of plenv-install:

$ git clone https://github.com/tokuhirom/Perl-Build.git $(plenv root)/plugins/perl-build
$ plenv install 5.26.2

Recently I became its maintainer, and have released a new version to CPAN. The new version contains the following changes:

Use MetaCPAN API

#66, #67, #73, #74, #75, thanks Grinnz, anttilinno, djzort, sjn.

Now Perl::Build uses fastapi.metacpan.org to find available Perl versions and Perl tarball URLs. MetaCPAN indexer is quite fast. So, as soon as a new Perl is released, you should be able to install it by Perl::Build.

Use HTTP::Tinyish for https support

#72, thanks AnaTofuZ.

Baby Moose Yummy

It is still alias day here in the Moose-pen

So it is day two of looking at aliases and SQL and as part of this I had some time last night to have a peek about the SQL standard. So far I am doing things correctly, for table and field. Though in my first incarnation of table alias I used the 'AS' keyword in there, if you look at some of the check-in history you will see it. This 'AS' will work in a few SQL engines but not all so I was correct in dropping it out.

From my tests of yesterday’s post and last night's reading I got thinking again on the promise that Database::Accessor is making to a DAD;
'Whatever is passed into a DAD will be as valid as possible'.
What I have to look at now is one of the system rules that I want to enforce; Only 'elements' that match the name or alias of the current 'view' are passed into the DAD

Is language just a set of strings?

The newest entry on my Ocean of Awareness blog: "Is language just a set of strings?"

"The languages human beings use with each other are powerful, varied, flexible and endlessly retargetable. The parsers we use to communicate with computers are restrictive, repetitive in form, difficult to reprogram, and prohibitively hard to retarget. Is this because humans have a preternatural language ability?
"Or is there something wrong with the way we go about talking to computers? How the Theory of Parsing literature defines the term "language" may seem of only pedantic interest. But I will argue that it is a mistake which has everything to do with the limits of modern computer languages."

About blogs.perl.org

blogs.perl.org is a common blogging platform for the Perl community. Written in Perl and offering the modern features you’ve come to expect in blog platforms, the site is hosted by Dave Cross and Aaron Crane, with a design donated by Six Apart, Ltd.