Managing Myriad Modules

Modules mentioned:

Config::IniFiles
Config::Tiny
Config::Tiny::Ordered
DBIx::Admin::CreateTable
DBIx::Class
DBIx::Connector
File::HomeDir
Log::Dispatch
Log::Handler
Module::Build
Module::Metadata::Changes
MooseX::LogDispatchh
Plack
Perl::PrereqScanner
Role::Basic

Over the years I written quite a few modules, some destined for CPAN,
some for customers, and others - Local::* - for my own use.

For some days now I've been cleaning up the latter group, and I thought
I'd blog about the decisions involved.

o Config files

Firstly, I've kept the convention of starting all config file names with
'.ht', to make it easy of block downloads of such files, via various web
server options.

But I've changed my convention about where within a distro's directory
structure to ship such files. Originally if, say, a module was called
Local::Wines, then I'd use lib/Local/Wines/.htlocal.wines.conf.

A module, Local::Wines::Util::Config was dedicated to reading this
file, and performing some basic validation on its contents.

But now, a digression on the structure of such config files.

I decided all such files would be read by Config::Tiny and be in 3
sections, so they look like:

[global]
host = localhost
[localhost]
# Local web site options (i.e. on my PC).
[webhost]
# Remote web site options.


There are times when I really wish Config::Tiny handled nested options,
but I do without them.

In special cases, e.g. for Module::Metadata::Changes
which manages machine-readable CHANGES files (called Changelog.ini),
I used Config::IniFiles
but otherwise I stick to Config::Tiny.

Actually, for one customer I had to preserve the order of options within
a section, so I released Config::Tiny::Ordered.

But back to my standard format. Validation just ensures [global] is
present, and that the 'host' option points to another section which
also exists.

But since almost every project I work on these days is for a CGI
script with a database back end, the number of config files seems to
never stop growing.

So, I re-wrote Local::Config to perform its original task, and the
read any other module's config file too.

.htlocal.config.conf just specifies boring things like my web server's
doc root, my Perl modules directory, and so on.

But now I can use

my($config) = Local::Config -> new -> config;


to return a hashref of options from .htlocal.config.conf, and I can also
use

my($module_config) = Local::Config -> new
(
config_name => '.htlocal.wines.conf',
module_name => 'Local::Wines',
) -> get_config;


to get the config file belonging to any other module.

This means I can now zap all modules called Local::*::Util::Config.

I've already put the config file names into .htlocal.config.conf,
and I'll next drop the config_name => '...' parameter.

Back to the discussion of shipping such files. I chose config/
within each distro's directory.

The original location lib/Local/Wines/.htlocal.wines.conf
had the problem that's you'd need to have write access to the
directory where modules are installed if you wished to patch
the default config file. So how do we solve that problem?

Enter File::HomeDir.
By shipping a little script with each module, scripts/copy.config.pl,
I make it easy to copy each config file to a directory such as -
for Local::Wines - ~/.perl/Local-Wines. This is what File::HomeDir
under Debian returns for calls to my_dist_config().

To summarize:

- .ht*.conf to name such files.
- Config::Tiny to read such files.
- Local::Config to wrap Config::Tiny, with validation.
- File::HomeDir to specify where to copy them to.
- scripts/copy.config.pl to do the copy.
- All Local::* modules to share Local::Config.

o Logging

This can be a complex issue, but shouldn't be. For a while I've been
using MooseX::LogDispatch
which is an interface to the wonderful Log::Dispatch

Recently (for me), the lightning-fast Log::Handler
appeared, so I decided to zap all my wrappers for MooseX::LogDispatch,
which I had called Local::*::Util::LogConfig, and to use one generic
wrapper, Local::Logger.

It takes a config file of course, so each project can log to its own
database, since almost every config file contains a dsn, username and
password.

I was tempted to use Role::Basic
at this point, but since each logger has a module-specific config file
I decided to skip that at the moment.

To summarize:

- Standardize on Log::Handler::Output::DBI.
- Local::Config to supply the config options.
- Local::Logger to wrap Log::Handler's code.
- All Local::* modules to share Local::Logger.

Note: Since each logger has its own connexion to the database, it's
vital that CGI scripts disconnect explicitly, especially when running
under persistent environments such as Plack
If you're getting the dreaded (Postgres) error message
Connexion limit exceeded for non-super-users, that could be why.

o Build.PL

Recent changes to Module::Build
meant (grrrr) I had to patch all my Build.PL files, to add

configure_requires =>
{
'Module::Build' => 0.38,
},


so that was done too. I have not yet done this for modules shipped
to CPAN, that'll come later.

Also, I use Perl::PrereqScanner to see what modules I use, and a little script to compare that with what's mentioned in Build.PL. That produced a few surprises, too.

o DBIx::Connector

I've adopted DBI::Class
as the manager of database handles.

Note: Since each connector has its own connexion to the database, as
with loggers, it's vital that CGI scripts disconnect explicitly,
especially when running under persistent environments such as Plack.

o DBIx::Class

I've tried many, too many, Perl modules to manage the Perl-Database
server interface, but now I've standardized on DBI::Class

Nevertheless, I want to be able to bootstrap a new project, and to
drop and re-create all tables, without having to need DBI::Class.
How to do that? It took me several steps.

1) I used to ship Local::*::Util::Create, which contained all the code
to drop, create and populate tables.

I've abandoned that approach in favour of Local::*::Database,
Local::*::Database/Create (drop, create) together with
Local::*::Database::Import and Local::*::Database::Export.

The trick is to write Local::*::Database::Create without relying on
DBIx::Class. Then 2 tiny scripts, drop.tables.pl and
create.tables.pl, shipped in scripts/, can get me started.

With tables created, I can run yet another tiny script,
scripts/generate.schema.classes.pl, which outputs a set of files
in Local::*::Schema.pm and Local::*::Schema/*.

Now scripts/populate.tables.pl (for constant tables) can use
DBIx::Class, and Local::*::Database::(Export, Import) can too.

Lastly, the main application can utilize DBIx::Class. It's
just a matter of deciding how to partition the attributes
among the modules, to facilitate the goal of drop/create without
using DBIx::Class, and all other code having it available.

Here's how I do it:


package Local::Wines::Base;

use strict;
use warnings;

use DBIx::Connector;

use Local::Config;
use Local::Logger;

use Moose;

has config =>
(
default => sub{ return Local::Config -> new(module_name => 'Local::Wines') -> get_config },
is => 'rw',
isa => 'HashRef',
required => 0,
);

has connector =>
(
is => 'rw',
isa => 'Any',
required => 0,
);

has logger =>
(
is => 'rw',
isa => 'Local::Logger',
required => 0,
);

use namespace::autoclean;

our $VERSION = '1.19';

# -----------------------------------------------

sub BUILD
{
my($self) = @_;
my($config) = $self -> config;
my($attr) = {AutoCommit => $$config{AutoCommit}, RaiseError => $$config{RaiseError} };

if ( ($$config{dsn} =~ /SQLite/i) && $$config{sqlite_unicode})
{
$$attr{sqlite_unicode} = 1;
}

$self -> connector
(
DBIx::Connector -> new($$config{dsn}, $$config{username}, $$config{password}, $attr)
);

if ($$config{dsn} =~ /SQLite/i)
{
$self -> connector -> dbh -> do('PRAGMA foreign_keys = ON');
}

$self -> logger
(
Local::Logger -> new(config => $config)
);

} # End of BUILD.

# -----------------------------------------------

__PACKAGE__ -> meta -> make_immutable;

1;


Now we can incoporate that into Local::Wines::Database::Create like so:


package Local::Wines::Database::Create;

use strict;
use warnings;

use DBIx::Admin::CreateTable;

use Moose;

extends 'Local::Wines::Base';

has creator =>
(
is => 'rw',
isa => 'DBIx::Admin::CreateTable',
required => 0,
);

has engine =>
(
is => 'rw',
isa => 'Str',
required => 0,
);

has time_option =>
(
is => 'rw',
isa => 'Str',
required => 0,
);

use namespace::autoclean;

our $VERSION = '1.19';

# -----------------------------------------------

sub BUILD
{
my($self) = @_;

$self -> creator
(
DBIx::Admin::CreateTable -> new
(
dbh => $self -> connector -> dbh,
verbose => 0,
)
);

$self -> engine
(
$self -> creator -> db_vendor =~ /(?:Mysql)/i ? 'engine=innodb' : ''
);

$self -> time_option
(
$self -> creator -> db_vendor =~ /(?:MySQL|Postgres)/i ? '(0) without time zone' : ''
);

} # End of BUILD.

# -----------------------------------------------

sub create_all_tables
{
my($self) = @_;

# Warning: The order is important.

my($method);
my($table_name);

for $table_name (qw/
log
sessions
grapes
styles
vineyards
wine_makers
wines
/)
{
$method = "create_${table_name}_table";

$self -> $method;
}

return 0;

} # End of create_all_tables.

# --------------------------------------------------

sub create_grapes_table
{
my($self) = @_;
my($table_name) = 'grapes';
my($primary_key) = $self -> creator -> generate_primary_key_sql($table_name);
my($engine) = $self -> engine;
my($result) = $self -> creator -> create_table(< create table $table_name
(
id $primary_key,
name varchar(255) not null,
upper_name varchar(255) not null
) $engine
SQL
$self -> report($table_name, 'created', $result);

} # End of create_grapes_table.

# --------------------------------------------------

sub create_log_table
{
my($self) = @_;
my($table_name) = 'log';
my($primary_key) = $self -> creator -> generate_primary_key_sql($table_name);
my($engine) = $self -> engine;
my($time_option) = $self -> time_option;
my($result) = $self -> creator -> create_table(< create table $table_name
(
id $primary_key,
level varchar(9) not null,
message varchar(255) not null,
timestamp timestamp $time_option not null default current_timestamp
) $engine
SQL
$self -> report($table_name, 'created', $result);

} # End of create_log_table.

# Etc, etc.

# -----------------------------------------------

sub drop_all_tables
{
my($self) = @_;

my($table_name);

for $table_name (qw/
wines
wine_makers
vineyards
styles
grapes
sessions
log
/)
{
$self -> drop_table($table_name);
}

return 0;

} # End of drop_all_tables.

# -----------------------------------------------

sub drop_table
{
my($self, $table_name) = @_;

$self -> creator -> drop_table($table_name);

} # End of drop_table.

# -----------------------------------------------

sub report
{
my($self, $table_name, $message, $result) = @_;

if ($result)
{
die "Table '$table_name' $result. \n";
}
else
{
$self -> logger -> log(debug => "Table '$table_name' $message");
}

} # End of report.

# -----------------------------------------------

__PACKAGE__ -> meta -> make_immutable;

1;


These 2 modules become the pattern for the code used in all modules
Local::*::Database::Base and Local::*::Database::Create.

Our next step in arranging for code which does in fact use DBIx::Class
to use Local::Wines::Base.

Firstly, we run scripts/generate.schema.classes.pl:


#!/usr/bin/env perl

use strict;
use warnings;

use DBIx::Class::Schema::Loader 'make_schema_at';

use Local::Config;

# -----------------------------------------------

my($config) = Local::Config -> new(module_name => 'Local::Wines') -> get_config;

make_schema_at
(
'Local::Wines::Schema',
{
dump_directory => './lib',
},
[
$$config{dsn}, $$config{username}, $$config{password},
],
);


Then, this bit is easy:


package Local::Wines::Database::Base;

use strict;
use warnings;

use Local::Wines::Database;
use Local::Wines::Schema;

use Moose;

extends 'Local::Wines::Base';

has db =>
(
is => 'rw',
isa => 'Local::Wines::Database',
required => 0,
);

has schema =>
(
is => 'rw',
isa => 'Local::Wines::Schema',
required => 0,
);

use namespace::autoclean;

our $VERSION = '1.19';

# -----------------------------------------------

sub BUILD
{
my($self) = @_;

$self -> schema
(
Local::Wines::Schema -> connect(sub{return $self -> connector -> dbh})
);

} # End of BUILD.

# -----------------------------------------------

__PACKAGE__ -> meta -> make_immutable;

1;


I.e. Any code wanting access to the db and the schema just includes:


...

use Moose;

extends 'Local::Wines::Database::Base';

...


But, I hear you say, db was not initialized. Err, correct.

Let's look at scripts/export.as.csv.pl:


#!/usr/bin/env perl

use strict;
use warnings;

use Local::Wines::Database::Export;

# -------------------------------

print Local::Wines::Database::Export -> new -> as_csv;


and Local::Wines::Database::Export:


package Local::Wines::Database::Export;

use CGI;

use Local::Wines::Database;

use Moose;

use Text::CSV_XS;

use Text::Xslate;

extends 'Local::Wines::Database::Base';

has whole_page =>
(
default => 0,
is => 'rw',
isa => 'Any',
required => 0,
);

use namespace::autoclean;

our $VERSION = '1.19';

# -----------------------------------------------

sub BUILD
{
my($self) = @_;

$self -> db
(
Local::Wines::Database -> new
(
query => CGI -> new,
)
);

} # End of BUILD.

# -----------------------------------------------

sub as_csv
{
my($self) = @_;

my(@row);

push @row,
[
'wine_maker', 'vineyard', 'style', 'grape', 'comment', 'vintage', 'rating', 'review_date',
];

for my $wine (@{$self -> read_wines_table})
{
push @row,
[
$$wine{wine_maker},
$$wine{vineyard},
$$wine{style},
$$wine{grape},
$$wine{comment},
$$wine{vintage},
$$wine{rating},
$$wine{review_date},
];
}

for (@row)
{
print '"', join('","', @$_), '"', "\n";
}

} # End of as_csv.

# -----------------------------------------------

sub as_html
{
my($self) = @_;

my(@row);

push @row,
[
{td => 'Wine maker'},
{td => 'Vineyard'},
{td => 'Style'},
{td => 'Grape'},
{td => 'Comment'},
{td => 'Vintage'},
{td => 'Rating'},
{td => 'Reviewed'},
];

my($wine);

for $wine (@{$self -> read_wines_table})
{
push @row,
[
{td => $$wine{wine_maker} },
{td => $$wine{vineyard} },
{td => $$wine{style} },
{td => $$wine{grape} },
{td => $$wine{comment} },
{td => $$wine{vintage} },
{td => $$wine{rating} },
{td => $$wine{review_date} },
];
}

push @row,
[
{td => 'Wine maker'},
{td => 'Vineyard'},
{td => 'Style'},
{td => 'Grape'},
{td => 'Comment'},
{td => 'Vintage'},
{td => 'Rating'},
{td => 'Reviewed'},
];

my($tx) = Text::Xslate -> new
(
input_layer => '',
path => ${$self -> config}{template_path},
);

return $tx -> render
(
$self -> whole_page ? 'whole.page.tx' : 'basic.table.tx',
{
row => \@row
}
);

} # End of as_html.

# -----------------------------------------------

sub read_wines_table
{
my($self) = @_;
my($rs) = $self -> schema -> resultset('Wine');

my($date);
my(@wine);

while (my $result = $rs -> next)
{
$date = substr($result -> review_date, 0, 10);

push @wine,
{
comment => $result -> comment || '',
grape => $result -> grape -> name,
rating => $result -> rating,
review_date => $date eq '1900-01-01' ? '-' : $date,
style => $result -> style -> name || '',
vineyard => $result -> vineyard -> name,
vintage => $result -> vintage,
wine_maker => $result -> wine_maker -> name,
};
}

return [sort{$$a{wine_maker} cmp $$b{wine_maker} || $$a{style} cmp $$b{style} || $$a{grape} cmp $$b{grape} } @wine];

} # End of read_wines_table.

# -----------------------------------------------

__PACKAGE__ -> meta -> make_immutable;

1;

Phew! That's enough for today.

Leave a comment

About Ron Savage

user-pic I try to write all code in Perl, but find I end up writing in bash, CSS, HTML, JS, and SQL, and doing database design, just to get anything done...