Perl in a Business Application

Perl in a Business Application - Musings of an Architect

Everybody knows that Perl is not the right language for a large scale enterprise application. This is common knowledge, right? But why is that? Explanations are as many as there are people explaining. Everything from "it's a script language, therefore slow" to "its free syntax breeds discoherence" to "Perl developers are horrible individualists".

Well, I didn't believe this, and I went on to help in a startup which wants to build some fintech systems, the first aim of which is to integrate with Finnish banks and collect daily payments from a customer's bank account.

It was decided to use Perl as the core language. If Perl is (was) good enough for Goldman Sachs and Morgan Stanley it surely is good enough for us. So off to build a framework!

Two Failed Attempts for System Architecture

We decided to do the web part with Dancer2 and build our own object system with mappers to read from and write to database and a clever filing metaphor with a class called BusinessObjectManager which creates, stores, restores and retires (removes) one object at a time. I had previously worked with a similar kind of metaphor in a C++ system. That was, of course, much more rigid, whereas with Perl we didn't seem to be able to create the safeguards we wanted to prevent future developers from abusing the system. The design just grew more complicated. I considered employing Moose but then instead took a whole different approach.

An approach familiar from Java web programming, Java internal services, which concentrates on business processes instead of business objects. I built an example service called SimulateBank with a simple structure: At the top was Dancer2 SimulateBank::Web package creating the WWW interface, this used SimulateBank::Client package which in turn called the JSON REST Api. Api was done with Dancer2 using SimulateBank::Api package which in turn used internally SimulateBank::Service which spoke to database directly. SimulateBank::Service implemented "services" like finddeposits, _createdeposit, _reportdeposit_ and canceldeposit_. These are processes which codify the business rules of the company. We do not care about objects, we only care about interfaces and processes.

This second approach was much more convenient and efficient. However, it still required code duplication and syncronization between Client and Service. What's more, I wasn't happy with it because it didn't feel "Perlish". I was resonably happy with the "super-structure" of the code, the division between Web and Api. But I felt I wasn't using the possibilities of Dancer2 framework with its plugins, like Dancer2::Plugin::Database and Dancer2::Plugin::Queue. I felt I was doing the same job twice when implementing my own interfaces to database and message queue instead of using the readily available Dancer2 facilities.

├── SimulateBank
│   ├── Api
│   │   └── Transactions.pm
│   ├── Client
│   │   └── Transactions.pm
│   ├── Common.pm
│   ├── Service
│   │   └── Transactions.pm
│   └── Web
│       └── Transactions.pm
└── SimulateBank.pm

And then it struct me! I had looked at the whole problem from the wrong angle. My approach was code first. I tried to create a perfect structure for the future Perl programmers to use.

Towards a Perlish Approach

Perlish approach is result oriented. After all, do we not pride ourselves on using a language which is fast to program with? Software only matters if it's put into production. But what about all that horrible Perlish hacking, the quick-and-dirty way?

That is the Perl way! To create a working solution today. Not to worry about tomorrow.

In my experience the most far reaching problems in software development are not done by programmers, or they are done by programmers when they have been forced into roles that should be done by others, such as database designer, integration architect and user interface designer. These are the people who should do the worrying. They are paid to design long-term solutions!

Business App Blues

The purpose of most enterprise applications is to collect data, and then deliver or distribute it, or act upon it. So is ours. From the very beginning we started to plan our datamodels and the resulting database schema meticulously. For instance, the application has only SELECT and INSERT access to many of our tables to prevent the loss of past state information. The whole schema has only one sequence and its value is inserted in every table so that the whole flow and order of operations in database is trackable.

Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.

Fred Brooks, The Mythical Man-Month: Essays on Software Engineering (1975, 1995), https://en.wikiquote.org/wiki/Fred_Brooks

With Fred Brooks' quote in mind, I approached again the issue of code first. Code could not come first. Data and datamodels came first. Code second. In fact, code comes third, because second place is given to apis, most notably the REST apis created with Dancer2.

Interfaces and Silos

Database schemas and system apis are both interfaces to data. They are the longest lasting parts of the system - and the ones that are the most difficult to change later. Code isn't. It can always be refactored and improved and tested against the unchanging interfaces.

If the interfaces are locked, especially the database schema, and we can be 99% sure that our data is always protected from a malfunctioning program, then it's time to give the programmer free hands to create the best code he can.

Furthermore, the future of our application is in constant change - like in many startups. Microservices is a natural way to extend this way of thinking. Different parts of the system become microservices and silos whose implementation code is their private part. This code can be quickly changed and it must have no connection to any other silos' code. This allows very radical changes if need be, such as web programming frameworks, math packages or even Perl interpreter version.

And what's best, none of the changes in the code threaten the stability of the whole system. Code quality becomes a matter of code reviews. Our backs (interfaces) covered, coding with Perl is fast and fun, because it just works (TM).

The Perl Way

There were several times when we were second guessing our decision to use Perl. This whole story happened in the course of one year's time. I consider myself lucky to have been given the chance to go through that whole mental process. I believe I understand Perl a lot better now - not perhaps as language but as a way of seeing software development and organizing development projects.

The first version was indeed only good for throwing away like Brooks writes in The Mythical Man-Month. We saved some parts and also some ideas from the second version. Speed is of the essence. The internal services for accessing database are mostly skipped and Dancer2 database plugin is used to fetch the data directly. Most action happens right in the same package where there the Dancer2 REST interface endpoints are defined because in most cases the data fetched from or written to database requires no additional handling. So there is no need to create additional layers, especially when the rigid database schema assures that fetched data is always sound (no nulls, no missing values or missing foreign fields). While quality control was earlier exercised only via code reviews, those are now complemented with api tests and rigid database schema modelling.

[Software engineering is the] establishment and use of sound engineering principles to obtain economically software that is reliable and works on real machines efficiently.

Friedrich Bauer (1972) "Software Engineering", In: Information Processing. p. 71

Git::MoreHooks::CheckIndent - Force accurate indentation in Git commits

Git::MoreHooks::CheckIndent Released

Git::MoreHooks::CheckIndent is a plugin for the Git::Hooks framework. It checks that indentation is correct in the committed files. Please read Git::Hooks documentation on how to use plugins.

Stop the press! Git Already has …

Yes, it does indeed! The not-so-well-known Git configuration parameter core.whitespace is quite versatile in this respect. It supports different values, even tabwidth to tell how many space characters an indentation (tab) must be equal to.

core.whitespace : A comma separated list of common whitespace problems to notice. git diff will use color.diff.whitespace to highlight them, and git apply --whitespace=error will consider them as errors. You can use prefix “-” to disable any of them (e.g. -trailing-space):

  • blank-at-eol treats trailing whitespaces at the end of the line as an error (enabled by default).
  • space-before-tab treats a space character that appears immediately before a tab character in the initial indent part of the line as an error (enabled by default).
  • indent-with-non-tab treats a line that is indented with space characters instead of the equivalent tabs as an error (not enabled by default).
  • tab-in-indent treats a tab character in the initial indent part of the line as an error (not enabled by default).
  • blank-at-eof treats blank lines added at the end of file as an error (enabled by default).
  • trailing-space is a short-hand to cover both blank-at-eol and blank-at-eof.
  • cr-at-eol treats a carriage-return at the end of line as part of the line terminator, i.e. with it, trailing-space does not trigger if the character before such a carriage-return is not a whitespace (not enabled by default).
  • tabwidth= tells how many character positions a tab occupies; this is relevant for indent-with-non-tab and when Git fixes tab-in-indent errors. The default tab width is 8. Allowed values are 1 to 63.

CheckIndent options

Git core.whitespace doesn’t allow fine-tuning. If your repository has several different policies active for different files, then you are out of luck. Unless you use CheckIndent.

CheckIndent has only two configuration options, file and exception.

githooks.checkindent.file File Type Specific.

A regular expression matches against the file name. If config has several file items they are used in their order of appearance until a match is found. When a match is found, the parameters are applied to check the file.

Parameters are indent-char (allowed values: space, tab, both) and indent-size (allowed content: an integer number).

[githooks "checkindent"]
    file = ^proj1/old/.* indent-char:both indent-size:2
    file = \\.(c|h|cpp|hpp)$ indent-char:tab
    file = \\.py$ indent-char:space indent-size:4

Every file in directory :prog1/old support indentation with both space and tab characters, but indent size is only 2. If file name matches this regexp (= is located in directory prog1/old), then the following lookups are skipped. If file is located somewhere else and is a C/C++ file, then tab is the only allowed indent character. If space is not allowed, then parameter indent-size is meaningless and skipped over. Python files however get the opposite treatment: only space is allowed and indent size is strictly 4 spaces.

githooks.checkindent.exception Exceptions to the rule

After finding the faulty lines ever line is matched against the exceptions. If a match is found, then we skip the error. The parameter exception consists of two parts. The first part is the same regular expression as in file. The second is another regexp. This regexp defines the allowed exceptions.

[githooks "checkindent"]
    file = \\.(c|h|cpp|hpp)$ indent-char:tab
    exception = \\.(?:c|h|cpp|hpp)$ ^[[:space:]]{0,}[\\s]{1}\\*

In the above example the C/C++ files allow a comment to be indented with one extra whitespace. This is according to the Doxygen comment pattern.

[ This entry simultaneously published at author’s blog Exercises in Restful Integration and Continuous Delivery. ]

IO::Iron - Towards Update to Version 3

IO::Iron::Applications - Command line tools for Iron.io services

IO::Iron::Applications is an auxiliary package for IO::Iron. IO::Iron contains the library for using the Iron.io cloud services in Perl programs. IO::Iron::Applications contains command line programs to operate those services.

IO::Iron::Applications is my addition to the IO::Iron interphase library package which I wrote earlier. The Iron.io WWW interface hud.iron.io Dashboard

Dist::Zilla as a Continuous Delivery Tool

I just recently converted my IO::Iron distribution to using Dist::Dilla as a release and build automation tool. Dist::Dilla is mainly targeted at people writing free software Perl packages for releasing into CPAN (Perl free software archive) but if used properly it can make easier the releasing of any software.

Before