Perl in a Business Application - Musings of an Architect
Everybody knows that Perl is not the right language for a large scale
enterprise application. This is common knowledge, right?
But why is that? Explanations are as many as there are people explaining.
Everything from "it's a script language, therefore slow" to
"its free syntax breeds discoherence" to
"Perl developers are horrible individualists".
Well, I didn't believe this, and I went on to help in a startup which wants
to build some fintech systems, the first aim of which is to integrate with
Finnish banks and collect daily payments from a customer's bank account.
It was decided to use Perl as the core language. If Perl is (was) good enough
for Goldman Sachs and Morgan Stanley it surely is good enough for us.
So off to build a framework!
Two Failed Attempts for System Architecture
We decided to do the web part with Dancer2
and build our own object system with mappers to read from and write to database
and a clever filing metaphor with a class called BusinessObjectManager
which creates, stores, restores and retires (removes) one object at a time.
I had previously worked with a similar kind of metaphor in a C++ system.
That was, of course, much more rigid, whereas with Perl we didn't seem
to be able to create the safeguards we wanted to prevent future developers
from abusing the system. The design just grew more complicated. I considered
employing Moose but then instead took a whole different approach.
An approach familiar from Java web programming, Java internal services,
which concentrates on business processes instead of business objects.
I built an example service called SimulateBank with a simple structure:
At the top was Dancer2 SimulateBank::Web package creating the WWW interface,
this used SimulateBank::Client package which in turn
called the JSON REST Api. Api was done with Dancer2 using
SimulateBank::Api package which
in turn used internally SimulateBank::Service which spoke to database directly.
SimulateBank::Service implemented "services" like finddeposits,
_createdeposit, _reportdeposit_ and canceldeposit_. These are processes which
codify the business rules of the company. We do not care about objects,
we only care about interfaces and processes.
This second approach was much more convenient and efficient. However,
it still required code duplication and syncronization between Client and
Service. What's more, I wasn't happy with it because it didn't feel "Perlish".
I was resonably happy
with the "super-structure" of the code, the division between Web and Api.
But I felt I wasn't using the possibilities of Dancer2 framework with its
plugins, like Dancer2::Plugin::Database and Dancer2::Plugin::Queue.
I felt I was doing the same job twice when implementing my own interfaces
to database and message queue instead of using the readily available
Dancer2 facilities.
├── SimulateBank
│ ├── Api
│ │ └── Transactions.pm
│ ├── Client
│ │ └── Transactions.pm
│ ├── Common.pm
│ ├── Service
│ │ └── Transactions.pm
│ └── Web
│ └── Transactions.pm
└── SimulateBank.pm
And then it struct me! I had looked at the whole problem from the wrong angle.
My approach was code first. I tried to create a perfect structure for
the future Perl programmers to use.
Towards a Perlish Approach
Perlish approach is result oriented.
After all, do we not pride ourselves on using a language which is fast to
program with? Software only matters if it's put into production.
But what about all that horrible Perlish hacking, the quick-and-dirty way?
That is the Perl way! To create a working solution today. Not to worry about
tomorrow.
In my experience the most far reaching problems in software
development are not done by programmers, or they are done by programmers
when they have been forced into roles that should be done by others, such as
database designer, integration architect and user interface designer.
These are the people who should do the worrying. They are paid to design
long-term solutions!
Business App Blues
The purpose of most enterprise applications is to collect data, and then
deliver or distribute it, or act upon it. So is ours. From the very beginning
we started to plan our datamodels and
the resulting database schema meticulously. For instance, the application has
only SELECT and INSERT access to many of our tables to prevent the loss
of past state information. The whole schema has only one sequence and
its value is inserted in every table so that the whole flow and order
of operations in database is trackable.
Show me your flowcharts and conceal your tables, and I shall
continue to be mystified. Show me your tables,
and I won’t usually need your flowcharts; they’ll be obvious.
Fred Brooks, The Mythical Man-Month: Essays on Software Engineering (1975, 1995), https://en.wikiquote.org/wiki/Fred_Brooks
With Fred Brooks' quote in mind, I approached again the issue of code first.
Code could not come first. Data and datamodels came first. Code second.
In fact, code comes third, because second place is given to apis, most
notably the REST apis created with Dancer2.
Interfaces and Silos
Database schemas and system apis are both interfaces to data.
They are the longest lasting parts of the system - and the ones
that are the most difficult to change later. Code isn't. It can always
be refactored and improved and tested against the unchanging interfaces.
If the interfaces are locked, especially the database schema, and we can
be 99% sure that our data is always protected from a malfunctioning program,
then it's time to give the programmer free hands to create the best
code he can.
Furthermore, the future of our application is in constant change - like in
many startups. Microservices is a natural way to extend this way
of thinking. Different parts of the system become microservices and silos
whose implementation code is their private part. This code can be quickly
changed and it must have no connection to any other silos' code. This allows
very radical changes if need be, such as web programming frameworks,
math packages or even Perl interpreter version.
And what's best, none of the changes in the code threaten the stability
of the whole system. Code quality becomes a matter of code reviews.
Our backs (interfaces) covered, coding with Perl is fast and fun, because
it just works (TM).
The Perl Way
There were several times when we were second guessing our decision to use Perl.
This whole story happened in the course of one year's time.
I consider myself lucky to have been given the chance to go through that whole
mental process. I believe I understand Perl a lot better now - not perhaps as
language but as a way of seeing software development and organizing
development projects.
The first version was indeed only good for throwing away like Brooks writes
in The Mythical Man-Month. We saved some parts and also some ideas from the
second version.
Speed is of the essence. The internal services for accessing database are
mostly skipped and Dancer2 database plugin is used to fetch the data
directly. Most action happens right in the same package where there
the Dancer2 REST interface endpoints are defined because in most cases
the data fetched from or written to database requires no additional handling.
So there is no need to create additional layers, especially when the rigid
database schema assures that fetched data is always sound (no nulls,
no missing values or missing foreign fields).
While quality control was earlier exercised only via code reviews, those
are now complemented with api tests and rigid database schema modelling.
[Software engineering is the] establishment and use of sound engineering
principles to obtain economically software that is reliable and works on real machines efficiently.
Friedrich Bauer (1972) "Software Engineering", In: Information Processing. p. 71
I blog about Perl.