This is in part because of cost, and also because the client involved are, in the long term, including the functionality in their SAP "solution".
I could rant for several megabytes about how this is a silly desicion, but it was made so far above my head I can't even see it from here. Seems like that's always the case when SAP is involved.
So, after spending roughly €20.000 on something it's being scrapped, and some SAP consultant will implement it (sort of, anyway) in their way, and bill €1.000.000. Yep, makes perfect business sense for all involved.
Aaanyway, never mind this blog. It's not relevant.
]]>My rewrite effort has been bogged down in security certifications and other time-consuming stuff, so I've been unable to sit down and do much actual code.
In stead, I've been reading Modern Perl again, and trying to wrap my head around PCI security standards compliance, and reading a whole stack of blogs and books about Perl (thanks, guys!).
Now that a lot of the modules I'll be using are more or less in place, including my new centralized security where "everything" will go to get permission to do "anything", it's time to start sowing the modified modules together in Catalyst.
Or is it?
I try to code by "Make a plan, and stick to it, unless it sucks."
Figuring out if a solution sucks is part of that, so I've taken a long, hard look at my Catalyst choice here.
Did I really pick Catalyst because it's the best tool for the job, or because it's the weapon I had in my hip holster at the time? Let's see what we're looking at here:
So, how does this mesh with my view of Catalyst?
Oh. Well... That doesn't look very good for Catalyst in this context, does it?
What else is there, that will do this job in a way that might be a little simpler for my successors to maintain?
How about Web::Simple?
I've been playing with it sort of as a toy lately, and I've grown to like it a lot, but will it really work in my business application?
Yes it will, and it will work great!
What the application will be doing is mostly simple CRUD work, which will be handled by DBIx:Class and a bit of wand-waving there, in pretty much the same way no matter what framework I build it in.
The really heavy stuff isn't done by the user-facing web script at all, but in a dedicated cron-run script, so for the heavy lifting it's irrelevant what web framework is used.
Then comes the brillant part: If done properly, it'll be easy to shift from Web::Simple to Catalyst later.
Right, so now I'm not building a Catalyst app anyway, because I don't really need the heavy lifting Catalyst can do.
The great thing about this is that all I really need to re-do of the work that's already been done is to rewrite a few of the tests and search-and-replace "Catalyst" with "Web::Simple" in the documentation. Seriously.
So far today I've collected a list of all the information I would like to store, and marked out all the information that is currently stored. It doesn't look like I have to add a lot of information, but I'm certainly changing how it's stored.
In the current version of the application a whole lot of the information is stored in static configuration files.
For example, the concept of a "Morph Rule" is a configuration variable.
So a case type is changed in accordance with a "Morph rule".
When this was designed it was not clear how often these things would change or need tuning, so it was hard coded in a way that would make it very fast to load at start-up, and very easy to access while running, but very impractical for the application to change.
Changes are currently done by manually editing .pm files.
This would have been fine if the changes were few and far between, but I find myself editing this "configuration" a whole lot more often than the design documentation of the application indicates necessary. In the past, these things "never" changed, but this business is becoming more and more dynamic, and changes more and more frequent.
Clearly, this has to be on-the-fly changeable by administrative users. If they have to toss it across my desk every time there is a change I won't be able to do much else.
Another thing that's very wrong about the existing model is the lack of foreign keys anywhere.
This is my doing of about a year and a half ago, and it shows my lack of experience with complex databases. My reasoning behind it is that if something goes wrong it's a lot less complicated to untangle the mess if you're not constrained by a bunch of foreign keys. Since then it has become clear to me that in such an emergency you can simply suspend foreign key enforcement.
Oh, the sins of the past... I'm just glad I have a chance to undo them. With what I've learned since then I'm sure I can build a better model.
MySQL (or variant), InnoDB, and foreign keys, all supported by some nice DBIx::Class as model for a quick and snappy Catalyst application.
Yep, this is starting to taste like a real project now.
Swapping out Class::DBI with DBIx::Class is an easy choice, and I'll get back to that in a later post, but what of the engine behind it?
Currently, the whole thing is built on MySQL, but during the lifespan of this fairly complex CRM application I have seen some of the limitations of MySQL up close.
A lot of my contacts and friends also insist that while MySQL is fine for a website or any simple CRUD operations, it will let you down when it comes to more complex queries.
Rumor has it that these problems are both in performance and dependability, and while it might have been simple user error, I have seen both.
Keep in mind, however, that I am no DBA. What indexing my databases currently have I have gotten help from friends and colleges to build, or they were already built, and I inherited them.
What I think, however, is that the optimum choice of database is largely dependent on the use case.
For our statistics needs, for example, there is a lot of COUNT usage, and anyone worth their salt knows that PostgreSQL and most (all?) other MVCC-implementing database engines, even InnoDB for MySQL, will COUNT rather slowly.
A splash of googling turned up a PGSQL COUNT(*) workaround.
I must say, that's pretty good. The total number of rows in the database, however, is useless to me.
I need stuff like...
SELECT COUNT(DISTINCT(`caseID`)) FROM events WHERE userID = 31;
...to determine how many unique cases user 32 has logged events for.
Alright, so I update the user record every time a case is assigned, to reflect the "ownership count" for the user that lost it and the user that gained it. Right?
SELECT COUNT(DISTINCT(`caseID`)) FROM events WHERE userID = 31 AND loggedTime > NOW() - INTERVAL 30 DAY;
...Okay, now what? I need to know how many unique cases user 31 has logged events for in the last 30 days. Suddenly, it becomes very complex to keep updated. So complex, in fact, that it's better for overall performance if I just leave it with the straight-up COUNT syntax with no further hooplah.
In the end, it comes down to this: Where do I need the speed?
Both me and the people who will be using the statistics on a daily basis are fine with the statistics taking a few seconds to load. That's not where the performance is needed.
Where it is needed, however, is in the so-called Morphing, where our business rules are applied. It's also needed in Case Selection, where the cases are prioritized and the one that comes up on top is handed to the first available agent.
So, yeah, COUNT is relatively slow, but I won't be using that as much as long, complicated and confusing SELECT that don't have any COUNT in them at all.
It's these 50-line SELECTs I have to worry about. The WHEREwolves.
When it comes to these beasts, I set out to try and find dirt on each database engine. The problem with doing this kind of research is that anything I might find might be very much out of date. A major problem with any DB engine could be fixed in the latest release, so I need the data as fresh as I can get it.
How about June 2009? Since that article, PostgreSQL has advanced from 8.3.7 to 9.0.4 and MySQL from 5.1.30 to 5.5.14.
Obsolete data!
After over an hour just searching for data and even trying to run some benchmarks of my own, I spoke to a coworker on the subject. He's not doing software development for the company, but he runs a website with thousands and thousands of hits per day in his spare time. It runs MySQL.
He made me realize one major upside to MySQL: People know MySQL!
I won't be around to maintain this application for ever, and poking around the company a little reveals a healthy heap of MySQL people, but only the occasional "Yeah, I've used Postgres once".
So, it turns out that this decision won't be about dependability or performance at all. It will be about maintainability down the line, and about what people out there know how to use.
With the performance so seemingly close, and dependability reportedly being so much improved for InnoDB in MySQL in later versions, it seems we're staying with it.
I just never even thought of this reason to do so.
Thanks, Tom Erik. Your perspective was very valuable to me.
]]>I'm about to embark on a project to rewrite a mod_perl2 application to Catalyst.
I promised mst I would blog about it, or otherwise make the process known, so that others might follow my example.
Sure, I'll do that, but is it really a good idea to follow in my footsteps?
I'll leave that up to you, my reader. I obviously have at least ONE reader, as you're currently reading this. Right?
I can't share any of the source code in the larger context, since I don't own it. My employer would probably fire me outright if I shared the proprietary software, and I'll even do my best to keep my employers name out of this. This is about the process of rewriting, not about my job in general.
I also won't go into the old software too much, except where it's relevant for the rewrite, and I won't comment on the choices made when it was written or anything of the sort. This is about modernizing, not about dredging up the past.
Let's just hope I remember to update this as frequently as I want!
]]>