Sawyer X

Recently I got a nice small project work on: a web interface for a database with a simple search mechanism (Ajax for frontend with redirects to actual result pages).

I received the database in Excel form. No worries, we have the excellent Spreadsheet::ParseExcel so I'm not scared of spreadsheets. Bring it on!

And yes, the client did "bring it on". He brought it on with 260 columns, nonetheless. Each contained a "1" or "0" for match. "You just go over the columns here, look for '1', and then continue over to the product name, search it in this sheet over here and find the number to the right and return that to client - simple!"

Yes, two-hundred and sixty columns. Alright, so I'll just normalize it. "You don't need to normalizical nothing [double negative!], it's good the way it is" - "No, trust me, I need to normalize it" - "Alright, knock yourself out".

It's one of those "this is simple, I'll just flatten the columns into matching rows". I started with creating tables and using DBIx::Class::Schema::Loader (haven't used it in a while) to create clean beautiful schema files. That was really easy.

Then I started writing small importing scripts - only 3 for the entire project and all tables. However, in the second importing script, I found a problem with the schema. So, instead of starting from scratch, I figured "if I just add this column here and that one here, it will work". The idea was to continue since I was close to finishing. Somewhat like not taking a break because you're sure you'll be done real soon.

What started as "I'll just fix this here and keep going" ended up as "how the hell do I do the next step here?" and "how the hell do I fix this now?"

This is something I like to refer to as "under-engineering". It's a common side-effect you might experience when you take things too lightly and don't really stop and think when something doesn't fit. Like when you're writing a script and if you just work around a small problem here and another glitch there. You probably know that feeling.

Learn from my experience: when I decided it's messed up enough to start from scratch, I sat with a pad and pen. In 10 minutes, I finished outlining the new database structure, changed the schema in 5 more minutes and fixed the importing scripts and interface.

I finished the entire project in 1 hour of work instead of spending 3 hours trying to work around something I thought was small enough to not care about.

The problem with working around a glitch here or there is that they aren't necessarily the only ones. There could be 2-3 (what you would call "a small oversight") and they might end up being 20-30 workarounds. Try to stop and think whenever something doesn't work, because it might impact your project more seriously than you would imagine. I was able to start from scratch in this project, but could I do it in other projects? Probably not.

And here's a funnier encounter of the same form of "under-engineering", hailing from the daily wtf.