Total improvement: time spent inside CAG dropped by 57% and 44% in SQLA Total time elapsed: ~2 hours.
If this is not a sign of what makes the perl community great, I don’t know what is.
The code is not flying any any means, but the total runtime was reduced by 34%, which is not bad for an evening’s work.
SQL::Abstract Changes DBIx::Class Changes
(read on for comparison of the top 15 subs)
]]> BEFORE:Profile of test.pl for 5246s, executing 533397042 statements and 183082682 subroutine calls in 342 source files and 80491 string evals.
| Calls | P | F | Exclusive Time |
Inclusive Time |
Subroutine |
|---|---|---|---|---|---|
| 118331 | 1 | 2 | 573s | 573s | DBI::st::execute (xsub) |
| 18740893 | 5 | 1 | 200s | 280s | DBIx::Class::ResultSource::_columns |
| 8119393 | 93 | 93 | 198s | 247s | Class::Accessor::Grouped::get_inherited |
| 3250960 | 1 | 1 | 158s | 380s | SQL::Abstract::_quote |
| 28352404 | 27 | 27 | 126s | 126s | Class::Accessor::Grouped::get_simple |
| 5354668 | 4 | 1 | 121s | 404s | SQL::Abstract::_SWITCH_refkind |
| 4637845 | 6 | 4 | 113s | 318s | DBIx::Class::ResultSource::column_info |
| 111161 | 1 | 1 | 112s | 296s | SQL::Abstract::_insert_values |
| 5487339 | 1 | 1 | 107s | 124s | SQL::Abstract::_refkind |
| 5487339 | 2 | 1 | 94.7s | 219s | SQL::Abstract::_try_refkind |
| 4827092 | 8 | 5 | 71.8s | 395s | DBIx::Class::ResultSourceProxy::has_column |
| 3250960 | 6 | 3 | 66.1s | 447s | DBIx::Class::SQLAHacks::_quote |
| 118331 | 2 | 1 | 57.3s | 287s | DBIx::Class::Storage::DBI::source_bind_attributes |
| 4827274 | 3 | 2 | 55.4s | 131s | DBIx::Class::ResultSource::has_column |
| 133641 | 1 | 1 | 52.2s | 561s | DBIx::Class::Row::new |
AFTER:
Profile of test.pl for 3466s, executing 328628699 statements and 133802224 subroutine calls in 345 source files and 80406 string evals.
| Calls | P | F | Exclusive Time |
Inclusive Time |
Subroutine |
|---|---|---|---|---|---|
| 118331 | 1 | 2 | 304s | 304s | DBI::st::execute (xsub) |
| 8119393 | 93 | 93 | 198s | 246s | Class::Accessor::Grouped::get_inherited |
| 111161 | 1 | 1 | 111s | 270s | SQL::Abstract::_insert_values |
| 4637845 | 6 | 4 | 103s | 134s | DBIx::Class::ResultSource::column_info |
| 3250960 | 6 | 3 | 78.1s | 82.1s | SQL::Abstract::_quote |
| 4827092 | 8 | 5 | 72.0s | 330s | DBIx::Class::ResultSourceProxy::has_column |
| 118331 | 2 | 1 | 55.9s | 160s | DBIx::Class::Storage::DBI::source_bind_attributes |
| 4827274 | 3 | 2 | 54.1s | 66.4s | DBIx::Class::ResultSource::has_column |
| 118331 | 1 | 1 | 51.1s | 846s | DBIx::Class::Storage::DBI::_dbh_execute |
| 133641 | 1 | 1 | 51.1s | 470s | DBIx::Class::Row::new |
| 2103708 | 3 | 1 | 50.9s | 160s | SQL::Abstract::_SWITCH_refkind |
| 18740893 | 5 | 2 | 43.7s | 43.7s | DBIx::Class::ResultSource::_columns (xsub) |
| 2374996 | 4 | 1 | 39.3s | 213s | DBIx::Class::Row::store_column |
| 2236379 | 2 | 1 | 36.2s | 64.0s | SQL::Abstract::_try_refkind |
| 9791840 | 39 | 30 | 31.7s | 31.7s | Scalar::Util::blessed (xsub) |
Pretty good!
]]>So, while I disagree with the need to throw money at designers to make pretty websites for libraries whose target audience is Perl developers who already know how to work CPAN and probably know how the rest of us roll, it’s be nice to have websites for end-user applications. You know, the kind of stuff people who don’t give a damn about the API use, whether it be from the web or the command line. Those are the people that might not know to look in CPAN / perldoc. Those are the people who don’t care about our API docs, they just want interface docs and frequent use cases illustrated. In this case, it makes perfect sense to have pretty, user-friendly websites published. IMHO, you should also have a PDF manual. But for libraries? give me a break guy. We got better things to do.
]]>Yesterday, I was thinking about this same exact thing. I have a project where the compile-time hit of the collective libraries that make up the application is a little on the heavy side. I thought to myself that it would be good use something like Test::Aggregate, but I wondered if that could mean that some of my tests could be affected by a downstream dependency where the state could be changing at the class level. I don’t think it’s very probable, but it could happen and it would be a real pain to debug.
]]> This is where I came up with the idea that maybe it would be best to load all common libraries in an initial process and then run the actual tests in a fork to benefit from COW to maintain a single compilation phase but still keep run-time data separate. While the approach has it’s own issues, mainly that running the tests in parallel could be a bit of a PITA, I think the idea is worth exploring.Another thing that I’ve been thinking about lately, is structuring some of my tests as Moose classes and making liberal of use of exception handling to prevent tests from out-right dying and making my BUILD method responsible for making sure any data that the tests require is in good order and making DEMOLISH method responsible for cleaning up after this particular test. The idea here is to keep all the logic related to testing something and then restoring the previous state in the same place.
Again, I haven’t really given extensive thought to this, but I suspect that as some projects grow to a sufficient size, the ideas will resurface.
]]>Now, Here is how we interact with it:
And Finally, this is the log of events:
neat huh?
]]>I am aware of the existence of MooseX-Workers, but it didn’t ‘feel’ right for the job. Additionally, the MooseX-Workers approach of running jobs in a separate process via POE::Wheel::Run didn’t work very well for my approach. You see, in order to reserve a job from beanstalk, a connection must be open with the server. During a fork, I could either fork and keep file handles open, making sure the file-handle survived, or lose my connection and my reserved job. Additionally, the constant forking was a little too much over-head for the very simple jobs I do.
Instead of trying to make MooseX-Workers do what it wasn’t designed to, I decided to go with a preforking approach and roll my own approach. I have a manager process and n children processes. Children processes are free to work by themselves and block if they want to. Communications between the manager and worker processes happens via POSIX signals. Right now 3 commands are supported, pause (USR1), resume (USR2), shutdown (TERM & INT).
So far, the preforking manager is pretty basic, but I look forward to adding the following in the future:
These methods existed in the internal version this library was based on, but due to a bug in Path::Class::Dir, I was unable to add these features until Path::Class 0.18 was released with the fix. As of now, this module can be considered stable. I promise to add no code without extensive tests and change no APIs unless it's really, really necessary. Even then, there will be a long deprecation cycle and a back-compat layer.
If you haven't already seen it, I recommend you take a look at Cantella::Store::UUID. I use it to store many, many files in the file system without concentrating too many files in any one directory. Accessing files is quick because their location is deterministic (based on their UUID) and you can split the hierarchy into different physical devices to improve performance or add capacity. It's not very high-tech, but it works surprisingly well if you have huge numbers of files to store. Another nice thing is that you can mirror it with r-sync or create incremental back-ups using rdiff-backup.
]]>The idea behind MooseX::TypeMap is simple: it holds a series of "Entry" objects which have type_constraint and data attributes. You can then give MooseX::TypeMap an instance of a Moose::Meta::TypeConstraint and it will give you back the data associated with that type-constraint or, optionally, it's closest super-type that TypeMap knows. That's it. It's simple and it's short and it's hardly very exciting, but it will make my life easier.
Take, for example, a Cantella::Data::Tabular renderer, which needs to render different cells based on their associated type-constraint. Different people may want to have different rendering rules, and a good way to implement the conversion would be using MooseX::TypeMap to automatically fetch a closure or printf format based on the cell's type. A sane default could be shipped with the formatter and the user could either modify it or replace it to match his/her needs in only a couple of lines of code.
]]>Cantella::Data::Tabular is a library for working with (surprise, surprise) tabular data. It contains three main objects, so far. Cells, rows and tables. A table has zero or more rows, and row has zero or more cells and a cell has zero or one value. Additionally, cells will be able to store certain meta-data to facilitate round-trip conversions from certain formats like ".xls". One of the decisions I made from the start was to disallow cells with 'undef' as a value. Cells can either have a value, which must be defined, or have no value. Additionally, I decided that I wanted different rows to be able to have a different number of columns. Lastly, I decided that the whole library should be completely unaware of what cell values mean. That means no references, no functions and no ranges. Values are simply values and it should be up to a higher-level library to decipher exactly what those values mean or don't mean. I did, however, build in cell-level type-constraints.
The library may seem limited, and may not grow to be expanded beyond the most basic needs, but that's the way I want it. Exporters and importers that write or read from Excel / CSV / HTML / PDF files will be made available separately as they become necessary. What I don't want to end up with is a spreadsheet library. I wouldn't mind building another library on top of C::D::T that is able to perform basic spreadsheet functions, but it's a slippery slope to go down.
As of right now, the library has basic tests and is almost fully documented. It hasn't been released to CPAN yet, but it will as soon as it does something useful. I just wanted to let people know this thing exists and invite any comments, suggestions, feature requests or criticisms you might have.
]]>Which all-in-all is a pretty easy way to create Create / View / List functionality for one of the Collections in your Model. Look for more actions to be added in the near future.
At present, the web interface to the Reaction repo is undergoing maintenance and possibly migration, so I don't have a better link for you, but I hopefully will soon.
]]>I decided from the beginning that I wanted a well-defined API for the storage layer, so that as different back ends would be easy to write and swap around. While writing the tests for the first one, I encountered a problem that’s been haunting me since: how to deal with locks.
Originally, I imagined a job would either be locked, or not locked and manager processes would be in charge of releasing locks if workers died / failed. Then I started thinking, what if a manager process dies? Who releases the locks? So I decided to add time-based automatic lock expiration. This introduced a new problem, a possible race condition. What if a job takes so long that it’s lock times out before the job is done? What do I do then?
A possibility is to have a manager process / thread that monitors its workers and updates the lock-expiry time, which is the route I am likely to take. I don’t particularly like it, but I can’t come up with anything better right now.
I’d love to hear from anyone that has any experience with this kind of problem.
]]>