Crowdsourcing self-confidence
Update 12 hours later
I received a massive amount of feedback: here, on the mailing list and in private email. I humbly thank all of you for the many words of encouragement. I am planning to summarize in more detail what I took away from this in a separate post soon (once I am back from an internetless trip this weekend).Original Text
TL;DR - There is this thing called DBIx::Class. It has a number of users, and a number of staunch non-users, which is all fine. Bottom line - it seems to be relatively important. For good AND for bad, I happen to be an integral part of this project for nearly 5 years. A number of my friends (who badgered me into writing this) believe that I am in a relatively unique position to "to boldly take this project where no ORM has gone before". Furthermore I am at a life-junction where I indeed can consider dedicating more time to such a goal (which includes formalizing proposals, seeking funding etc - this however is all step #2).The reason for this post is to resolve the #1 problem holding me back from pursuing any of the above - I have conflicting indicators whether my continuous involvement with DBIx::Class does actually deliver a net benefit to its current and potential future users. After several months trying to reconcile facts (detailed further down) surrounding this project, I came to the conclusion that I can not figure this out by myself. And I will remain stuck until I do figure it out. Therefore I am reaching out to the community with what amounts to a blatant "vote of confidence" request (although, as everything in life, it's actually not that simple).
This will be a rather frank and unusually weird post - you have been warned.
To frame the gravity of the problem some graphs and numbers. Compare the overall history (top green-ish graph) with the history of my commits (first red-ish on the left). The impact numbers do not look much different (yes, I know these are crude measurements, the point is to just establish a rough trend):
lib/ t/
Code lines1 before I got involved (331a564): 10425 24990
Code lines in recent master (d71502b): 25292 44322
What `git blame -w` attributes to me at d71502b: 11676 22130
There are two distinct schools of thought concerning projects of such size and contributor layout. According to the "size matters" camp - I am the current undisputed project pumpking and get to dictate everything, starting from architectural decisions all the way to brace style2. On the other hand followers of the "community matters" scripture would (rightfully) claim that the project is doomed to fail as it hinges on the availability and/or whim of a single individual.
I personally believe mostly in "community matters". And therein lies the problem - I repeatedly failed to build a following around the (hopefully obvious) long term goals of this project. I tried everything I had in my (rather limited) toolbox. At one time or another we had: mandated code review before merge to master, formalized todo lists, weekly irc meetings, attempt of a time boxed release schedule (even though I absolutely abhor the idea), and other stuff I can't currently remember. All of these invariably ended up in quiet failure, and/or deadlock. My personal traits do not help matters either. I have a number of convictions (which I continue to consider non-negotiable). They can roughly be summarized as: The time of a library developer is under no circumstances more valuable than the time of any product developer/system integrator/product tester/etc who happen to be using said library. This (understandably) puts me at odds with pretty much anyone trying to hack on the codebase (both regular and drive-by contributors).
The solution seems to be clear - I am a bad fit for a project manager and need to let someone else take the reigns. But guess what - multiple times in the last several years this is exactly what happened. The unexpected turbulence in my non-digital life would conspire to simply drive me away from Perl and CPAN for multiple months at a time. For reasons still unknown to me, instead of someone else taking the lead and charging ahead, all activity freezes during such times. As I feel largely responsible for "abandoning ship" I succumb to the urge to come back, hack for a while, and the cycle starts over. Such cycles repeated enough times for a very clear trend to emerge: in the land of DBIx::Class/SQL::Abstract/SQL::Translator there seem to be no willing successors. I should note that not so long ago Catalyst was in a similar situation. Kudos to John Napiorkowski for picking up that part of CPAN.
This brings us to the present problem. I have the vision, plan and (apparently) skill to take this project towards something akin to "Plack for RDBMS interaction". I currently have the availability, and seem to have ways to procure funds to maintain this availability. But I find myself unable to pursue any of it, because most of my energy gets sapped away to attempt to continuously prevent implicit3 or explicit crap from entering the codebase. When I fail, I get to play janitor just to put the baseline in a state on which I can once again start building upon, and get basic stuff like CI-testing work again. And while I am at it I naturally get to politely endure whining how we are not doing this or that correctly, and why there hasn't been a release yet. Thus the question boils down to:
Do I begin ignoring some of the noise and continue the present course, or do I take the continuous strain as sign that I should give up and go do something else? Would DBIx::Class benefit more from my continued, albeit somewhat abrasive, presence at the helm (which implies continuous rejection of "worse is better", slow and careful "ready when ready"-style release cycles, multiple layers of checks and testing, and generally slow vetting and acceptance of external contributions)? Or do I officially disclaim any "residual power" and let the project float adrift until someone else steps up and takes it in some direction?
This is not something I can unilaterally decide myself, regardless of the number of commits or authored lines. And this is why this post was written. Thank you in advance for responses/suggestions/guidance/whathaveyou. And most importantly thank you for your time reading this.
Cheers!
[1] Measured by a crude tool which strips anything pod-like away first
[2] Uncuddled 'else' of course!
[3] It's count-the-lob's time \o/
Musing on the pitfalls of the Swiss Army Chainsaw
Although I am not qualified to comment on your position, I think I would like to congratulate you for reaching to the community for advice and help.
I hope those who know best can guide you forward.
Kudos on your brio.
I am not really sure how to comment on this, but I'll give you my point of view.
(for non-riba readers, I am one of the contributors to DBIC and admit to knowingly writing the linked to explicitly bad code)
I could never really take the helm, I think. I am a marginally smart person and know DBIC better than most, and there are still parts of it to me that I can't seem to grok or get comfortable with. The only people that could do that right now that I know of are you and mst. I think that that alone means you should take charge and if you can make it even a part time job you should do it.
I have trouble speaking about generalities, and will mention specific examples of things here. There are a number of features I've written for DBIC that started out less than perfect, and most of them have gone nowhere. The mythical date ops, SQL Server 2012 pagination, the linked bad code, and even ::FilterColumn, which should have started out as HRI ready and still is not. BUT I am not upset by these facts. All of those things are not production ready and non-prod-ready ORMs means serious problems. I will gladly defer to you on these matters. The frustrating thing from my POV is the long gaps when you get busy in meatspace (or burnt out, from my perspective I can't tell which it is and for better or worse assume it's burnout.) The linked bad code, for example, was really only put on CPAN so we could easily try it out at work and make sure that the generated SQL was safe. I know of at least two not-so-good bits in that code that, were you around, I'd gladly discuss and attempt to fix.
I think that what needs to happen is you need to take over as the DBIC pumpking, and those of us in the DBIC world that are sensible need to basically defer to you, though ofc you will always be willing to discuss that. What that means is that unvetted (ie drive by) committers should to an extent see a unified front that we stand by what you say, that we are striving for excellence at release time.
Feel free it ping me on irc, email me, or comment again on the blog to keep discussion public.
Yes, it does benefit from your leadership. Please continue.
I am completely down for whatever you decide and I almost wrote to you directly about two months ago when you finished that huge round of improvements to DBIC mostly on your own to tell you how fantastic you are and how lucky all the end devs like me are to have you working on this project.
I can completely understand your feelings here. You're the one pulling the cart though, and while I enjoy community, if one listens to it without significant filtration, it becomes just another broken bureaucracy.
Going back and retrofitting most of my code to match new ideas/idioms/etc would be preferable to seeing you burn out or give up. You are a Perl star.
I'm not just happy. I'm grateful.
Peter,
FWIW...
First, I have been following the mailing list for the past few years. In that time a pattern has emerged - I rarely read most entries EXCEPT for when I see your name or Rob Kinyon's.
Secondly, DBIC is what it is today largely due to your vision. It is widely used because of that vision, and it seems like you see the next steps with some clarity.
For both those reasons, please enter my vote for your continued involvement.
How much do you care? If you don't care, then just walk away and let the project stall. (C.f. Module::Build)
If you care, then fix the project. I think you have the moral authority given the amount of work you've done and the apparent lack of progress when you aren't actively involved to dictate how the project should run.
If other committers put in crap that wastes your time, take away their commit bit (or right to merge to master) and make them submit branches for your review.
If people complain about your management style and choices tell them to suck it up and deal.
If you think you still need to be annointed BDFL somehow, this is the wrong venue -- you have to put the ultimatum to a DBIC venue: "Either I'm BDFL or I walk." If they don't like it, *they* can walk.
More generally, in the past, I've applied an 80/20 filter to situations like the one you describe.
What are the 20% of things that give you 80% of the joy you get from the project?
What are the 20% of things that give you 80% of the headaches from this project?
Do more of the first and eliminate the latter.
Recruiting for a long standing project with a high barrier to entry and an architecture that's bulging at the seems is hard.
During the periods when I've tried to be more active, I've not been able to give out nearly as many commit bits as I used to either.
I suspect what we need to try and achieve is to get DBIC a bit more decentralised - have it be a specific framework build atop a more-like-Plack-for-DB-stuff - but you already know that's what I have in mind and we both already know it's going to be a big-ass job and we'll see if it pans out or not.
But refusing crap patches is necessary; I stood up for your quality filters to Caelum and I'll stand up for them again if you want me to (people seem to have stopped dying for a bit so I might actually be around more :).
Honestly, as I've said to you a few times over the years:
1) the problems we have with acquiring contributors these days are technical more than personal
2) your standards are pretty similar to mine, and I still don't understand why you regard whining about applying them as anything other than a chance to ask the contributor if they want some cheese
So far as I'm concerned you've been the DBIC chainsaw delegate for years; the fact that normally sensible people like frew and xdg seem to think you aren't already that suggests that if anything you're being too soft a touch with people.
You keep driving DBIC forwards, I'll keep trying to get the tools together so we can pull the architecture to pieces and make it more amenable to volunteering people again.
-- mst
@xdg If I was planning to keep doing this on volunteer-basis only, then yes, your comment makes much sense.
However (as hinted in the post) I am toying with the idea of formalizing my work on DBIC as something that actually puts food on the table. Precisely because (as mst said) it is an immense amount of work, I can't hope to take a stab at in my "spare time".
Under such an arrangement the stakes are different: I can't consciously ask to "get paid to do whatever I want". Hence why I wrote all of this - I needed some input on my "perceived value".
@everyone_else - Thank you for the kind words. They do make a difference!
This is gonna take a bit, and I apologize for that, but there's a lot to say.
First of all, you're quite technically gifted. Beyond the ability to bring ideas into term, you also have a plan of where to go, what to accomplish and how. You are in an excellent position to be the dictator, without a doubt.
That being said, you can be abrasive. It's not necessarily a reason not to be the dictator. Linus has been proven it very possible. Mst is another good example. A good way to work this out is to delegate some things.
There has to be someone that is less abrasive than you. Even a few. :) How about settling with them on the things you refuse to incorporate, and have them interact with the new contributors? How about getting them to slim down the list of pull requests to remove the rejected with a kind message? Just an example.
Also, having a vision of where and how you want the project to be is great. Still, always be open for discussion. Maybe you will be convinced, maybe you won't, but it will make you *accessible*, which is far more important than agreeable.
And with all my heart, all the best for you. I rest easily knowing you're at the helm of DBIC.
Whenever I think about how I would organize development around one of my modules, such as DBM::Deep, I use you as my guiding principle. mst may be able to recruit the crazy-good people, but you make sure the ship steams forward without hitting icebergs.
I think there should be a way to formalize the relationship such that you are able to do this job (for it is truly a job) 50%+ of the time. We (and others) can discuss a way of eliciting some form of sponsorship from the large corporate users of DBIC.
@steve.schafergr - Wow. You made my day. :)
Sounds to me like you've got some clear ideas about where you want to take it, and the desire (for some definition thereof :-) to do it. I think you have two choices:
If you're not comfortable with the first, then do the second.
20 months ago I gave my first-ever Perl talk. Afterwards you (and some others) came and gave me some encouraging words, along the lines of "keep doing it". I did. You should too.
Peter,
I am a user of DBIx::Class, no developer. And although I have to struggle sometimes to get my head around it's complexities, I understand how important this module is for my work and for many others as well. Meanwhile I do use it all over the place and would hate to go back to pure SQL.
From a pure user point-of-view, I have invested quite some time in understanding DBIC. I would hate to see that investment go down the drain, because the project lead steps down and nobody picks up where he (you) left.
Unfortunately I cannot judge who would be capable to pick up after you. I follow the IRC channel often, and the mailing list sometimes. Some names come up more often than others, but that's no basis to judge someone as project lead.
So, yes, I would love to see you continuing this project. And yes, I would be willing to contribute to the funding of the project. Not with a huge amount of money, more like a pittance... but nevertheless.
Thanks for all the work.
Cheers,
Ekki
A number of other people have already commented on the main point of your blog post, so I am just addressing a secondary point.
You say: I have the vision, plan and (apparently) skill to take this project towards something akin to "Plack for RDBMS interaction".
Have you written or explained this vision at all yet? How that would differ from DBIx::Class as it is today?
From my perspective at least, the way DBIx::Class is today or anything resembling it is more analogous to something at the level of Dancer or Catalyst than something at the level of Plack.
I believe that the most Plack-like analogy for RDBMS interaction would have to work roughly like DBI does (or each individual RDMBS for that matter), having a command design pattern, using SQL or some other language that is at least as expressive, such that interacting with the RDBMS is represented as routine calls, eg $foo_result = do_task_foo(@foo_args);.
Any DBIx::Class that works like the current version, would be better as an interface built over such a Plack analogy, rather than being the Plack analogy.
That prior comment was by Darren Duncan, though it doesn't say so. I will also add that for anyone having trouble posting a comment, try going straight to Submit rather than using Preview, as going through Preview gives a session timeout error.
Committees don't write good software, small dedicated teams do. Usually the teams are highly inequitable in terms of measurable project artifacts. Checking the contributors graph on a few other github projects will show almost exactly the same profile as this one. If anything, based simply on my biased sampling survey of those graphs this project may be more equitable than most.
It is clear that there is a net benefit to the Perl community from this project. Simply checking the number of other projects that depend on it is enough to highlight this.
Users and contributors would not be whining if they were not already invested in the success of the project. Try to unload the emotional content and deal with the facts they contain. Those who complain the loudest can often be turned into advocates by giving them a little attention.
Rough consensus and running code.
I won't address the main point of the blog post; other comments have adequately expressed my thoughts.
For the past years, you have been the face I put behind DBIx::Class, and can only say good things about your work.
As a regular DBIx::Class user I welcome your work and hope that you keep doing for a long long time.
Peter, whether it's volunteer or not doesn't matter. Either you're in charge or your not.
It sounds like you are. So, fix the management parts of the project in the way that you want. That's what being BDFL means.
Rule #1: Peter is always right
Rule #2: When Peter is wrong, see rule #1
Whether you get *paid* for your work is separate. That's between you and the funders. Can you articulate a compelling vision? Can you demonstrate your ability to manage the project to delivery on goals?
The stronger a BDFL you are, I think the more compelling the case for paying you to make things happen.
David
Reply to Chris, who said:
'Committees don't write good software, small dedicated teams do.'
Indeed.
This reminds me of an old and blunt saying:
To win the high jump you hire someone who can jump 7 feet, you don't hire a team of people who can each jump 1 foot.
Reply to Peter:
Seems like you're agonizing over how to be firm re other peoples' code, or over how hard to be when being firm.
Well, given the huge amount of support I read here, I'd say:
Cheers
I have no idea what Plack for DB would look like, but the fact that you do makes you amazing. And I already knew you were amazing. That means you're doubly amazing.
All that said, do what's best for you. Do what you want to do.
Please continue to be the "architect". If you feel the need to "back off" from being the main contributor of actual code, I'm sure you could become more of a Linus ... but the "unique vision" part of it is quite important; it shapes how everything fits together.
Hi Peter,
I'm neither contributor nor user of DBIx::Class, but as an attentive observer, I always wondered how such a big and complex piece of software could evolve in a consistent way under the collective hands of many people with possibly different visions. This development model has the richness of the bazaar, but if you want to reach the steeple of the cathedral it may make it a bit more difficult. So, as I understand, one of the points of your post is that you would like to preserve collective energy, but you feel alone in taking the hard decisions (and also doing the hard work).
Psychologically, it is always a burden to remove somebody's contribution a posteriori : both the contributor and the rejector will feel bad, they may argue, etc. So maybe you would be better off by putting some rules at the entry gate : for example ask everybody to contribute in branches, and keep for yourself the right to merge into master. I guess everybody would accept this way, and it may make your life a bit easier.
Just my 2 cents...