On The Mojolicious Codebase
There has been some discussion this week about forks of pieces of Mojolicious. Frustrating discussions over what is proper and how to discourage forking. It has been a long week to be honest (and thankfully the recent incident has been peacefully resolved, see postscript).
But then finally, just today, I’ve chosen to see things a different way. I’m really happy to see what lengths people are willing to go to in order to use Mojolicious. This includes addressing a perceived need for streamlining by taking a maintenance burden onto themselves and forking that code that which they need. They see the value in Mojolicious’ code, if not the value in the toolkit as a whole.
There is a definite victory for us there. Mojolicious contains code worth having and worth using. Imitation is, after all, the sincerest form of flattery.
We have seen forks of Mojo::Base (the object system), Mojo::DOM (the HTML/XML DOM parser), Mojo::JSON, Mojo::Template, Mojo::EventEmitter, Mojo::Date. There are probably others we’ve missed. We haven’t yet seen a fork of Mojo::IOLoop, which surprises me, seeing as it is my favorite part of the Mojo toolkit.
Another we haven’t seen forked is Mojo::UserAgent, which is probably the one we are asked most often to spin out. Why is that? Because to do so you’d need to first have to fork Mojo::Base and Mojo::IOLoop. You would need all the Mojo::Transaction and Mojo::Message classes and subclasses of course. Now with the message classes you would need to fork Mojo::JSON and Mojo::DOM since messages have handy access to these representative forms of their content.
To test it you would need some simple html runner class, something ultra small, not a framework per-se just a handler. The rarely discussed Mojo class fits that bill, so you could fork it too. Then since you would need to test the real-time features of Mojo::UserAgent you would need a server class, since testing nonblocking calls with true mock server is near impossible. So you would fork Mojo::Server::Daemon.
With the Daemon server and a Mojo micro-app you could now test Mojo::UserAgent, but how do you actually write the tests? You need to start a server and attach it to the user agent. It so happens that Test::Mojo can handle this for you. It would then be nice to have convenient test methods built in to simplify the testing, DOM and JSON parsing for example. Test::Mojo has these and once again you see the need for Mojo::DOM and Mojo::JSON too.
At this point you have most of the Mojolicious distribution. Mojolicious, the web framework itself, is only a few more classes on top of these building blocks.
Even for us, spinning out these modules would be hard. Not just from the sheer effort of maintaining it either. For example, recent refactoring in the transaction classes would have spanned a large number of these proposed distributions had they been separated. The time in preparing and coordinating the releases (let alone the hassle it would have taken for users of the dependent classes to uniformly upgrade) would have been incredible. As it was, I’m sure most of our users had no idea the refactoring happened, and in time they will be pleasantly surprised by the features we will be able to offer them now that we have.
Now you could accuse this of being a Majestic Monolith post, maybe it is. But as of Monday March 22, 2016 the entire Mojolicious distribution is only 8491 lines of code. It doesn’t have that characteristic quality that most people fear from monolithic projects: bulk. Indeed Mojolicious has been steadily getting smaller over time, even as the feature set has increased. This doesn’t include tests of course, because with 11237 of them it would make the framework look a lot more bloated than it is.
So yes, it may seem a bit odd to install a web toolkit just to get a JSON parser, and well, perhaps it is. But why are you using JSON? For many people it is the data interchange format of the web. Why not try out Mojo::UserAgent? It lets you do nice things like
say $ua->get('api.metacpan.org/v0/release/Mojolicious')
->res->json->{version};
This is all the more true of an HTML/DOM parser library. Surely that data came from or is going to the web. By comparison (and only to counter the idea of size), consider for example HTML::Tree and its dependencies. They total almost the same number of lines of code as the entirety of Mojolicious. Plus they require a compiler. Should you then need one, and unless you have Perl 5.14+, you still don’t have a user agent. Mojolicious really is amazingly tiny for all the features it boasts.
So now having argued that forks of all but the most peripheral classes are really out of the question for the complexity reason, we can get to our actual concern for forks: the quality of our code and yours. Sadly most of the forks of Mojolicious classes are out of date. We don’t doubt the intentions of the fork projects, but they are usually only one person. Mojolicious has a development team, a large community, and an incredible track record of bug fix speed. Downstream modules may not pick up on those as quickly.
But of course we can only respond to bug reports if we see them. Our real fear is that a critical bug may be reported to one of these forks. If the bug doesn’t get reported back to us how can we get the fix to our users too? Even if it does, the gears turn slower and it takes longer for us to address issues. This isn’t an abstract fear, both cases have happened.
Mojolicious has gotten to be the high quality code that is in such high demand because of this dedication to quality. We hope you will value this dedication more than a few bytes on your hard disk. And who knows, once you’ve installed it for Mojo::JSON or Mojo::DOM, maybe you’ll start to use Mojo::UserAgent or Mojo::IOLoop too.
Postscript:
I had written this post even as there was some controversy over a forked module. The incident has brought to light one use-case for a fork: compatibility with older Perl versions.
While we still would prefer to discourage forks, we can understand that there do remain some needs for Mojo toolkit modules on Perls older than 5.10, our current minimum support version. Indeed in the past there was a blessed fork of the entirety of Mojolicious with the goal of 5.8 support. That effort has concluded as its author no longer needed it, however, since it was only ever on GitHub and was clearly named mojo-legacy, it is unlikely that new users will confuse it for the true Mojolicious distribution. From that example, we can concede the notion that such projects have validity. But please come and talk to us first if you would like to maintain such a labor of love.
Also, it really has been a long week for me, please keep the comments civil and respectful. I’ll be moderating a bit more tightly than usual.
Nota Bene: the forks this article refers to do not include github forks, those are really a different concept and only cursorily related. We value contributors and before you can Pull Request, first you must fork. The intent is the issue; when you fork on github, you do so to merge.
As I delve into the deeper Perl magic I like to share what I can.
I did a light research (well, just a query in DuckDuckGo actually) and didn’t find anything, so I’ll just shoot in the dark. Hopefully in a civil and respectful way, even if I said “shoot”.
I was somehow tickled by the use of “discouraging” word, wondering… what does this mean?!? Mojolicious is released under the Artistic License 2.0, which basically does nothing to prevent a fork. So, this must be something done on different grounds.
Is this public condemnation of a fork? Like, a swarm of core developers writing heated posts about how bad would it be to use a forked module, dooming your codebase and making pimples appear on your face? Well, that would be a waste of time, and I guess you have better things to do. (Use the immediate outer ring with respect to the core team maybe? Just joking!) So, this must be something done on different grounds.
A notice/public statement, maybe? I’m having a hard time thinking how this would be phrased, as what you have to do is probably show the advantages and disadvantages of forking. At this point, the light hit me: this very post is the discouraging part! Advertising the virtues (and technical reasons) of keeping it together, while at the same time fighting the implicit objection that “I don’t need all that stuff”. To this regard, the post is very good.
One step ahead would probably be understanding whether people want to enhance what’s in Mojolicious or just eliminate code that’s “not needed” in their use cases. The latter is addressed by your post, the former is probably better addressed with “encouraging to participate in the project”.
So there we are, what I really had issues in the beginning was the very word “discouraging”. It’s a negative one, and likely to upset people. I’m a dreamer, but a slight change in terminology might make wonders. And, probably, the only single warning I would put about forking: “Feel free to exercise your right to fork. Notify us about bug reports/fixes if you want, but don’t expect to receive them from us. Respect our right to evolve our codebase as we see fit.”.
Last, I think that communities in general take the forking of a project very seriously, which is might sound somehow strange considering how easy it would be. But IMHO people came to appreciate the power of the community, which means that usually a big fork was associated with a big fracture in the community. So I wonder: as long as your community is happy, why bother?
Yes indeed this was a perfectly respectful comment. Thank you.
I chose the word “discouraging” there as a rhetorical counterpoint to the second paragraph. It seems to have worked as it left you with the negative taste of how I had felt all week.
And yes Mojolicious is licensed Artistic v2 so as you say people are within their legal bounds to fork Mojolicious if they choose. However, I do actually stand by the word choice from a technical standpoint; for the reasons in my article we do still want discourage people from forking parts, and especially for the packaging reason.
When I was trying to educate myself about the issue of forks in the open source community I came across this article. I don’t know what the author’s credentials are but I did find some guidance in it https://jamesdixon.wordpress.com/forking-protocol-why-when-and-how-to-fork-an-open-source-project/ . In it the author lays out some groundwork on the serious nature of forking and what are (and aren’t) some reasons for doing so and how one ought to go about it.
That point is in the linked article too, we consider it a serious thing to fork someone’s code. Yet it seems that Mojolicious code is popping up quite regularly on CPAN. I hope that the authors take that choice seriously but given that the don’t seem to regard the linked post’s suggestions (they almost never come by and talk to us, one of my points too) that I wonder sometimes if they do.
Indeed in some cases they even change the code’s license when they fork it!
Therefore you are generally right. We do want to discourage forking because in practice it seems (from our view at least) that most times people are forking for this reason of “because I want less code and forking is easy”. We hope that people can see how little code Mojolicious is and then choose just to use Mojolicious rather than take what is usually considered a serious choice to fork. I really wanted to make our case but do so in a positive light. I hope it worked.
You bring up an interesting point, but I have a counter to that. What if a user is searching for some modules on MetaCPAN and comes to one of those forked modules when they might have used a Mojo toolkit module? Will they find the community? That person would be a valued member if they chose to be, but by finding a fork first, perhaps they never try Mojolicious or are aware of our community at all. That would certainly be sad.
I first came to Mojolicious because brian d foy told me about Mojo::UserAgent and how much he liked using it. Imagine that someone had actually successfully forked that module (challenges listed above aside). Would it be possible that I myself never would have come to the Mojolicious community?! What a difference that would have been for my life!
P.S. I was told that Ruby on Rails is 175k lines of code! I can start to see why people might be hesitant to depend on that for just a say a JSON parser! :-P
From where I stand it appears that Mojolicious attracts a different type of person to Perl than perhaps us ‘old schoolers’. Historically Perl and CPAN has followed the Unix philosophy of ‘do one thing and do it as good as possible’. Under that type of thinking when it is possible to break down a project into many, stand alone parts it makes sense to do so. Which is why we have stand alone projects like LWP, Moo/Moose/Class::Tiny, IO::Async, POE, etc. To my mind I find it odd that you’d use Mojo::Base in a stand alone project (outside of Mojolicious) instead of something like Moo or Class::Tiny (if you want a small Moose workalike). In fact I find it odd you’d write Mojo::Base at all :)
As you pointed out there are bits of Mojolicious that are not so well decoupled as to make it easy to bust apart, which is why that has not happened as it did for a few bits that were trivial to do so (such as the Mojo template system and what became DOM::Tiny). For stand alone event loops I’d prefer IO::Async so don’t need a stand alone version of that. But the fact is there’s a number of things on CPAN using Mojo::Base that could easily just use Moo or something else entirely and that would make the project better decoupled, from the unix point of view.
So really its just a mindset thing. CPAN was founded on the Unix principle of small, decoupled components. Mojolicious appears to be founded on the idea that the whole is greater than the sum of its parts. For people that find beauty in that way of thinking you will find Mojolicious attractive, but you should not be surprised at all that those of us that don’t will notice a handful of nice bits in your framework that we’d like to have independent of it, and we find the idea of installing the entire framework to get it weird and somewhat coercive. Its not really a fork in my mind. There’s no resolution here, this is just the way it is. You’ll have to tolerate it just like we have to tolerate all these modules on CPAN under the Mojo:: namespace just because they use Mojo::Base :)
So when we ‘fork’ its not so much about less code but rather we are following what seems like best practice design principles. And possibly we are trying to avoid the feeling of coercion I mentioned.
@Joel: thanks for the thorough answer and the pointers.
Re ‘discouraging’: if you wanted to communicate your feelings, you were successful. I’m still not sure it was the right tactic to let the feelings slip :)
The article is interesting indeed, but it is someone’s opinion and not a hard and fast rule. There are a couple of additional considerations IMHO:
it seems to talk about forking in the sense of “your thingie does most of what I want, but not exactly. I want to change that”. This seems supported by the suggestions given for discouraging a fork (plugins, inversion of control). In this case, it seems to me that the situation is different, like “I really don’t care now about what your thingie does overall, but there are interesting pieces that might be useful for some problems of mine”. Nothing new, but places Mojolicious in a slightly different position with respect to the article, to the point that a person doing a “fork” might not see that as a fork actually, because they’re trying to take some tiny bit out of a bigger thing (whatever their perception of relative sizes, of course);
it provides some prescriptions about “how to do it”. This is probably what common sense and politeness would suggest, so it’s a matter of personal attitude, to some extent. In your case, either this kind of decision is taken by a considerable amount of rudeness-oriented people, or there might be something else. Maybe people got frustrated in the past and just thought it a waste of time to try and communicate with the core developers - as open as you might think about yourselves, you might want at least to challenge yourselves from time to time about it. I might elaborate on that.
There is probably a mix of reasons. Someone that just thinks that it’s actually possible to have that good function without all the rest (john’s comment seems to be on this tune), and does not actually consider that “a fork”. Some that just got upset with you in the past because you didn’t pass the water (you were listening to the music and didn’t heard, but whatever). Some that didn’t read the article, and might be willing to collaborate later. Some that just got struck by the “I can do that!” lightning. Is it polite to get in touch with you? Definitely. Is it required? No.
Regarding your last consideration, in this I think it’s you to have a wrong perception of relative sizes. If someone had “successfully forked that module” it means that it was possible and, also, that it was a success, which is (at least for me, considering how unsuccessful my modules are) something rare. So why not? Probably it would be the same as DOM::Tiny, as john points out, and you might benefit from a refactoring for free.
This might of course drive some newcomers away. Do I need a user agent, but I don’t need the rest? I’ll go looking for a user agent. Mojolicious seems pretty strong from the visibility point of view in its own area to be quite sure that it would be extremely difficult not to read/hear about it when you need it for what it does overall. This might be different from what happened a few years ago, which is good for the community.
The last thing that comes to mind is that you might want to put some additional clauses in the license in order to make it clear that the link to Mojolicious must be explicit. If I did some “fork” of this type I would surely include a note about it - it just seems polite, but it’s me. You might want to enforce this, so that you can increase the chance that even if using a forked subsystem, a recipient will get to know about Mojolicious and its community. And yes, you will have to chase people at that point, but there’s really little you can do about it beforehand.
Ciao!
The irony of forking things that are themselves forks of sane modules. Joy.