Writing Large Systems in Perl is a Privilege Not A Right
I have had an experience which has caused me to reevaluate my relationship with Perl. Members of my local PM have also had to deal with this for the last six months or so. After a post-pub conversation with pozorvlak the other night, I have come the conclusion that while writing large systems in Perl is possible and sometimes even desirable, this is a privilege of showing that you can be disciplined and not a right.
Perl 5, at least, allows you to have many different ways of attacking a programming problem. This can be a benefit but it comes with two large problems. First, if your programmer is not disciplined, Perl is almost guaranteed to give you spaghetti code. Second, if your programmer is inexperienced, Perl will give you something that works but will be unmaintainable by either the programmer themselves or an experienced programmer. Perl does not teach nor require good programming practice.
When the programming problem is small, this does not usually cause unsolvable problems. However, as the problem/system grows and these systems get into production, the lack of the enforcement of instilling proper large scale programming techniques leaves Perl in the dust when thinking of other programming languages such as Java or Ocaml.
Now, yes, I do realize that you can write spaghetti code in any language. If you let an inexperienced programmer without proper mentoring at any language, you will get an unmanageable mess.
I guess what I am trying to say is Perl is great for people who “just want it to work” and do not describe themselves as programmers. Perl is also great for those who are experienced and disciplined programmers who want to build large systems. Where Perl breaks down is the “squishy middle” of people. Remember most people are lazy and undisciplined. Believe me, I am very lazy and undisciplined so I know whereof I speak. Perl allows this to an extreme when building large systems and this causes them to become, if not tackled directly at a management level, unmanageable messes of code that “works” but it becomes exponentially more dangerous to add anything to it. Even when you get disciplined coders in to work on it, the mess left behind by previous generations of code that cannot be improved.
The idea of test driven development is a community driven answer to this problem. “Write tests!”, they say. Again, this assumes that the programmer is disciplined. If not, the tests can become as bad as the code itself. Test driven development is a virtuous activity and should be integrated in to any development situation in any language. However, this will not solve the inherent problems within the language itself. This also assumes that the programmer’s management cares about cleaning up code messes. For an example of this, see all COBAL still infesting the world. Once something is written, management generally could not give a toss whether it is good or not and they only care when something breaks. One interesting research project would be to have a code analysis system that could give management a percentage chance of code failing. Basically, you cannot talk to management without some kind of “metric”. Logic does not seem to penetrate their collective skulls.
A secondary thing that I want to bring up is that Perl is slow. It is very, very slow. Now that may not bother many but it does bother me from time to time. I would like something that is a more efficient usage of resources. Mostly because using less clock cycles is bound to use less power and is thus more “green”. My grandparents lived through the last depression and learned the valuable lesson of “waste not; want not” and I think we should seriously consider applying that to programming.
So, I guess my want is that if I were doing programming interviews for Perl jobs, I would want to see code examples that were not in Perl. I would want to see proper use of modularity, function/class use, and other examples that would show me that the person under consideration is disciplined in their use of all the systems available in a programming language to write good, large systems.
Anyway, I guess that is my rant for the day. A programming language should enforce certain kinds of discipline on the programmer. Perl 5 does not do this well enough for me to feel comfortable writing large systems in it.
Celticist, Computer Scientist, Nerd, sometimes a poet…
In any other language mistakes and problems like
bad architectural decisions never corrected
overly complicated and uber-abstracted OO
mis-understood or mindlessly (randomly) applied Design Pattern
never designed to scale but suddenly a thousandfold of entities to handle
lack of speed not caused by the language but by bottlenecks in design
plainly haven’t written any tests to speak of
horribly “designed” APIs
never really refactored but constantly added new features under pressure “because you were a small company and the customer threw money at you…”
and so on can happen all the same as in an application written in Perl.
I’ve seen large applications written in Perl, PHP, Java and ASP. I have yet to see any language that can keep programmers from writing broken systems.
Perl has nothing to do with development principles like abstraction, separation of concern, when to use a procedural, an object oriented, or a functional approach. If one doesn’t understand these, there isn’t anything a language can do to keep one from building systems the wrong way.
This debate is in my opinion nothing more than a mixup of two separate issues: The first issue is language complexity, the second is the design of long-lasting large applications. Basing your large-scale solution on some specificness of the language makes fragile systems, no matter in what language. Nice examples of this are PHP’s dark days of magic quotes and implicit variables.
I simply believe that even if at the most basic level of programming language design there might only be one way to do some operations, as soon as you’re writing anything more complex than what you would in a shell script, there’ll be lots of ways. If people can’t decide when to use ‘for’ and when to use ‘map’, how will they cope when they’re faced with the problem of designing a reusable UI system?
Most code I have seen is unnecessarily convoluted, by a large factor. In more rigid languages, this convolution just manifests as over-engineered spaghetti architecture, rather than under-engineered spaghetti code.
Good programmers will write good code no matter what language (or tools, or environment) you give them. They will find a way to make Brainfuck readable if that’s what they are being forced to work with. Bad programmers will write code badly, no matter what language (or tools, or environment) you give them.
Unfortunately, good programmers can’t be grown on trees. So solace is sought in the design of languages, supposed to “force discipline and teach good practices” on the mediocre masses, even though the true cause cannot be addressed that way, if any.
So it goes.
(Don’t get me wrong. It is possible to design languages to be either more or less helpful to programmers. But essentially the only way in which they can matter is making it easier – to write code, and especially to spot mistakes once made. If you try to make them teach discipline, people will merely learn to misapply that discipline.)
I think that you give too much credit to Test Driven Development. In my opinion, it’s more likely to produce poorly designed good. It’s goal is merely to ensure that the things that you code give the right answers, but since it is a bottom-up technique where you focus on minutae, you’re likely to end up with a poor design. Not only that, when you need to change an architecture, most of your testing time is thrown away.
TDD works when you start with a mostly complete specification and it’s down to a simple matter of programming. You already know what most things are going to look like and how the components interact with each other. However, unless your pumping out the same sort of code over and over again (and that implies you aren’t designing properly for re-use), you probably don’t have a good design in place.
TDD will not help you avoid poor designs, and that’s most of the problem you are talking about.
And, if you think Perl spaghetti code is bad, you should see some of the large systems in Java. They have real books devoted to programming-in-the-large that should drive people crazy. I think programming practices in Java are over-engineered. Everyone who knows what they were doing seems to have escaped to Ruby, leaving the poor designers and architecture astronauts behind. :)
The concepts of “bad code” and “bad design” are simply a matter of perspective. I’ve worked with plenty of developers who hate any code they didn’t write, and therefore their solution is to always rewrite stuff when they get their hands on it. I hate everything I wrote yesterday, because I’m a better programmer today than I was yesterday. Does that mean it’s bad code? No. How do you define “bad” and by who’s judgement?
My points are these:
1) Code can always be improved.
2) Designs can’t always be improved, but they can often be factored out of existence in future iterations.
3) If it does the job it was intended to do, then that’s some pretty damn good code if you ask me.
I’ve worked on some pretty horrible large systems in static languages - one of them in a language that was explicitly designed to teach “good” practice! So no, I don’t think that language features (or their lack) can make undisciplined programmers write readable, maintainable large systems.
I’m increasingly wondering if the Haskell people aren’t on to something with their horror of mutable state, though.
Let’s go past the ‘you can write spaghetti in any language’ and ‘Perl is line noise’ and try to be more specific. What do you think makes a language more suitable for big systems? Can we add this to Perl? If not to the core then perhaps through libraries or maybe code inspectors or just social practice? One thing often mentioned is static typing - this is not very probable that we’ll have it in Perl5 (but it is there in Perl6). Other things?
I personally think that the Perl community is too automatic in rejecting the ‘programming patterns’ stuff. It is true that many of these patterns don’t suit well dynamic languages, and that using them blindly leads to over-engineering - but I think there is a useful core in there, and maybe for dynamic languages we could make up other patterns. Fortunately on the other hand the Perl community is never a monolith - and you can see people experimenting with stuff like Inversion of Control (Bread::Board anyone?).
Ok, I didn’t think my sorry little rant would cause so many responses. I figured that it would be put in the “oh god here he goes again” category. Anyway, I will attempt to answer some of your themes.
First, however, I want to address the fact that I am not a disciplined programmer and I would be the first to admit that I am part of that “squishy middle” that I was discussing in the original post. I am a lazy, lazy man as many will attest. I can also write some bug ridden code.
I think this is a good observation and I think that we need as a programming community to identify good programmers and have them mentor those who may be in the “squishy middle”. I think the value of mentoring and pair programming is underestimated. As Hackathons have shown, programming (like most everything Human) is a social activity. The idea of the lone cowboy hacker is really over rated. Human beings are, by their very nature, social creatures and they need to be treated as such even in the most technical of fields like programming.
Yes, I have noticed this especially with Selenium testing. When the UI of a web app is still in flux, the tests are actually a time-sink that you may not get back. When the system is stable and only fixes and patches are being applied, it is fine. When you make major changes though, you end up throwing away lots of time and tests.
I think this is more about style than substance. I often make fun of the Factory-Factory-PlanningCommittee-Factory-MotherMayIHaveAnObjectPlease style often applied in Java. I think there are clean, robust ways of using Java that don’t involve this style. However, I haven’t written in Java in a long, long time.
Yes, the “not invented here” syndrome. I have had to deal with it on several occasions. I think the best thing is go get the library that does the job. If it breaks or doesn’t have the feature that you need and you find it a mess, it is just being a good citizen to attempt to fix it up. Open Source is a great advance in this regard.
Yes, I think I remember you telling me this story. I think it was a lack of leadership (leaders are also cannot be grown on trees unfortunately) on the part of management.
I tend to agree but this might have more to do with the social milieu that I inhabit these days.
Indeed, and one of the reasons that I stick around even if I have my doubts about the language itself.
I think it was a lack of leadership (leaders are also cannot be grown on trees unfortunately) on the part of management.
It was more to do with the CEO and the project lead both being Real Programmers, who didn’t see the problem :-(
Ah. :-(
Gauss said something along the lines that it was the notions that are important, not the notation. It may be the case that the notation may affect the notions but is it true to say this lessens with continued exposure to the notation?
Certainly I don’t think it is controversial to suggest that the notions will become more useful the more work you put in to them. Perl gives you the flexibility to refine your notions against any philosophy/paradigm you choose to adopt. Sure, you’ll make mistakes as you go but the notation is not to blame for that.