Perl and Me, Part 6: Perl Is Engineering and Art
This is part 6 of an ongoing series where I explore my relationship with Perl. You may wish to begin at the beginning.
This week we look at the question of creativity in programming.
Last week I talked about how the linguistics baked into Perl allow for maximum programmer creativity. But of course all that discussion contains a hidden premise: that allowing for programmer creativity is something desireable. But is it really? Maybe programmer creativity isn’t that great an idea.
Certainly the folks who created and advocate Python seem to believe that. In the book Learning Python there is a sidebar which addresses the question of how Python compares to Perl. This section is entitled “Python Is Engineering, Not Art.”1 On the question of programmer creativity, it has this to say:
But as anyone who has done any substantial code maintenance should be able to attest, freedom of expression is great for art, but lousy for engineering. In engineering, we need a minimal feature set and predictability. In engineering, freedom of expression can lead to maintenance nightmares. As more than one Perl user has confided to me, the result of too much freedom is often code that is much easier to rewrite from scratch than to modify. This is clearly less than ideal.
Well, there certainly is a lot to chew on there. Let’s set aside the self-selection bias entirely (i.e., that the group of people both experienced in Perl and most likely to be talking to a Python trainer are probably those who have failed at being Perl developers and are looking for a more suitable language); it’s obvious and not worth more than noting. Let’s look a bit deeper and see if Mr. Lutz (the author of the book) perhaps has a point after all.
The first bit I have a problem with is his overly generic use of the term “engineering.” Once upon a time we were programmers, and proud of it. Nowadays it seems we need fancier titles, and thus we have become software engineers.2 But there are some radical differences between the field of engineering and the field of computer science. For instance, a rigorous proof of correctness is often achieved in mechanical engineering; in computer science, there is still a lot of debate about whether this is even possible.3 Engineers (at least the ones that I’ve known4) get to approach their problems logically, deconstruct them completely, then reassemble them with rigor, according to structured rules. There’s nothing wrong with solving problems like this. It’s an awesome thing that engineers do, and it’s a very cool way to earn a living.
But it’s not what we do as programmers.
As programmers, we begin with nothing, as a writer begins with a blank page. Oh, sure—we know where we need to end up: those are the specs, the description that the user has given us of what our program needs to do. Sometimes a writer knows how the story must end as well; even if they don’t, they know the general shape of it, at least have some idea of where it’s going. Just like the writer, we create: from nothingness, towards a vaguely defined goal which may very well change many times along the way, with an infinite number of different ways to get there, some short, some long, some elegant, some messy, and there will be pangs of birth, and there will be revisions, and other people will give us their opinions and sometimes they will be right and sometimes they will be about as wrong as it’s possible to be, and there will be blind alleys, and backtracking, and sudden epiphanies of A Better Way, and finally we will present our finished work, proud, knowing it could be better, wishing we could fix just one more thing, tweak just one more bit of code, but knowing that we must deliver it, the business needs it, the users need it, or perhaps we need it, if we’ve written it for ourselves, and we must let it go, for others to enjoy, for others to critique, for others to simply use in a utilitarian way, but it will always be our baby, even if it’s a little bit ugly, oh, sure, we know we could have done better, and we look forward to the opportunity to do better next time, but, butt-ugly or gorgeously elegant, intricately sophisticated or plainly functional, it’s ours, and there’s a little piece of us on the page, or on the screen, or in the bytes, and anyone who knows us will always be able to see us in there if they look closely. That’s what we do as programmers, and it’s messy and it’s glorious and it’s hard as hell and satisfying as all get out, but it ain’t engineering.
I’ve known programmers who approach writing code like engineers approach designing a piece of machinery, of course. They’ve been some of the worst programmers I’ve worked with. They lack flexibility, and they don’t make intuitive leaps, and you have to explain things to them in a very detailed way that tends to be frustrating and time-consuming. Again, there’s nothing wrong with this approach, when applied to a problem that warrants it; I’m sure there are even some programming problems that warrant it ... scientific analysis programs, perhaps, or tricky, high-level academic algorithms, or ... I dunno, there must be something. But I’ve never done that sort of programming. The programming I’ve done has always required creativity: you must be able to start with almost nothing—perhaps just a computer language and some tools like a particular database—and you must be able to see how to code your way to the finish line. Now, there are some mathemeticians and engineers who are creative in this way, definitely,5 and of course there are always going to be those super-geniuses who are good at both approaches, but most of us mere mortals are good at one or the other. I’d make a lousy engineer. I don’t have the patience for all that rigor, and I hate rules. But programming ... I’m good at that.
So perhaps it is true that freedom of expression is lousy for engineering (although I think many of my engineer friends would beg to differ), but that’s utterly irrelevant. Because programming requires creativity. Programming requires freedom of expression. Freedom of expression is crucial to communication. And communication is what it’s all about. Here’s what our Python-lauding author has to say next:
Consider this: when people create a painting or a sculpture, they do so largely for themselves; the prospect of someone else changing their work later doesn’t enter into it. This is a critical difference between art and engineering. When people write software, they are not writing it for themselves. In fact, they are not even writing primarily for the computer. Rather, good programmers know that code is written for the next human being who has to read it in order to maintain or reuse it. If that person cannot understand the code, it’s all but useless in a realistic development scenario. In other words, programming is not about being clever and obscure—it’s about how clearly your program communicates its purpose.
Besides displaying an abysmal ignorance of artists,6 the problem with this passage is that it wants to have it both ways. It says that code should be simple—simplistic, even—so that anyone can understand it. If we were to apply this logic to the English language, for instance, it would be saying that everything you write should be on the level of Dick and Jane. Well, okay: I suppose we can make an argument that, when you write for the lowest common denominator, you do increase the chances of other people understanding what you meant. And, yet, human communication doesn’t use that level, neither in common speech nor in common writing. You don’t write your emails like that, do you? I’m not writing this blog post in that fashion. About the only things we have today that actually do use that level of communication are (some) instruction manuals, and do you find those easy to understand? Hell, even the books that teach our children how to read don’t sound like that any more, thanks to that visionary Dr. Suess, who Wikipedia tells us wrote stories which were “often characterized by imaginative characters, rhyme, and frequent use of anapestic meter.” In other words, they were creative. So our Python devotee here quite rightly identifies that, when we write programs, we should be writing for other humans, but then he doesn’t want us to communicate in the way we commonly communicate with other humans. I’ve puzzled and puzzed till my puzzler is sore, but I can’t make any sense of it.
But surely he has a point when he says that a programmer has to think of something that no other artist does: that, one day, another programmer will likely have to come along and modify their program. Writers and painters and sculptors don’t have to worry about that; their work is typically done alone. Even musicians and filmmakers, who often have to engage in a much more collaborative process than their fellow artists, still don’t have to worry about their work being changed later: once the vinyl is pressed or the film is printed, the work is done. Someone might do a remake one day, perhaps, but that’s nothing to do with the artists in the here and now. Programmers are unique in this aspect, are they not?
Well, almost unique. We could compare programmers to architects, who often have to marry the aesthetically pleasing with the functional, and who definitely have to think about making it easy for the plumbers to get to the pipes when they rust, or the electricians to get to the wires in the walls when the mice chew on them.7 Architects have to be creative, but also approach their projects with a logcial plan, just as we programmers do. But, even without this analogy, if programmers are just like artists in every way except that they have to think about other people maintaining their work, does that necessarily preclude them from being artists? Every artform is different, after all. This is just one of the ways our artform diverges from drama or literature or dance.
There’s only two other bits of this piece that I’ll comment on.8 The first is this sentence:
Because Python’s syntax model almost forces the creation of readable code, Python programs lend themselves more directly to the full software development cycle.9
Now, why would you want to brag about how your language limits expressivity? Since “readable code” is by definition a subjective measurement, what this statement really means is that Python forces you to write code that someone (Python’s creator, I suppose) considers more readable than other code, whether you like it or not, whether it’s appropriate for that situation or not, whether it fits the paradigm you need at the moment or not, no matter what. You’re gonna write like we say so. ‘Cause we said so. Get used to it.
Here’s the last one:
Python programmers can be wildly creative, too, of course, and as we’ll see, the language does offer multiple solutions for some tasks—sometimes even more than it should today, an issue we’ll confront head-on on this book too.
Can they, I wonder? Can Python programmers really be wildly creative? Perhaps. Although note the implicit disapproval of such shenanigans after the dash. Apparently wild creativity is something to be on the lookout for. Be sure and throttle any such behavior, should you come across it.
Now, my experience is not exhaustive. I’ve not worked in every kind of industry, nor on every type of program, nor in every possible language. But I’ve spent over 25 years (about 45%) of my life programming. I’ve been the grunt coder at the bottom, I’ve been the manager in the middle, and I’ve been the man at the top.10 I’ve worked on banking software, voting software, insurance software, advertising software, and web software. I’ve written code from scratch, rewritten code to match older software bug-for-bug, and maintained large codebases of up to two million lines of code, stretching back as far as a decade before I got there. I’ve worked for businesses, for individuals, for the government, and once for a political campaign. And my experience is that I’ve never found a programming problem that didn’t require at least a little bit of creativity to solve, and I’ve never worked with a good programmer who wasn’t a least a bit creative. Maybe your experience is different. Maybe Python is the right language for you. (Although I suspect it isn’t, else you wouldn’t be here ... and that’s our self-selection bias rearing its ugly head.) Or maybe what I’m saying rings true.
Also, I hate doing what other people say. So that’s probably a big part of it too.
Next week, I’ll ponder whether creativity is something that ought to be stressed more when people are taught computer science, and how little I know about how compilers are put together.
1 I have to thank the inestimable Gabor Szabo for bringing this article to my attention. If you want to see the full text of it, you could try Google Books, although it’s somewhat flaky, or you could read it reproduced on a blog post, but that’s not 100% accurate (possibly copied from an older version). I should also note that all instances of italics in my quotes are reproduced from the original.
2 This is mostly just my impression, based on my twenty-five years’ experience in the industry, but, if you’re one of those doubtful people who demands proof for everything, you could ask Stack Overflow or Yahoo! or Chron and they’ll all tell you that a software engineer is better than a programmer. I dunno; personally, I just magically became a software engineer when I switched jobs at one point, but I don’t really feel any different.
3 For instance, many people believe that you can’t prove total correctness for an arbitrary program due to the halting problem. Whereas other people think that as long as you can prove that a program properly implements its specifications, that’s sufficient. Of course, whether such things are feasible even if possible is a whole ‘nother layer of debate.
4 And I’ve known quite a few. In fact, one of my best friends works at NASA, helping to construct satellites and parts for space missions. He’s fond of performing some task and saying “see? it doesn’t take a rocket scientist,” then pausing, and saying, “oh wait ... I guess it did.”
5 On my Other Blog, when I essayed this topic, I mentioned proofs. Mathemeticians who are good at proofs make good programmers, I think.
6 An artist is keenly aware of their audience, even if they choose to disdain them. To say that people create art “largely for themselves” is just wrong-headed.
7 In fact, my personal opinion is that this close correspondence between programming and architecture is what enabled Christopher Alexander’s architectural work to be adapted to computer science so easily, which is why we now have design patterns.
8 Other than to wonder, perhaps rhetorically, if there really isn’t any way to make Python look good other than to bash Perl in comparison. I haven’t reproduced the worst of the Perl-bashing, as it’s deliberately incendiary, and also nothing we haven’t heard before. Anyway, I found myself more mystified than offended. I suppose there’s a particular type of programmer they’re trying to win over, and screw the rest of us. But I still think it’s an odd way to go about it.
9 Remember, emphasis was in the original.
10 Remember that I ran my own company for 12 years. If you were so inclined to read yet another long-ass series of blog posts from me, you could hop over to my Other Blog and check it out.
14 years in California, 25 years in Perl, 34 years in computers, 55 years in bare feet.
Speaking as someone who's spent over a year maintaining, repairing and replacing other people's awful python code, I can say unequivocally there is nothing intrinsic in the python language that makes it easy for the next guy working on your software.
I much prefer Perl- including reading and maintaining others code, but you can write garbage in any language.
Comment became a rant, that generally agrees with Samuel. I disagree with anyone (not necessarily you Buddy) who axiomatically contends that Python is more readable than Perl. It might be prettier, but I bet there are real readability issue that people miss in the name of "pretty".
https://blogs.perl.org/users/joel_berger/2014/01/on-the-relative-readability-of-perl-and-python.html
Also as someone who has written very complex scientific code in Perl, I can say, I NEEDED its flexibility. I don't know how I could have written this in Python given its limitations (notably in closures).
https://github.com/jberger/Physics-UEMColumn
Personally I think slight irregularity in the code promotes readability. When people do layout of text for magazines, the don't use fix width font and make everything the same, because we've learned that one thing that leads to reader fatigue is if all the lines are uniform. I've read code where I got 'highway hypnosis'. I'm not sure is is something that makes code better, although as has been pointed out there is a self selection thing going on.
I however have nothing against Python and feel their community writes good specs (WSGI and so forth). I just don't understand why in a book about programming python they'd feel the need to stop and bash Perl... That seems strange to me. Of course lots of things in the world seem odd to me; I just try to be cool and go with the flow.
Samuel,
That was my feeling as well, but I haven't had the experience in Python that you have, so I didn't want to say something I couldn't back up. :-)
In the end, it comes down to exactly what you said:
which we all know is true, and is in fact pretty much impossible to argue with. Which would lead us to conclude that the author of Learning Python knows it's true too. Which would lead us to wonder why he went to such trouble setting out this argument ...
Joel,
Good article. Your real-world Python experience enabled you to come up with some great concrete counter-examples to what Mr. Lutz was saying.
Well, I certainly didn't mean to imply that I was conceding Mr. Lutz's point that Python was more readable than Perl. I just didn't get into a readability discussion for two reasons:
1. The majority of my Python experience consists of wrestling with a seemingly simple patch to a Python script for about three hours which turned out to be not working because I had mixed spaces with tabs. After that I pretty much vowed: never again. ;-> So, as I said above, I just didn't have the experience to back up any sort of argument along those lines. But you do, so thanks for contributing your article!
2. I don't think that argument is really even necessary to debunk the whole thing. That is, even if we could agree, somehow, that Python was more "readable" than Perl, for some defintion of "readable," for some subset of the population, that doesn't automatically make it better for communicating with other humans, which even Mr. Lutz agrees should be your goal when programming. To continue to beat my dead horse analogy, "see Spot run" might be the most readable sentence in the English language, but that don't make it great communication. ;->
And of course even "pretty" is in the eye of the beholder. Python is what the Jargon File would refer to as a B&D language, but I've always thought of them as OTW languages (because they each purport to show us the One True Way). Funny how there are as many "One True Ways" as there people who will show them to us ...
jnap,
Excellent point.
I know, right? I personally don't have as high an opinion of Python as you do, but other than my little wrestling match (which I just mentioned to Joel, above), I can't really come up with anything negative other than philosophical disagreements. But surely there has to be some other way to make Python sound appealing besides trying to make Perl look bad in comparison?
The first part of your post is beautiful! Thank you!
The second, more rant-ish part of your post, is what I would say if I were as good a write as you: I have not one but two unpublished drafts of a similar deconstruction in my blogs.perl.org account.
I was once rather unbiased about Python and endeavored to learn it because Python was in use where I worked. This specific box in Learning Python poisoned the whole experience for me and I stopped trying shortly after reading it. (In case you were wondering, my contract wasn't renewed, which was certainly for the better.) You hit most of the ideas in my drafts. Here's one more to consider.
In Perl we like to quote Larry and say "There is More than One Way to Do It." The funny thing is that this isn't a design principle: it's a fact. Any real-world programming problem involves some thought about its design. There is usually more than one way to design a solution, and there is usually more than one way to implement your chosen design.
Pretty much any code base in any language with more than 50 lines could be rewritten. The difference between Python and Perl is the granularity at which you can perform those rewrites. In Perl, 90% of the time you can replace a single line at a time with a different expression (think splice instead of a pair of shifts, for example), whereas in Python you probably have to work in chunks of five lines. This difference means that on a line-by-line level, Perl code can exhibit more variety. As jnap points out, line-level variety is not necessarily a bad thing. Furthermore, once we get beyond five lines of code, they are both equally "susceptible" to design variations.
The most outlandish claim, of course, is that this difference in granularity leads to "maintenance nightmares." As everybody knows, Perl's reputation for maintenance nightmares arose because a whole bunch of terrible programmers wrote in Perl in the late 90s and early 2000s. If Python had been the language of choice at that time, Python would have a reputation for maintenance nightmares. I was offended that Lutz would blame this historical accident on the language itself. That such a statement is written in a book for those new to programming, biasing them against an equivalent language from the very beginning, only servers to heighten the offense.
Also, for the record, this box about Python vs Perl comes at the end of the first chapter of the book. The last section of that chapter compares Python to other languages, such as Java and C++. Most of those other comparisons get about a paragraph. Why would Mr. Lutz need 1.5 pages to compare Perl and Python? I suppose one could argue that the differences are subtle yet significant, and require more care to explain. I think that's a bunch of hogwash, given that his arguments don't actually hold up to scrutiny.
Some real interesting and well articulated thoughts by the community...I enjoyed reading all of the posts.
Some thoughts: line-level variety is intrinsic to Perl, just as a rich vocabulary of words, stolen from many different languages permeates English. Having so many ways to say the same things or perform similar actions, I can reveal in those choices, subtleties in the problem I'm trying to solve or the message that I want send. I love that...the ability to accurately and precisely say want I want to say. I often advocate a change to code in a code review simply to have the code more closely reflect the business rule. With a rich vocabulary this is possible.
I think the engineering vs art debate, if you want to call it a debate, is a false choice. The problem space determines to a large degree the approach necessary and in my opinion some solutions are certainly engineered, while others, and one might argue the majority of those we deal with more commonly are not. We need both approaches.
Writing a sorting algorithm for example, can be viewed as an engineering problem that utilizes mathematics, network theory and data analysis to optimize a solution based on a given sent of requirements. There may in fact be an optimal way to traverse a network and one can arrive at that from a rigorous engineering approach.
Other problems that are being solved with computers are more amenable to and require a variety of approaches and insights that only a creative mind can concoct. Trying to find a single engineered solution when a variety of approaches, each with its own efficacy and cost will do, is not the approach taken by software developers today. Since humans are usually ones who evaluate our software, not the laws of physics, one must be in tune with the fickle nature of the public that consumes our software. Of course, when I'm traveling over a bridge, flying in a plane or having a medical procedure performed by a robotic laser, I truly hope there was some engineering involved.
It depends is an acceptable answer.
First, I would never argue that python makes it hard to write bad code. It's very easy to write bad code in any language. I would argue instead that python attempts to make it easier to write good code, by reducing the number of idiosyncrasies a programmer must be informed of to make
I don't understand the benefit to software development in being able to express a meaning in any way but the best way. I don't think the arguments put forth here really contradict that. The problem: Who's best way?
In python's philosophy, the principle of least astonishment ranks higher than every developer being able to choose their "best way." This of course is very subjective, so there will always be people and languages on both sides of the issue.
A python developer is the kind of person who would say don't mix tabs and spaces for indentation in any language. Maybe we're a bit mechanical and rulesy, but that's why we have our own language where everyone likes to play by the same rules. We retain the ultimate form of expressivity here, in that for the most part developers have a lot of choice in what language they will work in. So to each his own.
animatedream: Thanks for your comment! Good to hear from someone on the Python side of things.
> First, I would never argue that python makes it hard to write bad code.
I understand that. The problem is, the person who literally wrote the book on Python made that exact argument.
> I don’t understand the benefit to software development in being able to express a meaning in any way but the best way.
Well, I suppose if there were always exactly one “best way,” not only to all people, but also in all situations, in all industries, for all audiences, in all paradigms, then it might be tough to argue with you there. But just two sentences later in your comment, you identify one of the biggest problems there. The creator of Python is no doubt a very intelligent and educated man. However, to say that what he has decreed as the “best way” is always correct for everyone is going a bit far, I think, no matter how much of a fan you may be. Am I wrong? But the problem goes deeper: even if we were somehow to agree that his “best way” is best for everyone, can we really agree that it’s the “best way” all the time? The fact of the matter is, different situations call for different techniques, there is no silver bullet, and anything that limits my ability to adapt my expressivity to the “best way” for a given situation is going to cause my programs to suffer.
> In python’s philosophy, the principle of least astonishment ranks higher than every developer being able to choose their “best way.”
Unfortunately, this is just as subjective as the former. I can assure you that I was damned astonished when I figured out that the tabs in my Python program counted as a different indentation level, thereby bumping those lines out of the preceding
ifblock. What astonishes one coder will be old hat to another. Therefore, “least astonishment” is unfortunately just as subjective as “intuitive” (or “best way”).> A python developer is the kind of person who would say don’t mix tabs and spaces for indentation in any language.
Fair ... perhaps. Trust me when I tell you that I never meant to mix them at all. I broke Python’s “rule” completely accidentally, but I was punished all the same. The problem with rules is that they tend to be reductive. I essayed this general topic on my Other Blog, where I used English grammar as my illustration, but the principle applies equally well here: life just doesn’t lend itself to a simple set of rules. Life is complex, and messy. It needs guiding principles, but not rules. Rules only make it harder when you inevitably have to break them.
> So to each his own.
That’s my attitude as well. Please note that I’m not realy attacking Python here. I’m attacking a Python author, and perhaps to a certain extent an embedded Python cultural value, which themselves attack Perl. I don’t really care for Python personally, but I’m happy to let Python programmers program in whatever they like. I just get a little irked when they attack Perl without (apparently) a proper understanding of what they’re bashing.