Why Doesn't the BBC Upgrade Their Software Often Enough?
There are three things that I really love about Perl:
- Moose
- Catalyst
- DBIx-Class
That's where the problem starts. It's not really a BBC problem, though. It's a problem for all "enterprise" customers.
I just tried to upgrade Moose and got the following message:
***
This version of Moose conflicts with the version of
Catalyst (5.80007) you have installed.
You will need to upgrade Catalyst after installing
this version of Moose.
***
***
This version of Moose conflicts with the version of
MooseX::AttributeHelpers (0.2) you have installed.
You will need to upgrade MooseX::AttributeHelpers after installing
this version of Moose.
***
***
This version of Moose conflicts with the version of
MooseX::MethodAttributes (0.14) you have installed.
You will need to upgrade MooseX::MethodAttributes after installing
this version of Moose.
***
***
This version of Moose conflicts with the version of
MooseX::Types (0.16) you have installed.
You will need to upgrade MooseX::Types after installing
this version of Moose.
***
I'm doing this for a personal project, Veure, but when I see message like that, there's an issue. Specifically, enterprise customers.
The Perl community is largely a community of JFDI hackers (thanks, Jesse!). However, corporate users have a different issue. We have admins. Lots of them. We have projects. Lots of them. We have boxes. Lots of them. We have operating systems. Lots of them. We have versions of Perl. Lots of them. We have databases. Lots of them. We have modules installed. Lots of them. We have tests. Some of them.
Now take those "lots" and start doing some imaginary math. We have very, very busy admins. As a result, we have to submit a reason for every change request. On a personal project, I'm quite happy to upgrade this code because there's no chance I'm going to impact others. In a corporate environment, things start to become more tricky. Admins would like to see this:
- One operating system
- One programming language
- One version of said programming language
- One database
- One standard set of modules
- One box (hah!)
- ... and so on
I know that some people dismiss this with comments like "your admins must not be very good" or something like that, but small systems turn into large systems which turn into huge systems. Along the way, admins leave, new admins start. They leave. More admins start. Critical business knowledge is lost and rediscovered. Business needs change. Systems evolve in ways that the previous architecture doesn't support.
The BBC is not a small company and frankly, I've loved the quality of people that I work with. These aren't idiots, but they are busy and requirement management is often akin to skeet shooting while blindfolded. Even that isn't as silly as it sounds because sometimes the law changes and we must rapidly adopt to it. Now that London is hosting the 2012 Olympics with its Bart-Simpson receiving a censored logo, we have to address this. Other TV stations may change direction and decide to work with the BBC or against the BBC. We don't know their decisions in advance and this again causes us to rethink what we do. Heck, I could (but can't, sorry), tell you fascinating stories about challenges we constantly face with iPlayer (Americans: think of it as the British Hulu) which force us to rethink technology solutions all the time.
In other words, what we do is hard. Very hard. People who don't work here have no idea the technical challenges we face.
Recently I proposed fixing an issue in our code base where package and filenames didn't match. As a result, multiple filenames contained the same package. This was apparently a bug workaround, but now it's a time bomb. Fixing the issue was denied. Why? It's not a problem yet, but getting new features out the door is a problem.
It's not just obstinate management. Part of the issue involves possibly upgrading to a new version of Catalyst. However, we can't just tell the admins "upgrade this". We have to tell them "upgrade this and here's why". That's because they're busy and if every one of our many, many teams was upgrading everything all the time, without having a concrete business case, fewer features would get out the door. Thus, I wanted to fix a potential bug (one which modern versions of these modules sensibly warn about), but I couldn't demonstrate that it was actually causing a problem. Thus, my business case was weakened vis-a-vis the fact that we're preparing for the Olympics.
Olympics versus "not a bug". I report. You decide.
I'm not taking a stand one way or the other. It's very, very important that these projects address issues they have and appeal to developers who want these issues addressed. However, as I've encountered time and time again in large environments, upgrading is problematic. As a result, we tend not to upgrade until we need a particular feature and can make a business case for it. Thus, for the best projects with the most active developers, individually we love it (I certainly do), but at a corporate level, it becomes a sysadmin nightmare. That's because we tend not to upgrade unless we absolutely have to but our software is so out of date that we're practically shutting down our work to upgrade everything we need, including rewriting large parts of our software to match. Meanwhile, we have to make sure that our publication latency is reduced as much as we can when journalists covering the Olympics or World Cup want to report something live.
I've successfully upgraded "not a bug" modules on the grounds that we need to ensure that when we must upgrade, we're not so far behind the curve that the upgrade is very difficult. I've not done this often, though. Our Catalyst is a year old (I think) and we've no plans to upgrade it because it's "not a bug".
There is a terribly tension between those who need stability and those who need new features. I don't have the right answer here, but the problem is not as simple as some would suggest.
Freelance Perl/Testing/Agile consultant and trainer. See http://www.allaroundtheworld.fr/ for our services. If you have a problem with Perl, we will solve it for you.
And don't forget to buy my book! http://www.amazon.com/Beginning-Perl-Curtis-Poe/dp/1118013840/
We've just come across this recently, having not upgraded for several years, it is a major project.
Having said that using local::lib to install into a test directory and then run our code base against has made the process far easier than the last time we took this on.
So totally agree, it's a big problem, although one that isn't as bad (local::lib, better/more CPAN testers, better testing of Perl modules as a whole, better testing practices of your own code - hopefully!) as it used to be.
I think projects like Strawberry Perl Professional are a fantastic start in helping, even if not solving this sort of issue.
It will be interesting to see what solutions do come as I know lots of people are interested in trying to solve this (Shopwright and others that are in development).
I think Ranguard has the right idea here… as I was reading this I was thinking the right approach would be to isolate the perls running those apps from each other, so that they can each change without affecting each other. Perrin Harkins has been advocating this approach for years. It should also be fairly easy to come up with a standardised way for sysadmins to manage these installations; thereafter, it would be much less of an issue to accommodate changes needed by each.
Having one perl per app may be the right direction, but still sysadmins like to use software packages. Doing a security update for one of the packages would be a nightmare if you now have 50 perls (one per app) running on their own. Each would have to be updated. That would need developers time.
I personally do not prefer CPAN for any deployments, but rather binary packages (.deb or .rpm). This is due to too many hickups when doing simple installation tasks where sometimes tests fail. So with binary packages the base perl could be the perl delivered with the linux distributions of your choice, and the app could pack any modules (even temporary some modules with security updates) going along in the app lib path and build a package for it. The system state is known (e.g. Debian Etch or something), so packages do know whats on the system (especially third party libs) and dependency management can be automated here.
The problem with binaries is sometimes there isn't and getting them is much more of a pain than CPAN ever was.
Yes, it's hard with RPM, when you do not stick to the base OS. You would have to trust rpmforge to get access to more perl packages here.
Still I actually tried to point out that for enterprise use, you can always build the software packages yourself and base it on packages available in the standard repository for your distribution. (And in the enterprise sector usually you only use distros that have maintained repositories). That way no exotic repositories are required at all.
Of course doing the same for source based dists (like FreeBSD, Gentoo) is similar.
You can do it that way, sure. I just find it easier to use CPAN. But everyone has their opinion on what's best.
A great post on a huge problem, and we see another side to this. We're building a Catalyst product which has to be installed by around 10 different organizations, each of which outsources to shops with different protocols. In one case, we had to *heavily* defend use of open source. In others, we just needed detailed scripts, but in all cases, we also needed to make everything upgradeable in a way that our system remains guaranteed to work.
This is not just a Perl problem. Java has the same issue (JAR hell), and Maven was a very elegant solution to managing version interdependencies, at the cost of a very heavy packaging description and standardized repositories of timestamped components as well as versioned. However, Java has the advantage of a more or less sealed virtual machine, so OS differences are much less important. In Perl, they are critical.
In the end, we just stuffed an entire Perl into our install directory through using standard installers. That way, what we provide is a simple installable component, easy to explain to even the most rule-based outsourcing engineers. Basically, we act as if Perl is inside out system, rather than something we depend on. This is not great for system sharing and performance, but does give us the flexibility we need.
The budgeting issues you describe sound like "everything must be urgent" and that nothing is allocated for maintenance. Unfortunately, software doesn't punish you quite so dramatically for neglecting maintenance as e.g. an internal combustion engine.
You should be treating dependencies as part of your application in terms of maintenance and deployment. Sharing common dependencies across multiple applications is just an optimization of disk space (and other resources.) But you have to treat it as an optimization.
Not that it's going to be super-easy, but looking at it like that might help you crunch the cost/benefit numbers. If you're treating dependencies as "the system", that makes them the domain of the sysadmin, but are they not at least 50% of the program? (Cut your horse in half and send it off with two riders?)
This is the reason we created the WebGUI Runtime Environment (WRE) for WebGUI. We solve all of the incompatibilities up front for our customers, and distribute it as a whole complete, tested, working system. It's a lot of work, but it allows our customers to use up to date software and dependencies, without the fears you mention.
To me there are basically two solutions to this problem. One is to go the way we did with the WRE. If you have a reasonably complex system the extra overhead is worth it vs complaints and nobody upgrading. The alternative is to not distribute software. Instead, make your software as a hosted SaaS model, which means it's either web based, or you distribute a simple client application to access it. Of course, not all software can go this route, which is where the WRE model comes in.
@JT: you know, I keep meaning to download WebGUI because it sounds like a very interesting product and I hear good things about it. Sounds like you've got a great setup there.