$3M says Perl5 needs a new major version number
Yesterday, Ovid started his discussion about moving the major version of Perl 5 to Perl 7. You know what else happened that day? Continuum Analytics won $3M from DARPA to undertake a huge renovation to NumPy. Three. Million. Dollars. Not for Python. For an extension for Python. Continuum plans to add all kinds of capabilities, but bear in mind that PDL already possesses at least one of those, namely built-in support for missing data. From the technical standpoint, we were already ahead, and somebody else won $3M.
NumPy is very well run, and very well organized, and there are many more libraries available for NumPy than there are for PDL. Those dollars are very likely to be well spent. I will not argue that this was a bad decision by DARPA.
But think about it. Do you remember just how amazing it was when Craigslist gave the Perl Foundation $100,000? Or when Booking.com gave the Perl Foundation 100,000 Euros? Now, multiply that by 30.
chromatic replied to Ovid's post by saying that we should write kick-ass software. Well, I have. I wrote a plotting library that I think is really good. It addresses one of the most important shortcomings of PDL, namely the lack of a default plotting library. It still needs a ton of work, but it fast and functional, today, and it has a strong foundation. The problem? Nobody uses the library besides me. Why? Because they already went through the labor of setting up some other plotting library, like PGPLOT, PLplot, or Gnuplot. That took a lot of effort, and climbing the learning curve took time for them. That means that the only way to really get users is to get new users. Now, tell me: where are new users coming from?
I repeat, where are new users coming from?
Potential new users ask their friends how they get their work done. Their friends say Matlab, or R, or Python. Potential new users are probably more likely to try using Ruby than they are to try using Perl because "it's cool." Everybody thinks Perl has gone the way of Tcl. And why shouldn't they? Tcl's latest major version is newer than Perl's.
What would I like? I'd like to see Perl 5.18+Moo released as Perl 7. Make it big. Make a splash. People will be wowed by the new syntax---even better than Python's or Ruby's OO sugar---and will be happy when p5mop finally hits and delivers a major speed-up.
Please, stop saying that "if you build it, they will come." We need hype.
This is my blog about numerical computing with Perl.
The NumPy project is an investment in specific improvements. Who says that the organizations giving significant cash to TPF wouldn't shell out a lot more if there was a ambitious but specific plan that would bring major benefits for them? I know at least one that could go do that, but I know of no big plans that would give them sufficient return on investment.
The 100kEUR from Booking.com funded Dave Mitchell and Nicholas Clark. It had an excellent ROI for Booking.com. But that plan does not scale to 30x.
I think you've pointed out a major part of the problem. We fail to exploit our existing network to raise money. Now, raising big money is not just about having the money available for work, it means that the people that gave the money have a stake, an interest in the project. They want the project to succeed, otherwise they money is poorly spent. So they help out in all sorts of ways, not just giving money.
What would it take to put together a $1 Million dollar fundraiser for Perl? How many companies using Perl could be leaned on to help?
Raising big money like that is definitely one way to get attention!
I fully agree.
Just see the vitality of CPAN to be sure that we only need visibility, attention. And I can not image a better way to do it that releasing Perl 7.
I am good with the changing it to "Perl YYYY.M". This gets us around the major version number perception elegantly IMO.
That doesn't mean new stuff can't go in or that old stuff is broken. It *can* mean that but it doesn't have to.
Visibility is needed, get PDL on GitHub, blog about PDL with Hadoop, and R, and other "hot" data science / data analysis tech, and post it on Hacker News. There's evidently still enough interest in Perl to get an article about a new version number on the front page of HN.
Correction, evidently PDL is on Github, but I think the rest would apply.
Hi
Write an article comparing your plotting package with the others. A bit of work? Undoubtedly. And yours does not have to surpass the others in all ways. The point is to give you something to brag about.
John, 1M$ is not very useful even if you manage to raise it. Having a plan for how you're going to spend it well is what gives the value.
In other words: 1M$ is a lot of money for just the pure benefit marketing in the sense of saying "look, we could raise a ton of money".
Come up with a plan that benefits people, and you can find money to fund it. TPF didn't have to bend over backwards to raise a quarter of a million for the core maintenance fund.
@Steffen, I can think of multiple projects for improving Perl's speed for data analysis. These include extensions and improvements to PDL as well as writing (finishing) Perl bindings for the Tiny C Compiler and pushing development on FFI::Raw. If I could get a full grant for doing this sort of work (i.e. something that paid a salary for multiple months) I would be absolutely delighted. However, I do not think these ideas are worth the small-dollar grants; I may as well do it out of my own intrinsic sense of motivation, which is what I have planned to do all along. Where would I begin such a discussion?
@john, I think this would be a fine idea. However, I remember Karen Pauley mentioning that the Perl Foundation once tried a large grant system and it didn't work out so well. I think that potential donors would be very cautious to contribute to a new grant scheme unless those issues were addressed.
@Delon, blogging sucks a *huge* amount of my time. I would love to blog about the stuff I do, but then I would have *no* time to actually write code. It's a terrible trade off. For example, I resolved a couple of weeks ago to try to regularly blog about different features in my plotting library, and you can see how well that has gone.
@Ron, note what I said to Delon, then multiply the already inordinate amount of time by a factor of 3 or more for each plotting library's learning curve. I don't have time to do that. I would *much* rather others help me out in that endeavor, hence the original post. :-)
@David: I've used PDL in the past, but am not a current user. It's a fine tool! I simply have no need for it.
At work, Perl code uses the CPU time of millions and millions of EUR worth of hardware and spends most of its CPU time on general OO overhead such as method calls as well as string operations (lots of concatenation, lots of UTF8-related operations). It also spends lots of time on hash manipulation. Speeding up any of those generally would be worth real money for us. Another avenue that would be really rather useful is finding a more efficient way of calling from C back into Perl. Perl => C is relatively fast, C => Perl is dog-slow. This would allow us to mesh C and Perl functions at a more fine-grained level for better performance. Furthermore, making the op-tree do full COW so that we can modify it at run-time even in an ithread-enabled perl would be a nice stepping stone towards being able to do more experimentation with optimizations.
Wrt. FFIs, I'm extremely skeptical. FFIs are so often purported to be vastly superior to writing C code against the Perl (or Python, or whatever) API, yet the showcases are always just the most simplistic examples.
Knowing David as I do, I highly doubt that "simplistic" use cases are all he has in mind.