PDL: Episode VI - a New Book
The title is clickbait. I ran short of time this week and am ~~recycling~~^Wconsolidating comments, replies and thoughts. Let's talk about Books!
I would love a new PDL Book. One that's completely different from the original to maximize the surface of engagement to a new audience. As a "sequel", It would have the advantage of being able to refer the reader to the first book for longer explanations and be able to jump right into how to solve significant problems. brian d foy has just finished his Mojolicious book, so I bet he's got loads of free time on his hands. (although I remember him in the middle of writing it in 2018, so you may have to wait a bit)
The premise behind the PDL Book is that it takes you on a tour of the features, trying to expose the useful parts as quickly as possible and yet still give you the Full Monty. In today's world, many coders, including yours truly, are unwilling adherents of SOOP (Stack Overflow Oriented Programming) with the attention span of 5-year olds who want to dive into the middle of a book and work their way backwards trying to understand the solution. I think it's an issue of motivation and, honestly, I'm surprised you've read this far before going off and checking your phone. :)
Never the less, people are still continuing to develop the first book. I think that the chapter on Installing PDL is vital to overcoming the barrier to entry. It has always been a sticking point and is off-putting if an installation doesn't work first time.
One of my favourite books for it's format is the Bad Data Handbook, 18 chapters of stories full of obstacles encountered, cursed at and eventually overcome. It makes you realize that Wisdom is not granted from on high, but hard-won by those who bear the burden of knowledge. I like to hear as much about what went wrong, both as a cautionary tale and as a ripping yarn. It keeps my attention to the end.
Taking as my motto, be the trouble you want to see in the world, these posts will be my contribution to a new PDL Book. With enough posts to make a couple of chapters, we could fire up a repo where those willing to write, edit, comment or code shall give this project the attention that it deserves. Not that it necessarily should be only PDL. Joe Kline's panel discussion in 2014 debated the question of whether Perl was a suitable language for science and one of the conclusions was as important to include talks about what things people were doing with Perl as it was to talk about how to do it. It helps to inform developers on which directions to consider with their code.
After some thought, I am also starting to get behind the idea of a "cookbook" format for Science with Perl. This might be a question for Dave Cross to weigh in on, as he's offered some assistance on the publishing process
Zubenel - if you can find the time to write about the trials and tribulations of getting PDL to install and run under Windows, successful or not, I would see that as a chapter in itself. There are so many niggles to work through in an ever-changing landscape. Did you manage to get it to install with Prima for displaying graphics?
Re-reading this post before submitting, I'm getting a warm glow from referring to the current PDL Book as the first book. It feels right.
Does anybody have a particular itch that you'd like to scratch with PDL and haven't had the time just yet?
I am a Freelance Scientist** and Perl is my Igor.
Indeed! I have PDL installed on a Raspberry Pi, which is connected to a telescope. The camera returns a fits file. I'd love to be able to do a quick preview of the image.
I think, to make it more viewable, it'll need a little light correction. A histogram, some math, save as jpg.
There is a lot there I dont know how to do yet. I will keep the original fits file, I'd just like to make a little on screen preview
I also plan to capture many-many long exposure's. It would be neat to stack them for the preview as well.
tldr: histogram, light correct (probably not even the right term), stack fits files, create jpg preview for web page
If you're automating file processing over several nights, have a look at PDL::DiskCache to treat your files as a tied array.
This gem was a throw-away comment by David Mertens in his talk Intro to PDL
Documentation also at http://pdl.perl.org/future/?docs=DiskCache&title=PDL::DiskCache
Enkidu, as I am only Perl beginner I do not think that my trials will be very useful and could make an entire chapter. Nevertheless I might try to find time to write about it but I do not know where to post it. Maybe I could send it to you? Regarding the book it seems a good idea. IMO it could have chapters with different strategies how to deal with data as: how to import it, clean and transform it, run exploratory analysis, visualize, create regression models and interpret them.
Zubenel, sometimes I blog my frustrations out in the hopes that it will be useful to someone in the future. Write what you can no matter how useful you think it could be and we can decide how and where to use it later.
As an author, you won't get better unless you write and I will do my utmost to enable that! If you can reply here, I would think you can also post here on blogs.perl.org (look for the Post link at the top). The locals are quite polite, so it's a good place to get started. It's not the best interface in the world, so if you get frustrated, I'll post it for you. I know some people with wordpress accounts, if that's your thing.
I'd love to write a PDL book. It's definitely on my list of interesting ideas.
However, good books take quite a bit of time and effort. It's not a matter of simply writing. Most of the pain is in sequencing—what's the minimal amount someone needs to know to do the next step?
My Learning Perl 6 book raised $40k on Kickstarter. I got about $37k after fees, and spent about half of that on buying the books wholesale to ship to people. I don't think I'd do a niche book like that for that little again because that's all the money I'd ever get. With Learning Perl, my royalties are still decent and that's been paying off for years.
Some people have mentioned a TPF grant, and they've affirmatively told me that they want to support something I'd do. However, their cap is $10k. I can make about that much in month of contract work. Although I don't write because it's making me rich, I do have bills to pay just like everyone else.
For anyone thinking about doing this, my advice is to ignore the mechanics. Forget about the tools, websites, whatever. Plenty of enthusiastic people start projects by setting up GitHub accounts, blogs, project boards, and whatnot. We can publish stuff on Perl.com as a start. The goal is to get content, not create a content management system (although that's more fun).
Instead, write half the book. Do that in Word, vim, whatever, in any format that you care to use. It doesn't matter because it can always be converted to something else. I mostly write in PseudoPod and have converters to Markdown, LaTex, DocBook, WordML, and so on (based on whatever the publisher wants at the end). Forget about the process for now, because that's really easy later when you get stuck or start to hate writing: drop into some fun ETL stuff (and that's all it is).
There are three sorts of books you get to write, generally: references, tutorials, or cookbooks. Programming Perl is a reference book and was the basis for a lot of the original Perl documentation. But, it's also literally at the limit of its physical binding and a massive undertaking. Learning Perl is a tutorial, in which we carefully guide the reader to our goal and during which we expect readers to go in order. Effective Perl Programming is a cookbook in which you can drop into any chapter and start reading. There's a grouping and no sequence. These are easy to write because I allow myself to presume a level of knowledge that lets me ignore almost everything I have to think about in the other two sorts of books.
When you are writing, figure out what you need to cover. So, pick something in the middle and just write the chapter. Then look at the chapter and list the assumptions you made:
Now, divide those concerns into what you can cover in the book and what you can defer. For example, you can punt to Learning Perl or Intermediate Perl for many things, so you don't have to reteach these. You note the level of proficiency you expect upfront, give them some pointers, and move on. Where you draw the line depends on your audience.
For example, in Mojo Web Clients, I covered several basic Perl features that people may not have used if they've been programming Perl for a long time, such as postfix dereferencing and subroutine signatures. I wanted to use those features and they don't appear in any other Perl books (that I know about). I also covered some basic Mojo utility modules so I could use them without explanation later. I think both of these chapters are in the free sample on LeanPub.
After that, you have the list of things you need to include in the book. Your job as the author is to present those in an order and in a way that makes sense for the audience. That is probably not the way that you want to think about them. Randal Schwartz was always insistent that we never suddenly start using some feature we hadn't explained yet, and that approach generally works. This is painful indeed. You really beat yourself up because you have to justify everything you do. Every feature in every expression has to serve the point you are want to highlight, and feeds back into that list of assumptions. Things you do with no thought now become major points of contention because you can't casually drop those things on the reader and expect them to separate the importance of the point from its concrete demonstration.
Oh, hey, this is a comment, I guess, so I should probably stop. Finally, I'll say that I'd love to write a PDL book and if someone with deep pockets wants to fund it, lets talk.
I entirely agree. Content is key and the more I dig through the documentation, the more I see there is to write about. The "loads of free time" quip was tongue-in-cheek. Other than a lottery win, the one way I see a book getting written is for 20 people to think it'll be easy and then be too stubborn to stop. If someone out there does think they have what it takes to get a book over the line, I could be persuaded to release my stuff under CC-BY just for the joy of seeing something new pushed out there.
It warmed my heart when I recently tripped over 40 pages on the internals of PDL to find it was written only 3 years ago. It's still an active project with a lot to offer today's audience. I guess that means I should get another post out.