Day 9: Set operations with files (App::setop)

About the series: perlancar's 2014 Advent Calendar: Introduction to a selection of 24 modules which I published in 2014. Table of contents.

Have you ever encountered one or all of these situations?

1) You have two or ten log files, and want to know which IP addresses occur in all of them.

2) You have two SQL dump files, and want to know which tables are in the first but not in the second.

3) You have several files containing a list of usernames to create. You want to create all the users in order, but don't want to create twice.

Web sites with source code for hacking

Once in a while people ask me what open source projects could the contribute to? I can point them to any module on CPAN, but it can be a lot more fun to contribute to a project where the contribution has immediate and visible impact.

I don't mean projects like Bugzilla which is downloaded and run by the end user, but projects like MetaCPAN, or the web site of Rex.

I've started to create a list of web sites where the source code is available, and of course that is is written in in Perl. I tried to group them according to the level of activity of the project and where I could, I also mentioned the framework used for the project.

I got a lot of input from people on Reddit, but I am sure there are plenty of other sites. Please suggest more sites, or even send me pull requests.

Using Minion with a REST API

Minion is a job queue for the Mojolicious real-time web framework. Below is example usage that sends an email using a REST API. The email address is handed off to minion and then minion processes the task. Everything is in a self-contained file as a Mojolicious::Lite app.

In addition, a config file is shared between minion and the web app.

One cool thing about Minion is that it has support for a Postgres backend using Mojo::Pg. Given that, workers can be running jobs on different physical boxes as long as they use a compatible "Pg" connect string.

The web app is started like so:

$ /opt/perl minion_email.pl daemon

A Minion worker is started with:

$ /opt/perl minion_email.pl minion worker

Example usage:

Parsing: top-down versus bottom-up

[ This is cross-posted by invitation, from its home on the Ocean of Awareness blog. ]

Comparisons between top-down and bottom-up parsing are often either too high-level or too low-level. Overly high-level treatments reduce the two approaches to buzzwords, and the comparison to a recitation of received wisdom. Overly low-level treatments get immersed in the minutiae of implementation, and the resulting comparison is as revealing as placing two abstractly related code listings side by side. In this post I hope to find the middle level; to shed light on why advocates of bottom-up and top-down parsing approaches take the positions they do; and to speculate about the way forward.

Top-down parsing

The basic idea of top-down parsing is as brutally simple as anything in programming: Starting at the top, we add pieces. We do this by looking at the next token and deciding then and there where it fits into the parse tree. Once we've looked at every token, we have our parse tree.

Day 8: Stopwatch (App::stopw)

About the series: perlancar's 2014 Advent Calendar: Introduction to a selection of 24 modules which I published in 2014. Table of contents.

Okay, this post couldn't be simpler. stopw (from the App-stopw distribution) is a very simple CLI program. When run, it displays a running stopwatch, upon which you can press Spacebar to let it move down one line (so you can see/note the time you pressed the Space), or Enter to pause/resume, Z to reset back to 0, or Q to quit. I'm almost shameful that it took my Perl code more than 100 lines to accomplish this (I think the teenager me from the 1980's would surely have been able to whip up this task in about 20 lines of BASICA/GW-BASIC).

I now try hard to remember why I needed this in the first place, but can't recall. Probably to time some subtitle, or a song that I listen on the radio, or to exercise holding my breath, or exercise my reflex. But probably not.

Anyways, there is an existing CLI stopwatch on CPAN before I wrote this: App::Stopwatch (which includes the stopwatch utility), written by BESSARABV. However, it doesn't show subsecond precision, and does not allow for many controls like pausing/reseting. Plus, the name clashes with the Debian package of the same name.

Git-Like Menus


[Pleased as I was to get mentioned in a lightning talk in this year’s YAPC, I noted that my mention was in the context of writing blog posts that “don’t contain much code.”1  Well, fair enough: I’m a verbose bugger, and a wannabe writer, so my prose does tend to ramble.  But I can do code, dammit.  So, you know ... here’s some code.]

The other day I was working on my music library scripts,2 and I needed a menu for something.  Now, there are oodles and oodles of modules on CPAN to help you write menus.  I’ve looked at most of them, and tried quite a few, but long ago I settled on using the -menu option in IO::Prompter, by the Damian.  For a nice, pretty menu layout—say, something you do as a central feature for a program—it’s tough to beat.  It’s not perfect, by any stretch, but it offers some very nice features, such as (optionally) not requiring ENTER after a menu choice.

But that’s not what I wanted in this case.  What I was looking for here was a quick, compact menu ... sort of like what you get when you’re interactively staging a commit in git (that is, git add -p, or, probably more commonly, git add -i then choose “patch”).  Specifically, the features I wanted were:

Buftabline – forget Vim tabs, now you can have buffer tabs

I just released Buftabline, a Vim plugin that takes over the tabline and renders the buffer list in it instead of a tab list. It is designed with the ideal that it should Just Work, and has no configurable behaviour: drop it into your configuration and you’re done.

There are comparisons with several alternative plugins in the README, as well as an explanation of why you probably want this rather than using Vim’s built-in tabs.

Share and enjoy.

We are all Perl's ambassadors...

I wonder how many newbies in the whole history of Perl had their "llama" arrive from Amazon and shortly thereafter—in a fever pitch of excitement—managed to create a distribution they thought was the best thing since sliced bread and upload it to CPAN only to have their debut into our wonderful, loving community met by a CPAN rater. After which, naturally, they slowly backed out of the room, closed the door behind them, picked up their (PHP|Ruby|Python|Javascript) book and never coded a line of perl again...

Surely, no one can say but maybe—just maybe—we'd be better off with a system similar to Stack Overflow where a user has to have a certain number of points or special kind of badge (e.g. the Teacher, Ambassador, the "Empathy Lieutenant", the "brian d foy", etc) in order to leave a rating on someone's first distribution.

I dunno. It's worth a thought, because this whole thing just made me cringe...

Day 7: Set screensaver timeout from the command-line (App::SetScreensaverTimeout)

About the series: perlancar's 2014 Advent Calendar: Introduction to a selection of 24 modules which I published in 2014. Table of contents.

Here's the problem statement: For safety/privacy reason, I generally want my desktop screensaver to trigger rather quickly (say 3-5 minutes) to minimize prying eyes when I'm away or fall asleep and forget to lock. But when I watch videos, I want to set the timeout to a longer time (say the total duration of the videos plus 1 minute, or (better) 20-60 minutes which is the amount of time from I start the video until I doze off at night.

I used to have this line on my ~/.mplayer/config on my PC:

heartbeat-cmd="xscreensaver-command -deactivate &"

until I use a laptop with KDE plasma on it, which doesn't use xscreensaver by default. Plus I don't think disabling screensaver for the whole playback duration is sometimes not what I want (see above).

Tour of Imager

I’ve posted my slides for the Tour of Imager talk I gave earlier this week at MadMongers. If you are interested in processing photos or drawing in Perl, you’ll want to check this out.

[From my blog.]

mop minus proposal

I'm creating mop-minus-proposal project on GitHub. Current experimental mop implementation need big core change. so I think writable object orientation support is needed without core change or with minimal core change.


mop-minus-proposal

The following is example. This is not only specification. This code work on Perl 5.20.1+. Important thing is that the following code don't use Perl source filter, such as Filter::Simple.

How to write a Developer CV/Résumé that will get you hired

Slides from my LPW talk are now online!

How to write a Developer CV/Résumé that will get you hired.

Day 6: Think globally, act localizably (File::umask, Locale::Tie, Unix::setuid)

About the series: perlancar's 2014 Advent Calendar: Introduction to a selection of 24 modules which I published in 2014. Table of contents.

Local variables or dynamic scoping is a very nifty feature of Perl that nowadays do not exist in a lot of other languages. It allows you to temporarily set the value of a global variable during a block/scope during runtime, effectively localizing global variables. For example, even though some modules provide configuration settings as global variables, like Data::Dump's $INDENT, you can change the values of these variables without having a global effect by using local:

{
    local $Data::Dump::INDENT = ""; # disable indenting
    $res = Data::Dump::dump(...);
};
# here the value of $Data::Dump::INDENT is restored the original value

Spotify and Perl Article

Sawyer X recently gave a talk in which he outlined that merely talking about the stuff we are doing helps promote perl (watch his talk here on youtube). I believe he is absolutely correct.

Mike Schilli is one person who I think is doing a fantastic job of this. He writes a Perl article each month in Linux Magazine which is available in print and electronic editions. Here in Australia, it's available in most newsagencies (which is where I tend to stand around reading it... and sometimes I buy it). I think thats some pretty solid reach.

His article this month uses Perl to talk to the Spotify API. Which to me, is a cool way to introduce people to Perl via something that is well known and external to Perl programming problems.

There is a pay wall and by way of disclaimer, I have no affiliation with the author, the magazine or Spotify (although I do have a Spotify premium account).

Inline Grant Progess

Greetings from PPW 2014.

David and I had a good week on Inline, but I had to cut it short to prepare for and attend this year's Pittsburgh Perl Workshop. I gave a mixed bag talk yesterday morning and the first topic was the Inline grant: https://www.youtube.com/watch?v=vDRLIjojlhg

You can read about our progress in the Week #3 report here: http://inline.ouistreet.com/

Don't forget to read the week #2 report as well, if you missed that last week.

The Silver Camel goes to ... Mark Keating

At the end of the London Perl Workshop this year, we presented Mark Keating with a Silver Camel, to acknowledge everything he has done, and continues to do, for the Perl community, and particularly the UK Perl community.

Here's Mark shortly after being presented with his Silver Camel:

Photo by Wendy G.A. van Dijk

In case you're not familiar with Mark:

  • He has been chief organiser of the London Perl Workshop since 2008
  • He is co-founder and co-leader of North-West England Perl Mongers
  • He's been involved in the Google Summer of Code
  • He's director and secretary of the enlightened perl organisation
  • He's chair of The Perl Foundation's marketing committee
  • He's been a key player in the scheme to send newbies to conferences
  • He's talked about Perl at non-Perl conferences
  • His and Matt's company (Shadowcat) are long-term supporters of Perl

Day 5: Look ma, no 'argument list too long'! (App::rmhere)

About the series: perlancar's 2014 Advent Calendar: Introduction to a selection of 24 modules which I published in 2014. Table of contents.

If you spend enough time on a Unix/Linux shell, sooner or later you'll be faced with the task of removing files in a directory that contains a lot of those files (in my case, it's usually a Maildir with lots of spam/unread emails). The directory will contain so many files (like 50k or even millions) that doing rm * will fail with the annoying "Argument list too long" message because the * wildcard is expanded by the shell into a multimegabyte list that doesn't fit into the readline buffer. To delete the contents of this directory you will have to resort to some tricks, like using xargs -n, or going up one level, deleting the container directory, and later recreating it.

GitPrep 1.9 release, Atom feed of commit log support, short ssl url support

I released GitPrep 1.9. You can install portable GitHub system into Unix / Linux easily. It is second major release.

Because you can install GitPrep into your own server, you can create users and repositories without limit. You can use GitPrep freely because GitPrep is free software. You can also install GitPrep into shared rental server.

GitPrep (Document and Repository)

Features added in 1.9 are:

  • add ssh_rep_url_base to cnahge or hide ssh URL user directory
  • add atom feed of commits page. for example, http://somehost.com/kimoto/gitprep_t/commits/master.atom

In version 1.9, by using ssh_rep_url_base option, you can hide user home directory in ssh url when you use public authentication. For example, You can do the following.

ssh://gitprep@59.106.185.196:55555/kimoto/gitprep_t.git

And, Gitprep support atom feed of commit log, so you can check other people commit by rss reader.

Let's use usufully.

Example

You can try GitPrep example.

GitPrep Example

Download

Download

Document

GitPrep Document and Repositry

Quick post-LPW roundup

I've just arrived back from the London Perl Workshop. Lots of very interesting stuff. A big thank you to the organizers! I especially liked:

There were some good lightning talks too. I liked:

For those of you who wanted a copy of my slides, they are on github.

Scratching an itch - interpolable HTTP Status constants

When working on larger web applications, I prefer to use HTTP::Status to provide human-readable constant names in the code. This is especially helpful for anything other than the common 200, 404 or 500 status codes.

But the constants exported by HTTP::Status are basically subs:

if ($response->code == HTTP_OK) { ... }

this is fine for most cases, but not when you want interpolable variables, for example, in hash keys.

So I wrote HTTP::Status::Constants. It's a simple wrapper around HTTP::Status that provides read-only scalar constants for the HTTP_* constants.

About blogs.perl.org

blogs.perl.org is a common blogging platform for the Perl community. Written in Perl with a graphic design donated by Six Apart, Ltd.