Simple Linux Screenshot Application in Perl

Shutter, a powerful application for taking screenshots is possibly soon to be removed from Debian, taking with it the last desktop Perl application in that ubiquitous distro. That is a shame, but there is hope, I guess, some brave soul with take up the effort required to make it fit with GTK3s new APIs. That person is not going to be me. I don't have the skills or the time to do this. But hey, I am the author of the world's simplest GUI generator( probably). Surely it can't be that difficult to cobble together a few applications to make the world's simplest Screenshot tool? Does not need to be fancy, but needs to have a ton more features than the basic Screenshot application in Ubuntu, and be generally useful for day to day use even as a proof of concept.

Requirements

  1. Must be able to do everything that the built-in screenshot application does: -capture screenshots, save the images, and copy to clipboard.
  2. Add-on an external editor to annotate the captured images
  3. Delayed image capture
  4. Extract multiple application window captures, not just the active window
  5. (Animated screen captures e.g. to generate tutorials)
  6. (Add web camera support)

The first 4 I have been able to do relatively easily, using Imager::Screenshot, File::Copy, and a few external X11 tools: - xprop (included by default in Ubuntu) and xclip (needed sudo apt-get install xclip to install, adds clipboard functionality).

Collecting Window IDs and Names of the frame

A couple of ways to get window Ids on X11, of which xprop comes already installed in Ubuntu.

my $windowList= `xprop -root|grep ^_NET_CLIENT_LIST`;
my @winIds=$windowList=~m/(0x[0-9a-f]{7})/g;

xprop also gives us a way of getting the names of the windows

my $name=`xprop -id $id|grep '^WM_NAME(STRING)'`;
       $name=~s/WM_NAME\(STRING\) =//;
       $name=~s/\"//g;
       chomp $name;

Capturing the screenshot

Imager::Screenshot takes, would you believe, screenshots. Passed with a window ID it will capture the contents of the frame. If we know the window Ids of all the windows we could create a hash containing all the screens. While it is possible to include window decorations with the Winodws variant of Imager, this appears not to be possible for Linux. Nevermind...the code is simple.: -

my %images=map($_ => screenshot(hex $_) @winIds ;

In practice this takes a while if there are are many windows. Better to to these as and when needed.

Saving to clipboard

Now I know there is a module called Clipboard. I presume this uses xclip, itself, and as I know there is only one line that needs it, it is easier to install just xclip and : -

  system("xclip -selection clipboard -t image/png -i $workingDir.$workingFile ");

Wrapping it up with a GUI

It is trivial to set up this with GUIDeFATE, to provide all the functionality and much more in little more than 100 lines of code. Using GUIDeFATE means that we can use this with Qt, Gtk, Wx, or Tk. (Win32 not covered with this version, but probably not challenging to add the functionality). Connecting to GIMP is also trivial one line connected to an Edit button. A delayed capture is again trivial. As we know the names of the window, this can be displayed with the thumbnail. Now I don't imagine that this could ever replace a professional application, but as a proof of concept, this shows what can be achieved with relatively little effort. But with only a little more effort more can be achieved...e.g. the ability to generate animated tutorials. I am just thinking of how this might look in action...

Screenshot

This is the result of a full screen grab containing another running screen grabber (using Tk back end) grabbing the contents of the frame of another application... screenshot of screenshot in action.png

Ducktaping The Internet To The Desktop

Simple desktop applications are generally not what one considers a Perl specialization. Its expertise lies in generating processing and transforming textual data, hence its use in the web, and in tools like GUIDeFATE. This 'duct tape' manages to parse text, extracting relevant data, and absorbing information and producing a meaningful output efficiently. Displaying this output in a desktop application shouldn't be too difficult.

Challenge accepted

Not long ago I was inspired (by Ron Savage) to demonstrate ways of extracting data stored in databases with web front ends, merging contents of different sources of data and presenting it to the user, optionally making this data available offline in a local database. Should be possible to do this for the desktop user via graphical interface using GUIDeFATE. Specifically I wanted extract taxonomic information about plants, a field I have zero knowledge about. The Plants List is a useful resource with all plant species listed to anyone with a browser. Getting the list of all accepted and unresolved families and genera is trivial...use LWP::Simple and parsing the webpage form get ('http://www.theplantlist.org/1.1/browse/-/-/)' ) . From this, extracting species is a minor variation on the theme.

Wikipedia is another useful textual resource with images. From the genus or species being viewed, identifying any images on wikipedia is a mere case of generating a prospective URL, LWP::Simple::getting the source again, and using regular expressions to find the relevant section. I suspect that someone has a module that allows to navigate the HTML more elegantly rather than I have done, but as a proof of concept, this crude measure yields reasonable results.

All these can allow a simple desktop application to be developed using the rough and ready GUIDeFATE. The application allows you to search the entire Plants List directory, keeping a local cache of discovered data, and similarly searching for images from wikipedia (and keeping a copy of the images as well), to reduce the load on the respective servers).

Creating a GUI Interface

Straightforward using GUIDeFATE, of course making some allowances for the limited widget options.

use strict;
use warnings;
use GUIDeFATE;
# <  Initialise other modules and variables here >
my $window=<<END;
+-------------------------------------------+
|T The Plants List App                      |
+M------------------------------------------+
|  {Refresh Data} dd/mm/yy                  |
|  [                         ] {Search}     |
|  {Angiosperms  } 000   {Gymnosperms} 000  |
|  {Pteridophytes} 000   {Bryophytes } 000  |
|      No item on list                      |
|  +T-----------------+  +I--------------+  |    
|  |                  |  |               |  |
|  |                  |  |               |  |
|  |                  |  |               |  |
|  |                  |  |               |  |
|  |                  |  |               |  |
|  +------------------+  +---------------+  |
|  {<} {  Explore } {>}  { Upload Photo  }  |
|      0000 of 0000                         |
|  www.theplantlist.org     wikipedia.org   |
+-------------------------------------------+
END

my $backend=$ARGV[0]?$ARGV[0]:"web"; # allow command line backend selection
my $assist=$ARGV[1]?$ARGV[1]:"q";     # allow command line option selection
my $gui=GUIDeFATE->new($window,$backend,$assist);
my $frame=$gui->getFrame()||$gui;

# <  other code to be run before GUI activated  >

$gui->MainLoop()

# < rest of code >

Then adding LWP, websearch and data extraction, gives you an app like this.Qt_plantslist.png

Ducktaping the Web to a Websocket interface.

Well the real motivation for the initial challenge was to stop reinventing wheels that already work well, using clever tools like Mojolicious and other fantastic web frameworks. Can this work as a WebApp through GUIDeFATE? Apparently so, and in pure Perl. A few bugs apart, the webapp counterpart of the desktop application seems to work pretty much the same as the Desktop application. From GUIDeFATE version 0.11, setting the domain, listening port are both now possible, though still defaulting to localhost:8085. A new server is created which identifies an available listening port and a new client that survives losing focus. This makes it possible to have multiple different webapps being served and tested at the same time. WebSocket plantslist.png

So there you have it. I am afraid there are still a few bugs to squash here, and most significantly I have failed in my attempt to rationalise the position of GUIDeFATE and its slave modules in a suitable namespace. But I will next aim to add a couple of more widgets..., web links, on/off buttons, check boxes and sliders are on my list, and I suspect these will be required for the last GUI back-end to be added to the already existing Wx, Tk, Gtk, Qt, Win32 and HTML/ WebSocket backends.

Websocket Extension for GUIDeFATE - Dialogs and File Operations

GUIDeFATE, your favourite Quick-and-Dirty GUI designer for newbies is now acquiring a Web-socket interface. Now this is certainly not capable of competing with those genius applications Mojolicious, Catalyst, Dancer etc. Having only discovered Web-sockets a few weeks ago my yield is going to be decidedly sketchy. Of course a desktop interface is quite different to a web app...the machine running the interface is the same as the one the user is sitting at, and the program running the graphical output is the same as the one that is handling user interactions. When there is a client and server involved, a certain of communication is required between the two, both must of course be able to understand each other, even though they may be coded in different languages. GFweb (GUIDeFATE's Web-socket module) handles 1) the generation of the user interface 2) the initiation of a listening socket, 3) handles the communications between the two.

Dialog/Message boxes over websockets

So in the previous iteration of the GFweb we managed simple comms, button presses and interface updates. Things like the calculator and Rock-Paper-Scissors-Lizard-Spock work well. Buttons are connected to functions on the server, and each function completes an update to the state of the client immediately. Dialog boxes and File operations have been more tricky for me to implement. In the desktop modules it has been possible to call a function that triggers a dialog box and that function that triggers the dialog box returns the result...

 # function showDialog returns the result of the dialog box,
 # so can be inserted directly into, say, an if statement e.g. 

 if($frame->showDialog("Sure?",
    "This will wipe existing text...proceed?",
    "OKC","!")){ etc.

Now there probably is an easy way to do this in frameworks like Mojolicious etc. But as I have taken up wheel re-inventing, and already have started learning about Net::Websocket::Server, I have to figure out how to make a comparable interaction. This minimalist server can not trigger the dialog box and then proceed according to the result of that input. At least I have not figured out how to do this. So while under normal operations one triggers the client to create the dialog box first waits for user response and reacts to it. in the web app form, the routine that triggers the the dialog box and the one that reacts to it are distinct. The solution? Before triggering the dialog box, the programmer defines potential responses, by writing appropriate routines, and inserting these into a dispatch hash table. It is only after this point that the dialog box is triggered. The server later reacts to the actions of the user when announced by the client.

# the following lines add actions to each potential response to
# the Dialog.  Allowed responses Ok, Cancel, Yes, No and is
# defined by the parameter in showDialog e.g. OKC= Ok and Cancel,
# YNC is Yes No and Cancel
$frame->dialogAction(  "OK",       # action if ok clicked
     sub{$frame->appendValue("TextCtrl1","\nClicked OK\n"    )   } );
$frame->dialogAction(  "Cancel",   # action if cancel clicked
     sub{$frame->appendValue("TextCtrl1","\nClicked Cancel\n")   } );
# now trigger the dialog box
$frame->showDialog("The Dialog Title",
                   "The dialog Message goes here",
                   "OKC","!");

File interactions

Handling files is another issue. So a file selector box is relatively easy to trigger, the "onchange" parameter can then be used trigger the server to be ready to receive a file, and then the client proceeds to send the file as binary data. Simple.

Websockets can do binary transfers, but while the websocket protocol seems to allow very large files, with Net::Websocket::Server I could only transfer files less than 64k for some reason or the other. There is probably a way to increase the maximum transfer size. But the solution I did eventually was to create a BinaryBuffer object in JavaScript that takes the file, splits it into 64,000 byte chunks and handles the handshaking with the server until the file is reassembled on the server. Here is the screen capture log of the transfer process: -

Websocket File upload.png

Immediately after the file is loaded, the dispatch table is again queried to allow any routines that need to follow after file has been uploaded. The code is as follows, as before predefining actions for each of the potential reponses before triggering the file selector: -

# the following allows the user to upload a file to the server
# the server stores the file in a folder called dataFiles in the
# directory of the running application
$frame->dialogAction(  "Cancel",  # action if cancel clicked
     sub{$frame->appendValue("TextCtrl1","\nClicked Cancel\n")   } );
$frame->dialogAction(  "File",    # action to run after file loaded.
     sub{  
         # In this example add the filename to the text box
         # and set the Imagepanel to have this picture
        my $file=shift; #the name of the file is passed as parameter
        $frame->appendValue("TextCtrl1","\n Loaded File: $file \n"),
        $frame->setImage("Image2","dataFiles/$file")
     }  );
 # now trigger the file selector dialog box
$frame->showFileSelectorDialog("Pick a picture File",1,"");

File downloading is decidedly easier...an <a href="file-to-download" download>DOWNLOAD </a> sorts it out.

Still a bit of work to go before I put it on CPAN...not least of which is that I wish to create a Gardening Database viewer. The code is available in the test folder. I want to allow the developer able to define the listening port/host (currently localhost and 8085...useless apart for testing purposes) and also fix my implementation (currently generates an HTML file followed by a call to the system browser to load that HTML while the server is running in the background) so that it detects whether on localhost of called from across the network. I also need to start rethinking the namespaces of GUIDeFATE's accessory modules as pointed out by grinnz on reddit...maybe too late. I have tried shifting things in my setup, can't seem to get things working when I do.

Now I know what you are thinking..."Why on earth bother?". Sure this is crude, probably not very secure, and pretty deficient compared to most widely used frameworks. The answer is simple. GUIDeFATE is simple. It is sort of WYSIWIG, and will allow someone with little or no knowledge of graphical back-ends or HTML or Websockets to create a quick-and-dirty GUI application in pure Perl.

A new wobbly wheel for GUIDeFATE

A couple of weeks ago a little misunderstanding about what GUIDeFATE was about led to an interesting comment. This tool had been designed to simplify GUI design using a text editor, and met the challenges of having a number of weakly supported potential targets to produce a semi consistent interface. Essentially producing a GUI for Perl applications has never been easy. A number of back-ends exist ( e.g Wx, Tk, Gtk, Win32, Qt etc) all with different approaches, most with rather outdated documentation, often not universally easy to install. A newbie like myself was never going to find this straight forward. So a tool was born that abstracts out the complexities of GUI design, producing near identical interfaces in multiple back-ends, all from a textual representation. As each interface was accommodated and announced, came reports of other folk being unable to use said interface, followed by a new module to use another GUI toolkit; so now we have a tool that can now accommodate 5 desktop UI systems.Surely that must cover all bases? Apparently not.

In response to a blog out-lining the difficulty I had had with the Qt paradigm of SLOTS and SOCKETS for interaction between the UI and application logic, and the need for a custom dispatcher, was a comment relating to WebSockets. Now GUIDeFATE was intended to be desktop UI focused, so clearly WebSockets are not the Sockets at issue. But Ron suggested I try out other people's efforts before re-inventing the wheel, thus finding this particular wheel rolling in a different direction. I started finding out about Mojolicious, Dancer, Catalyst. These are clearly the efforts of geniuses, with multiple dependencies and difficult to crowbar into an existing toolkit without duplication and “re-inventing the wheel”. I requested advice.

Flexibility is proportional to complexity

To connect to my less efficient brain requires much greater simplicity. So I quickly abandoned all these powerful frameworks. I chose the simplest possible websocket client, with the simplest available server, unashamedly ripping off the code from the examples provided. Attached them to quite possibly the simplest UI generator in the world... GUIDeFATE, of course☺. The result is GFweb.pm. A module for GUIDeFATE that generates and starts both the server and the client for a web-app-like interaction with a UI nearly identical to the desktop applications and with nearly completely unchanged programming logic. The extra effort required to transform a desktop app to a web application? Just change the small string selecting which back-end you want and the address/port to use.

my $gui=GUIDeFATE->new($window,"web","q");

The wheels come off..

Websocket rpsls.png

Admittedly the tool keeps things simple: Only a handful of widgets, with limited interaction potential: hopefully just enough for a quick and dirty graphical application, simple enough for a noob like me with little knowledge of JavaScript or WebSockets. The reinvented wheel is decidedly wobbly. File selection and message boxes are not yet done...my problem is how to handle the client side responses which the server-side programming logic is waiting for to further process. I will figure it out, or somebody will point me in the right direction...it's inevitable. This experimental version is uploaded into GitHub. The applications need to be started in a folder containing the dependent files (preferably from a terminal) and popups need to be enabled (the application currently also starts a logging window to enable debugging).

Four backends now for GUIDeFATE...but the struggle continues

Just as you are thinking you got the hang of something, believing you know how to work the system, imagining you have a problem licked, then you wake up to find out it was just a dream. That has been my experience in developing GUIDeFATE (A contrived acronym for the project - Graphical User Interface Design From A Text Editor).

GUIDeFATE is a tool that simplifies GUI design by choosing only a few useful elements that make a usable graphical application and making those elements accessible to the beginner developer. Handy too if you want a quick and dirty desktop application in Perl. A tool that converts a string into a GUI, using one of the many backends that exist. So I have been making this over the past month or so, using the often woefully outdated resources found on the internet. I had a rudimentary Wx Backend after about a week, by another week I had Tk, and was persuaded to put the project up on Github. Soon Gtk was also covered easy as pie in 3 days. Even started working on Win32...(I only have Linux machines at home, fortunately also have more skilled help in this matter in the form of mpersico)...then CRUNCH...

A fellow redditor daxim suggested Qt. Great idea, I thought. Qt well supported on Linux and other platforms. And Gtk was a breeze to handle. Significantly Qt and Perl have had a close relationship. This would be a walk in the park I figured. I figured wrong. Just to get Qt development setup on my system was a struggle. Installs from CPAN failed, installs from repositories are no longer updated. ppas claiming to offer this also fail. Switched to Kubuntu, wrecking some of my settings, no joy. Near defeat, daxim once again gently guided me to a reasonable install of Qt4. Problems over? No. Just beginning.

Qt

Qt is sophisticated, powerful, all-encompassing. Getting control of this sophistication and power needs huge brains which I lack and a dextrous use of Qt Designer, which defeats the object of GUIDeFATE. Secondly, Qt internal design appears to have evolved in a parallel universe, isolated from other paradigms of GUI design, even though on the surface looking similar. Sort of like Convergent evolution that gave flies and birds wings and legs. To crowbar this interface into GUIDeFATE was not easy (for me).

Key to this is how GUIDeFATE allocates actions (eg on button click, menu select etc) to widgets...it merely takes a reference to subroutine in main:: that has not been yet created. An example from a button widget generator from the front end sends a reference to a list of parameters for each widget like this: -

addWidget(["btn",$bid, $label,[$locationX,$locationY,[$width,$height], \&{"main::btn".$bid}]);

GUIDeFATE's middleware for each backend stores this referenced list in a list of widgets before it sets up content in new(). This means that the developer can create his own functions easily as long as he knows the id of the widgets, the autogenerator can generate the skeleton for these functions and Bob's your uncle. All the other backends quite easily accepted a dereferenced function as parameter in their widget.

Not so Qt. Qt uses SLOTs and SIGNALss. A powerful system, SLOTS are attached to the Qt::MainWindow, which is a Parent of the Qt::Widget container (in my case) containing all the widgets. So the SLOTS are in a different place to the generator of the SIGNAL. There is no way to add slots dynamically (as far as I can see), and you have to declare them at the beginning...difficult if you dont know what the functions are going to be. Then you have select a SIGNAL to be associated with a widget and connect the two. A simple Perl example is here.

Fortunately there is SignalMapper. This allows SIGNALs to be mapped to a single SLOT in signal mapper. Then comes the complexity of extracting signals from different events by a dispatcher which distributes that signal to the correct function.

Setting up the SignalMapper in new(): -

$self->{SigMan} = Qt::SignalMapper($self);
$self->connect($self->{SigMan}, SIGNAL 'mapped(QString)', $self, SLOT 'mapAction(QString)');

Adding a Signal to the widget, and giving the signal mapper clues about the source for the dispatcher to work on:-

$self->connect($canvas->{"btn".$id}, SIGNAL 'clicked()', $self->{SigMan}, SLOT 'map()');
$self->{SigMan}->setMapping($canvas->{"btn".$id}, "btn".$id);

And creating a dispatcher: -

sub mapAction{
   my $widgetIndex=getItem(shift);                   #find the widget parameters
   my @widget=@{$widgets[$widgetIndex]};             #using the clue (the Id)
                                                     #provided by SignalMapper
   my $wType=shift @widget;                          #remove the type,
                                                     #which allows you locate
   if ($widgetIndex !=-1){                           
      if     ($wType eq "mb")   { &{$widget[3]};}    # the function referenced
      elsif  ($wType eq "btn")  { &{$widget[4]};}    # and call that function
      elsif  ($wType eq "combo")  { &{$widget[4]};}
   }

}

So learnt a new paradigm, and finally GUIDeFATE 0.08 has another backend.

Four backends supported.png

One thing I am struggling with is understanding packages and the differences between "use ", "use parent " and "use base ", and other ways of absorbing other modules into the code. It takes a lot of guesswork and fiddling and trying different combinations for me to get these backends to work. It may be the way their Perl interfaces and the C libraries work. More likely it is something to do with my lack of understanding, but it sure leads to messy code, and scrambled brains.