The times you want to avoid the short cuts. PWC-39

No spoilers here, just a rambling. There are probably countless times when one relies on the genius of others to complete tasks. If PerlWeeklyChallenges has shown me one thing, it is that there is probably nothing I can do that someone else can not do better, more efficiently, and more robustly. The key to good coding, one imagines, is to have code that survives not just the scenario presented in the challenge, but also accommodates other scenarios

The Hotel lights

Take the lights on puzzle. A first glance at the puzzle show that the times are all overlapping, so technically the lights should be on continuously from the first guest entry to last exit. So the simplest way to extract the time the lights were on is to get the earliest time and the latest time mentioned in the log. These can be extracted using four steps: load the log, extract the earliest and the last time; convert to minutes and subtract.

#!/usr/env perl
# Perl Challenge 39 Task 1
my $log =  # from the puzzle 
"1) Alex    IN: 09:10 OUT: 09:45
2) Arnold  IN: 09:15 OUT: 09:33
3) Bob     IN: 09:22 OUT: 09:55
4) Charlie IN: 09:25 OUT: 10:05
5) Steve   IN: 09:33 OUT: 10:01
6) Roger   IN: 09:44 OUT: 10:12
7) David   IN: 09:57 OUT: 10:23
8) Neil    IN: 10:01 OUT: 10:19
9) Chris   IN: 10:10 OUT: 11:00";

# extract earliest and latest times
my ($earliest,$latest)=(sort ($log=~/\d\d\:\d\d/gm))[0,-1];

#convertion function to produce minutes
sub time2minutes{
    my ($hours,$minutes)=split /:/,shift;
    return $hours*60+$minutes;
#print results 
print "Earliest time is $earliest,
Latest time is $latest.
Lights on duration is ", time2minutes($latest)- time2minutes($earliest);

Cheating is no doubt fun, gets the job done, but what if the data was non-contiguous, instead of what was presented in the challenge? We'd need to work out overlaps. The data was presented sequentially, conveniently for identifying overlaps, but what if it wasn't? Would it be better to to allow in your code to accommodate these possibilities and exceed the demands of the puzzle?

Extend the challenges

Tasks that may be completed using simple algorithms are great. Having entered the last couple of times it is clear that much can be done capitalising on previous skilful endeavours presented in CPAN. Doing a MetaCPAN search is trivial and solves many problems quickly. The RPN Challenge can be easily completed by using Math::RPN or Parse::RPN but doing so to solve a problem also fails in one component that appeals to me. Challenges exist to educate and push personal boundaries. Observing other folk's entries also entertain and inspire, so thanks you clever bods who also do the challenges. The greater challenge would be to include as many operations as possible with as little effort as possible.

RPN actually simplifies mathematics; for example, one could simply eval "$stack[0] $operation $stack[1]" and thus produce a generic translation for several built in binary operations, simply removing parameters from the stack and inserting results as needed. This way one could handle strings as well as numbers. For a comprehensive RPN calculator one would also need to include operations that manipulate the stack. And then there are other operations not included in the base Perl. Of course one could import Math::Trig, but seeing as we are building an RPN interpreter, why not code for functions like asin, tan etc in RPN? asin($var) can be calculated using atan2($var, sqrt(1 - $var * $var)). Covert that to RPN dup dup * 1 - sqrt swap atan2. Why would you do that? Of course it is slower, but the stack manipulation happens automatically, the whole thing can be done in one phrase, proves the capability of the interpreter, and gives you a warm fuzzy feeling inside.

Therein lies the problem. While challenges posed by someone maybe finite; the challenges you pose on yourself may have no end. Sleep beckons.

Circular observations

Selfies demonstrate one of the principal laws of physics, psychosocial development of teenagers and programming. The act of observation alters that which is being observed. It becomes more complicated if the observer is observing itself. As the observer changes, so might future observations particularly if the tool being used is known to impact the parameter that is being measured; rapidly one loses hope that any meaningful data will be obtained from such a direction of development. Sounds gibberish? Look at the collection of "duckfaces" on your teenage daughter's phone (if she ever lets you near it) and I am sure you will find no resemblance of those images to anyone you would recognise as your offspring.

Let's do it any way

It is so with programming. But let's not let that stop us from still trying eh? The problem I alluded to in my last blog post regarding memory leaks in some GUI back ends offers such an opportunity. I had observed these leaks when trying to use timers to do animations. The way these animations were created was by creating a simple svg file, and altering that file in Perl, saving it, and inserting that file into an image panel, and repeating. A slowdown was noted with some back-ends , and using top, one can see the memory being consumed by that Perl interpreter process progressively rising.

I want to automate this measurement, and visualise the output graphically. But I don't know the pid of the process until I start the application. But my application knows its own pid. And my application itself can generate graphics. It can measure how much memory it uses, storing the data captured in an expanding data structure (an array) using changing amounts of memory, and using a subroutine that takes time and memory itself. It can then generate an SVG chart which gets more data points added each cycle, saving it and reloading it using processes that itself are suspect in memory leaks, and finally rendering this graph on the leaky UI. A fun-filled fool's errand that may deliberately exaggerate leaks thus demonstrating and identifying problematic systems while not, at first glance, offering fine grained analysis.

The pmap <options> <pid> system command (Linux).

$$ in Perl gives us the Process ID. Passing this as a parameter to pmap delivers a running application's memory footprint. It not only describes the total memory it uses but also subdivides the amount by internal structures which may give more clues as to what may be hungry for RAM. We can (for now) discard all the extraneous data and focus on the last line of the output (containing the total memory used) and progressively generate a chart of the excess memory being used with each cycle within the application and deliver it to the image panel widget in the UI.

 sub memRequired{
     my $mr=`pmap $$`;             # call pmap
     $mr=(split('\n',$mr))[-1];    # get last line
     $mr=~s/[^\d]//g;              # just keep numeric portion
     return 1*$mr;

Keeping an eye on the ball

qt Chart.pngThe animation script from before is now modified to allow selection of which animation is being done. The animation may be stopped without stopping the timer, and It is also possible to overlay the chart on top of the other SVG animations already playing. So I have done a couple of simple tests. It is clear it is not the timer objects themselves that are directly involved in the memory leak. It is the Image panel update process. Tk and Wx need ImageMagick...but Tk leaks more than Wx. In these tests Qt appears the leakiest (Figure 1) although not obviously using ImageMagick.

GT3 Chart.pngGTK3 and GTK2 both lose no extra memory with SVG animations generally. The slideshow (jpg images once updated every second) appears to show transient extra memory used, going up in step wise fashion, as if dependent on the size of the image displayed (Figure 2).

Tk Chart.pngFor the backends that do use memory while playing SVG images there appears to be a consistent amounts of memory lost each time. The images have varying complexities, but this seems not to matter...all the svg images are 200px vs 200px. Most animations in these tests update 10 times per second, but clock and the slide show updates the frame once a second. Switching between the 1/10 sec and 1 sec produces a predictable patterns of memory leak.

I would like to see if there are other ways to put an image into a frame other than using Image::Magick. ImageMAgick does support a huge number of formats though. I would prefer to be able to migrate to Imager. This seems more tightly integratable into Perl.

Re-Entry of GUIDeFATE into CPAN

Any way the timer handling is now more or less uniformly supported across Gtk2, Gtk3, Qt, Wx and Tk. I would not use it for animations in anything other than using the Gtk2/3 options. GUIDeFATE for almost the last year has been not synced with CPAN. Version 0.13 will see it back on GitHub but with some changes. I will be removing the sub-modules from the base path into a folder of its own, so GFTk becomes GUIDeFATE::GFTk, starting 1st October. The latest version on GitHub will also be updated roughly at the same time.

Time Flies, Memories leak...GUIDeFATE getting a timer

It appears that I have not had single blog post for 1 whole year. Let's face it, there isn't enough time in the day to everything one might want to do. Procrastination steals more than just time though. The longer one is away from a particular activity the more difficult it is to return to that activity physically or intellectually. Some of you will recall that I developed what I claimed to be the world's simplest GUI designer, GUIDeFATE. This was the topic of a couple of presentations already but as all such projects, remains in a state of flux.

One key component of a GUI is a continuous loop that keeps monitoring human interactions and responding to this by updating the interface. What if we wanted the interface to update continuously, like a monitor, without a connection with a human input trigger? It is certainly not possible, for example to create an animation by just doing foreach(1..$x){sleep $interval; updateImage();}. This merely blocks the GUI mainloop, until the loop is completed with the display updating only after the loop completes and the control goes back to the GUI's mainloop. Instead one needs an event monitor, and more specifically a timer. The ideal scenario may have been to use something like AnyEvent or Ev, but sadly this can not be applied to all the backends; at least I do not know how one might do this. What follows is my usual trawl through the internet to source examples of timers in GUI applications with each of the various backends.

Plug-in Timer

I discovered that Tk and Gtk can work with AnyEvent/AE fairly easily. Qt and Wx have their own way to create timers. While Qt seems to be supported in the documentation for Any::Event, the general recommendation appears to be not to use Qt. Qt Slots have to be predefined and can not be made on the fly. A little jiggery pokery and we have a simple way to abstract timers through multiple back ends. BUT all is not as straightforward as one would like, as I discovered when trying test animations in various backends.

To abstract the timer in the various supported back-ends, Gtk3, Gtk2 (thanks chanio), Qt4, Wx from the UI string requires a single line added to the textual representation of the UI e.g.

my $window=<<END;
|T Animation               |
| +I--------------------+  |
| | clock.svg           |  |
| |                     |  |
| |                     |  |
| |                     |  |
| |                     |  |
| |                     |  |
| +---------------------+  |
|                          |

timer 100 ecg 1


GUIDeFATE detects the timer line, and in the case above, executes the subroutine "ecg" every 100 milliseconds. Any::Event (gtk or Tk) or Wx will happily accept a callback function reference from a string...e.g. \&{"main::$functionName"}, Qt needs a predefined callback subroutine as a slot. While this subroutine could then be used to jump to the desired subroutine, there is some inflexibility in that the number of timers that one could create are limited.

Testing timers.

I figured the best visual way to test a continuous timer would be to make a slide show updating an image window; or perhaps using a image that is updated to make an animation. The easily portable way to make an animation in Perl (without added dependencies) I figured, would be to make SVG images with only minor textual changes (something Perl is good at) required to make a graphical change. Here is GUIDeFATE being used in an application running 4 different animations made by continuously redrawing an SVG file and loading the recreated image into a frame.


When I tested the timers for such animations I found 1) that Wx, Tk and Qt applications progressively used bigger and bigger amounts of memory, with Wx particularly slowing down to a stand-still after a few hundred timer loops 2) Gtk and Gtk2 seemed to use much the same amount of memory regardless of how long it ran for. Now Tk and Wx both used Image magick to put the images into the image panel. Could this be the cause? more testing is necessary.

Things to do

GUIDeFATE 0.13 is not yet on Github or CPAN...I need to sort a few things out first.

1) One of my targets is to restructure GUIDeFATE distribution so that it does not pollute the namespace as much as it does now. I would like to move the back end modules so that they are not in the root names space e.g. GUIDeFATE::GFWx. Before that (I think) I need to wipe the old GUIDeFATE from CPAN...not sure how to to do that without creating havoc.

2) I need a few methods to delete timers, change intervals, or the callback functions...should be easy, but need to to choose methods that will be easy, and of course test them.

3) I need to test on non-Linux machines...I have no Windows PCs or Macs.

Hopefully will have this ready by the next Perl Meeting, who knows?

LOGOesque extension to Language::SIMPLE - Feature creep diverts attention.

In an attempt to standardise vector graphical operations in GUIDeFATE (the world's simplest GUI designer) across different back-ends and keep things simple, a little diversion was needed. For a standard vector drawing one might use the computationally cumbersome SVG format directly and draw that on the widget. The problem is that one needs to be able to manipulate the graphics easily and AFTER deployment. I needed to script the creation of the drawing. Now lots of applications use a script language of their own, e.g. gnuplot, matlab, Kalgebra, R etc, so clearly this is a useful functionality on its own. What would be also useful is a portable scripting platform. This platform could serve my purpose, but being modular, could be easily repurposed for other applications.

Enter Language::SIMPLE (not yet on CPAN), the Simple Integrated Modular Programming Language Experiment, extended it with a LOGO to SVG translator. Dubious, I know, and vastly more powerful SVG and DOM modules exist already. But the main strength of Perl has been its ability to parse and manipulate textual such as an SVG file or a script that generates that file. And the goal of this utility would have been to generate complex graphics from simple scripts, the scripts themselves potentially being generated programmatically. These graphics could then be loaded into a widget for display. Sure, the goal is not to be Inkscape, but to provide a reasonably usable set of textual graphical commands that could be scripted to create fairly complex vector imagery in a portable format, and integrateable into a GUI.

The architecture maybe could look something like this:- Now I am not sure the best way to do this. The modules are application specific. The modularity currently comes from using a do statement:-

   # $file is the extension file and contains the *contents* of a hash
   #  this hash is then inserted into %extensions  
   my %extension=do $file or die "Failed to load extension in $file $!";
   foreach my $thingy (keys %extension){

This loads the module which is structured as the contents of a hash. The current version of logo.ext supports creating lines, poly-lines, polygons, paths, circles, rectangles, text, colours and fills using LOGO-esque commands. SIMPLE already offers variable definition, arithmetic and some trigonometric functions along with loops, conditional branching. So it is now relatively trivial to draw images on the screen, whether it is the graphical explanation diagram above, or some random flowers below, and either save an SVG file, or insert into a widget (e.g. in a GUIDeFATE application, with any of its back-ends: Tk, Qt, GTK, Wx, and who knows, possibly Win32 etc)


Certainly it was not the intention to keep adding features to the extension, rather than focusing on GUIDeFATE. But it has been relatively simple to add commands, and I do hope to add groups, patterns and transforms. Animation should be possible thoughI am certain the performance is not going to be great, (unlikely to be able to recreate Asteroids or Elite). Adding limited 3D graphics like OpenSCAD would be another interesting diversion.

One thing for certain, testing SIMPLE and the logo extension has been made much easier because of the ability to rustle up a quick and dirty GUI application using GUIDeFATE. Which reminds me, I do need to get back to fixing GUIDeFATE proper. But few more tweaks first...

Language::SIMPLE - The Portable Integrated Scripting System

Introducing SIMPLE

SIMPLE is an experimental attempt to integrate end user scripting into Perl applications. It actually stands for Simple Integrated Modular Programming Language Experiment. The idea is that an application one develops might allow an end-user to run scripts they have made, in a custom language targeting the application's operations. Sounds silly? A script parsing utility written in another scripting language? I do a lot of silly stuff.

Some time ago, IpiGears1.jpg attempted to use a Raspberry pi for robotic and other IO applications. To save time and allow for flexibility, I had developed piGears, a scripting tool that could handle the GPIO and also have some basic flow control as well as support for I2C devices. It had been useful for my experiments with electronics, but was a rather niche utility.

Fast forward a few years and I am working on GUIDeFATE (probably the world's simplest GUI designer). I wanted a way to integrate drawing vector images to a canvas and manipulating them, and perhaps exporting them. GUIDeFATE was designed to handle a number of back-end graphical toolkits, and each had had quite different ways of handling the canvas.

Now, I confess, my coding skills are rather mediocre. Complex graphical manipulations would require a certain degree of struggling from me. But all the existing backends for GUIDeFATE can potentially handle SVG. If only I could create a way for the user to produce SVG graphics simply...

I realised I could rejig the ancient code I had written to make a more generic scripting module. All this module would need to do is to be a script interpreter allowing simple variables, subroutines and flow control, onto which anyone could bolt on customised commands and bingo your end-user can control robots, draw pictures programmatically or do anything, Now does that make sense? For many, probably not. But if you were going to parse a text script, the best language to do that in is probably Perl.

Enter Language::SIMPLE. It contains elements necessary to make a programming language without actually any components that make the programming language do anything useful. It can parse a set of instructions. It understands some variables, although technically all its variables are global. It understands subroutines, some limited flow control and code folding. With a little bit of flesh on this skeleton more sophisticated tasks can be performed from end-user scripts. SIMPLE is not on CPAN, and wont be at least until version 0.05, and only if there is some demand for such a utility. With the demo extension it is trivial to make a vector drawing program using LOGO-esque scripts.

Screenshot from 2018-08-29 23:02:57.png