Promises promises ...

Problems with concurrency, a post from the hard reality.

This worries me a bit I must admit; a few years ago, before the official release of P6, I tried my hand at concurrent programming in rakudo and it was so broken that i simply shelved P6 for over a year, it was simply not ready for a lambda dev like me. I would have loved to be in the trenches but I did not have the time to fight with core dumps.

As the speed of P6 is not stellar, I believe many of us, lambdas, will try to run things in parallel, it's also fun, to make them more attractive. The concurrency mechanism in P6 look good on paper, good enough so I can find myself dreaming of throwing in a few keywords in places I think concurrency would make a difference.

There is a nice silver lining to this one, It took a very short time before a lead developer answered me. I believe his answers may be interesting to read.

A description of the problems I encountered is here: gist

The emperor's new clothes

I noticed that I have been working on Data::Dump::Tree for 18 months which makes me a Perl6 developer with almost 2 years of "experience", \o/. What I know is that I am going to need a few more years to get around all it has to offer. But after two years it is still fun to learn new things about Perl6.

I also noticed that 90% of my time went into DDT, the rest went into two puny modules not worth mentioning, except that 3 modules sound better than one. Development takes times, development with documentation and test, and trying to get quality and functionality takes ages, no wonder Perl6 took this long; Actually it is pretty impressive what was achieved 2 decades.

DDT has matured enough so it is time for a new interface, the old interface is deprecated but still there, for a while at least. Few weeks ago I introduce ddt to replace dump as an interface, one of the interfaces. It was both to make it closer to dd the rakudo built-in dumper, and also to stop using the work dump which I don't like much. with all the work put in it is still a dump, no way.

The new interface look like this:

sub/method **ddt** $data_to_dump, $data_to_dump, *:adverb*, :named_argument, ...

This interface accepts the following adverbs:

  • :print prints the rendered data, the default behavior without adverb

  • :note 'note's the rendered data

  • :get returns the rendered data

  • :get_lines returns the rendering in its native format

  • :getlinesintegrated returns a list of rendered lines

  • :curses opens a NCurses interface, NCurses module must be installed

  • :remote sends data to a listener

Most of the time you'd just ddt your data structure and option and be on you merry way.

:curse is new, when called with that adverb, DDT will open a curses window, render the structure and let you move around, folding and unfolding data as necessary. It's not fast but it is the best way to look at complex data so far (apart from the DHTML rendering). the only limitation, except speed, is that it can not render unicode yet, that is a limitation of the NCurses module not DDT.

:remote, is also new, this one I needed when developing :curses and it is quite practical so I gave it an interface. It sends the rendering to "listener" which prints it on its terminal. If you are using Tmux or Screen, you can split your window and have data rendered in another terminal.

I think DDT is now feature complete, I want it to be because I badly want to start working on Asciio 2.0 and preferably in Perl6.

There are still a few things left:

  • Profile it to speed it up a bit. I did try once but no success, maybe some good hearted guru will give a hand or at least pointers. Now DDT is a tad heavy, does a lot of things it can't be expected to be a speed demon.

  • Add more built-in types, specially "exotic" ones like taps and supplies.

  • Test with Releases rather than bleeding edge,

  • I think there will be a :remote_fold when I get the time to write it. The idea is that you can send a data rendering, to another process, that you can fold, scroll, search, ... when I have a little time.

  • find a co-maintainer, since we are just a few hundreds all busy with loads of things, it is going to a hunt. If you have some skill with html/javascrip, the DHTML module could use so polishing touch.

the-cant-be-without-screenshot

Imgur

Take a walk on the C side, ddt, du du, du du ...

Someone on #perl6 asked if Data::Dump::Tree (DDT) could display an int32 properly, and the adventure began.

I have programmed in C and C++ many years; from hardware related code to mangling libraries for RPC across different cpu architectures . Few years ago we had a very large testing and integration project that used Perl as the main development language; we had to write quite some XS even if we used SWIG extensively. So NativeCall is something that is really of interest even if nowadays I have little use for it, I still think it's a nice way of interfacing libraries.

Data::Dump::Tree works with types but NativeCall works with representation, types, and what I believe is dynamically create types for array (with VMArray representation), so it took some times to comprehend and get DDT to groak it.

The documentation for NativeCall is here https://docs.perl6.org/language/nativecall#Getting_Started

I'll walk you through how the different NativeCall elements are displayed by DDT. The general principle is to make it clear what belongs to NativeCall while hiding a bit of the naming specific to NativeCall.

Subs

DDT will show that a sub is a NativeCall sub by adding the "NativeCall" tag

sub add_p6(Int, Int) returns Int  { * } # normal sub
sub some_argless_function() is native('something')  { * }
our sub init() is native('foo') is symbol('FOO_INIT') { * }
sub add(int32, int32) returns int32 is native("calculator") { * }
sub Foo_init() returns Pointer is native("foo") { * }

Imgur

Pointers

There are two ways to declare pointers

class MyHandle is repr('CPointer') {...}
my Pointer[int32] $pointer ;

Imgur

Arrays

my int32 $int32 = 7 ;
my @with_int32 = $int32, 7, 8 ;
my int32 @int32 = $int32, 7, 8 ;

my $string = "FOO";
my @carray := CArray[uint8].new($string.encode.list);

my $carray_titles = CArray[Str].new;
$carray_titles[0] = 'Me';
$carray_titles[1] = 'You';

Imgur

In the code I create an int32, put it in a normal array, create an int32 @array, and two CArrays.

As you can see the int32 displays as an Int (newest version of DDT does not display the type for Ints to reduce noise). That is because being an int32 is an attribute of the variable not the value. The same goes for an int32 put in a normal array, it displays as an Int.

But int32 @array is different, we can catch it by type not representation. DDT sees it belongs to NativeCall, it adds "array" tag to it.

CArrays are handled in a similar manner, make them look as normal type arrays plus the "CArray" tag.

Types

class Types is repr('CStruct') {
    has int8 $.a1 ;
    has int16 $.a2 ;
    has int32 $.a3 ;
    has int64 $.a4 ;
    has uint8 $.a5 ;
    has uint16 $.a6 ;
    has uint32 $.a7 ;
    has uint64 $.a8 ;
    has long $.a9 ;
    has longlong $.a10 ;
    has ulong $.a11 ;
    has ulonglong $.a12 ;
    has num32 $.a13 ;
    has num64 $.a14 ;
    has Str $.a15 ;
    has CArray[int32] $.a16 ;
    has Pointer[void] $.a17 ;
    has bool $.a18 ;
    has size_t $.a19 ;
    has ssize_t $.a20 ;
}

ddt (Types, Types.new), :flat(0) ;

When you render an undefined NativeCall structure, ddt will list the elements of the structure and their type on the right-hand side. If you render a defined NativeCall structure, ddt renders then name of the element followed with '.' ~ type of the element in the C world, on the right hand side the value in the Perl world.

Imgur

CStructs, CUnions, and complex structures

class Point is repr('CStruct') {
    has num64 $.x;
    has num32 $.y;
    has int32 $.z = 3;
}

my $point = Point.new: :x(2e56), :y(10e10) ;

class Parts is repr('CUnion') {
    has int32 $.xyz;
    has int64 $.abc;
}

my Parts $union = Parts.new: :abc(10 ** 10) ;  

class MyStruct is repr('CStruct') {
    has Point $.point;  # referenced
    has int32 $.flags;
}

Imgur

Type handlers

You can also create a DDT handler for a NativeCall structure you create, this makes it up to you to render the structure. Below a very simple example that just renders a string.

#define a NativeCall structure
class StructiWithHandler is repr('CStruct')
{
    has int32 $.flags;
}

# define a DDT handler
role DDT_SWH
{
multi method get_header (StructiWithHandler $s)
    { 'In DDT Handler', '.' ~ $s.^name, DDT_FINAL  }
}

Imgur

What's left?

I need to understand the difference between has and HAS, from NativeCall, and see if I can differentiate them. I also would like to run on a lot of different input data; if you are using NativeCall, you can help by trying it on your data and send me the results. If you encounter errors, open an issue with code to repropduce it or reach for me on #perl6.

Perl 6 Data::Dump::Tree version 1.5

For those who do not know the module, Data::Dump::Tree renders your data structures in a tree fashion for legibility. https://github.com/nkh/P6-Data-Dump-Tree

It also

  • can display two data structures side by side (DDTR::MultiColumns)

  • can display the difference between two data structures (DDTR::Diff)

  • can generate DHTM output (DDTR::DHTML)

  • can display a folding structure in Curses (DDTR::Folding)

  • can display parts of the data structure Horizontally ( :flat() )

  • can be used to "visit" a data structure and call callbacks you define

  • install Term::ANSIColor and get colored output (highly recommended)

This blog entry is about the the latest changes.

I'll start with dd, the rakudo built-in dumper, which I learner to hate and love. I certainly love it when I am debugging my module and I can't use my own data dumper. I have in this version copied a few of its features which I found better.

This new release would not happen without the endlessly caring people on #perl6.

Snatched from dd

There is an added interface to Data::Dump::Tree, you can now use ddt instead for dump. I added this to make it easy to sneak in a dump via ddt if you already have typed d. ddt is not a replacement for dd I see it more as a complement for when dd's output is incomprehensible, that is when your dump is more than two or three lines of output. dd is also not good with references.

You can also "use DDT ;" instead for "use Data::Dump::Tree ;".

On the other hand dd is much slower. And some options make it slower but it is not a tool for fast generation but for data comprehension.

I will only talk about the latest additions in this blog, check the documentation for all the details, also check the example directory that I try to keep well fed.

More from dd

  • ddt takes multiple argument

    ddt $a, @b, :options ;

  • ddt will display the name of the dumped variables

Integration of Seq

Display of sequences was minimal till you used a role. the role has been integrated, because I found myself displaying a lot of sequences and It was a pain to always have to use the role for such a basic type.

Match objects

Matches still have a minimalist display and you have to use a role to get more from your matches, and by more I mean much more legible output than from dd

Here is an example with a few filters applied. Without the filter it would still be much easier to comprehend but not at compact. The example is ten times shorter that dd output

Imgur

Folding

The module generates folding data which can be used to fold and unfold data in a command line application. A generous soul may want to help build such application. There is a Curses example but I have not worked on it much.

DHTML

The dhtml rendering now uses the colors defined in the dumper so the output looks very close in a browser and a terminal. Imgur

Attributes:

  • display is attribute rw trait

  • Attributes origin

Imgur

role support for types having a handler in DDT

here an Array that is also a MaxLines

Imgur

Vertical, horizontal, and columnar layout

ddt dumps data vertically which means that there can be some scrolling involved when rendering large data. In some cases you want your data to be displayed in a more compact way. * :flat * has been added as an option, it has an api of its own and the documentation can be found in the LayoutHorizontal module. You get control of what and where gets flattened.

here is an example with data at level zero being laid flat with dd output for comparison, note that you can forget about references being showed by dd:

Imgur

Laying hashes flat or just the first level flat already makes the rendering more compact and still is readable. But laying arrays flat is counter intuitive; you get the values after each other when you want to have them in more readable columms. ddt supports columns (under your control). the rendering in flattened layout is a slower as each and every element is rendered individually.

here an example of a more complex data structure: Imgur

Divers

  • ANSI glyph have been removed and replaced by unicode

  • better rendering of junctions, pairs and sets

  • dumped values can contain ANSI colors

  • and you can display renderings beside each other

in this example a flattened rendering side by side with a vertical layout rendering:

Imgur

App:Asciio 2.0

I started gathering requirements for the next version of Asciio which will probably be written in Perl6 (Go is a strong contender albeit not as much fun)

bunny.png

Help by adding your requirements as an issue at https://github.com/nkh/Asciio-2.0

Integration with Ditaa is on the list since someone made most of the work, without even contacting me once. http://wiki.cornempire.net/_media/asciiart/diagram2.png

SVG output will be v…