This worries me a bit I must admit; a few years ago, before the official release of P6, I tried my hand at concurrent programming in rakudo and it was so broken that i simply shelved P6 for over a year, it was simply not ready for a lambda dev like me. I would have loved to be in the trenches but I did not have the time to fight with core dumps.
As the speed of P6 is not stellar, I believe many of us, lambdas, will try to run things in parallel, it's also fun, to make them more attractive. The concurrency mechanism in P6 look good on paper, good enough so I can find myself dreaming of throwing in a few keywords in places I think concurrency would make a difference.
There is a nice silver lining to this one, It took a very short time before a lead developer answered me. I believe his answers may be interesting to read.
A description of the problems I encountered is here: gist
]]>I also noticed that 90% of my time went into DDT, the rest went into two puny modules not worth mentioning, except that 3 modules sound better than one. Development takes times, development with documentation and test, and trying to get quality and functionality takes ages, no wonder Perl6 took this long; Actually it is pretty impressive what was achieved 2 decades.
DDT has matured enough so it is time for a new interface, the old interface is deprecated but still there, for a while at least. Few weeks ago I introduce ddt to replace dump as an interface, one of the interfaces. It was both to make it closer to dd the rakudo built-in dumper, and also to stop using the work dump which I don't like much. with all the work put in it is still a dump, no way.
The new interface look like this:
sub/method **ddt** $data_to_dump, $data_to_dump, *:adverb*, :named_argument, ...
This interface accepts the following adverbs:
:print prints the rendered data, the default behavior without adverb
:note 'note's the rendered data
:get returns the rendered data
:get_lines returns the rendering in its native format
:getlinesintegrated returns a list of rendered lines
:curses opens a NCurses interface, NCurses module must be installed
:remote sends data to a listener
Most of the time you'd just ddt your data structure and option and be on you merry way.
:curse is new, when called with that adverb, DDT will open a curses window, render the structure and let you move around, folding and unfolding data as necessary. It's not fast but it is the best way to look at complex data so far (apart from the DHTML rendering). the only limitation, except speed, is that it can not render unicode yet, that is a limitation of the NCurses module not DDT.
:remote, is also new, this one I needed when developing :curses and it is quite practical so I gave it an interface. It sends the rendering to "listener" which prints it on its terminal. If you are using Tmux or Screen, you can split your window and have data rendered in another terminal.
I think DDT is now feature complete, I want it to be because I badly want to start working on Asciio 2.0 and preferably in Perl6.
There are still a few things left:
Profile it to speed it up a bit. I did try once but no success, maybe some good hearted guru will give a hand or at least pointers. Now DDT is a tad heavy, does a lot of things it can't be expected to be a speed demon.
Add more built-in types, specially "exotic" ones like taps and supplies.
Test with Releases rather than bleeding edge,
I think there will be a :remote_fold when I get the time to write it. The idea is that you can send a data rendering, to another process, that you can fold, scroll, search, ... when I have a little time.
find a co-maintainer, since we are just a few hundreds all busy with loads of things, it is going to a hunt. If you have some skill with html/javascrip, the DHTML module could use so polishing touch.
the-cant-be-without-screenshot
]]>I have programmed in C and C++ many years; from hardware related code to mangling libraries for RPC across different cpu architectures . Few years ago we had a very large testing and integration project that used Perl as the main development language; we had to write quite some XS even if we used SWIG extensively. So NativeCall is something that is really of interest even if nowadays I have little use for it, I still think it's a nice way of interfacing libraries.
Data::Dump::Tree works with types but NativeCall works with representation, types, and what I believe is dynamically create types for array (with VMArray representation), so it took some times to comprehend and get DDT to groak it.
The documentation for NativeCall is here https://docs.perl6.org/language/nativecall#Getting_Started
I'll walk you through how the different NativeCall elements are displayed by DDT. The general principle is to make it clear what belongs to NativeCall while hiding a bit of the naming specific to NativeCall.
DDT will show that a sub is a NativeCall sub by adding the "NativeCall" tag
sub add_p6(Int, Int) returns Int { * } # normal sub
sub some_argless_function() is native('something') { * }
our sub init() is native('foo') is symbol('FOO_INIT') { * }
sub add(int32, int32) returns int32 is native("calculator") { * }
sub Foo_init() returns Pointer is native("foo") { * }
There are two ways to declare pointers
class MyHandle is repr('CPointer') {...}
my Pointer[int32] $pointer ;
my int32 $int32 = 7 ;
my @with_int32 = $int32, 7, 8 ;
my int32 @int32 = $int32, 7, 8 ;
my $string = "FOO";
my @carray := CArray[uint8].new($string.encode.list);
my $carray_titles = CArray[Str].new;
$carray_titles[0] = 'Me';
$carray_titles[1] = 'You';
In the code I create an int32, put it in a normal array, create an int32 @array, and two CArrays.
As you can see the int32 displays as an Int (newest version of DDT does not display the type for Ints to reduce noise). That is because being an int32 is an attribute of the variable not the value. The same goes for an int32 put in a normal array, it displays as an Int.
But int32 @array is different, we can catch it by type not representation. DDT sees it belongs to NativeCall, it adds "array" tag to it.
CArrays are handled in a similar manner, make them look as normal type arrays plus the "CArray" tag.
class Types is repr('CStruct') {
has int8 $.a1 ;
has int16 $.a2 ;
has int32 $.a3 ;
has int64 $.a4 ;
has uint8 $.a5 ;
has uint16 $.a6 ;
has uint32 $.a7 ;
has uint64 $.a8 ;
has long $.a9 ;
has longlong $.a10 ;
has ulong $.a11 ;
has ulonglong $.a12 ;
has num32 $.a13 ;
has num64 $.a14 ;
has Str $.a15 ;
has CArray[int32] $.a16 ;
has Pointer[void] $.a17 ;
has bool $.a18 ;
has size_t $.a19 ;
has ssize_t $.a20 ;
}
ddt (Types, Types.new), :flat(0) ;
When you render an undefined NativeCall structure, ddt will list the elements of the structure and their type on the right-hand side. If you render a defined NativeCall structure, ddt renders then name of the element followed with '.' ~ type of the element in the C world, on the right hand side the value in the Perl world.
class Point is repr('CStruct') {
has num64 $.x;
has num32 $.y;
has int32 $.z = 3;
}
my $point = Point.new: :x(2e56), :y(10e10) ;
class Parts is repr('CUnion') {
has int32 $.xyz;
has int64 $.abc;
}
my Parts $union = Parts.new: :abc(10 ** 10) ;
class MyStruct is repr('CStruct') {
has Point $.point; # referenced
has int32 $.flags;
}
You can also create a DDT handler for a NativeCall structure you create, this makes it up to you to render the structure. Below a very simple example that just renders a string.
#define a NativeCall structure
class StructiWithHandler is repr('CStruct')
{
has int32 $.flags;
}
# define a DDT handler
role DDT_SWH
{
multi method get_header (StructiWithHandler $s)
{ 'In DDT Handler', '.' ~ $s.^name, DDT_FINAL }
}
I need to understand the difference between has and HAS, from NativeCall, and see if I can differentiate them. I also would like to run on a lot of different input data; if you are using NativeCall, you can help by trying it on your data and send me the results. If you encounter errors, open an issue with code to repropduce it or reach for me on #perl6.
]]>It also
can display two data structures side by side (DDTR::MultiColumns)
can display the difference between two data structures (DDTR::Diff)
can generate DHTM output (DDTR::DHTML)
can display a folding structure in Curses (DDTR::Folding)
can display parts of the data structure Horizontally ( :flat() )
can be used to "visit" a data structure and call callbacks you define
install Term::ANSIColor and get colored output (highly recommended)
This blog entry is about the the latest changes.
I'll start with dd, the rakudo built-in dumper, which I learner to hate and love. I certainly love it when I am debugging my module and I can't use my own data dumper. I have in this version copied a few of its features which I found better.
This new release would not happen without the endlessly caring people on #perl6.
There is an added interface to Data::Dump::Tree, you can now use ddt instead for dump. I added this to make it easy to sneak in a dump via ddt if you already have typed d. ddt is not a replacement for dd I see it more as a complement for when dd's output is incomprehensible, that is when your dump is more than two or three lines of output. dd is also not good with references.
You can also "use DDT ;" instead for "use Data::Dump::Tree ;".
On the other hand dd is much slower. And some options make it slower but it is not a tool for fast generation but for data comprehension.
I will only talk about the latest additions in this blog, check the documentation for all the details, also check the example directory that I try to keep well fed.
ddt takes multiple argument
ddt $a, @b, :options ;
ddt will display the name of the dumped variables
Display of sequences was minimal till you used a role. the role has been integrated, because I found myself displaying a lot of sequences and It was a pain to always have to use the role for such a basic type.
Matches still have a minimalist display and you have to use a role to get more from your matches, and by more I mean much more legible output than from dd
Here is an example with a few filters applied. Without the filter it would still be much easier to comprehend but not at compact. The example is ten times shorter that dd output
The module generates folding data which can be used to fold and unfold data in a command line application. A generous soul may want to help build such application. There is a Curses example but I have not worked on it much.
The dhtml rendering now uses the colors defined in the dumper so the output looks very close in a browser and a terminal.
display is attribute rw trait
Attributes origin
here an Array that is also a MaxLines
ddt dumps data vertically which means that there can be some scrolling involved when rendering large data. In some cases you want your data to be displayed in a more compact way. * :flat * has been added as an option, it has an api of its own and the documentation can be found in the LayoutHorizontal module. You get control of what and where gets flattened.
here is an example with data at level zero being laid flat with dd output for comparison, note that you can forget about references being showed by dd:
Laying hashes flat or just the first level flat already makes the rendering more compact and still is readable. But laying arrays flat is counter intuitive; you get the values after each other when you want to have them in more readable columms. ddt supports columns (under your control). the rendering in flattened layout is a slower as each and every element is rendered individually.
here an example of a more complex data structure:
ANSI glyph have been removed and replaced by unicode
better rendering of junctions, pairs and sets
dumped values can contain ANSI colors
and you can display renderings beside each other
in this example a flattened rendering side by side with a vertical layout rendering:
]]>Help by adding your requirements as an issue at https://github.com/nkh/Asciio-2.0
Integration with Ditaa is on the list since someone made most of the work, without even contacting me once. http://wiki.cornempire.net/_media/asciiart/diagram2.png
SVG output will be via a2s, https://github.com/dhobsd/asciitosvg, as the author is already an asciio user and some of his users are too.
The project welcomes help.
]]>And suddenly I remembered that I promised myself, at YAPC::EU, that I would write a tmux plugin to implement a yakuake like pane handling.
With some time and energy on my hands I created a directory, initialize a repo and was on my way to write some perl. I took one hour to draw a few ideas and that's when GranPa Bash knocked at the door.
I know, you know, everyone knows that Perl build a lot on bash, at least its syntax, and in this very case it seemed to me that writing this in Bash would be better. It's mainly command calling, there is no complicated process to map, it should be quite easy in bash.
And I was in the mood!
Every second year I beat myself up because I always refuse to learn more sed or awk. It's not that I don't use them, it's just that every time I do, I get so frustrated that I end up writing some Perl.
As for Bash, Having had the displeasure of fixing a huge code database that I did not write, I simply think it's the devil's work and I have forbidden it in all the project I manage. If it's more than 5 lines of Bash, then it should be written in Perl; no discussion. A rule I am keeping (well, not in this case).
But this time I was on! I'd write some Bash, and maybe learn a thing or two about it in the process.
Note that it is a bit of a silly idea to learn more Bash that I know I will not let myself use when I have Go (finally people are writing command line apps again, the Java years were miserable years), and JS/NodeJS and so much fun P6 on my list. But I thought that after writing a P6 module, it would be fun to look at more "archaic" languages, there's a lot to learn from them, the design considerations they implement, and the people who wrote them. I have a great respect for what developers did years ago just an evening spend in coreutils docs makes me go "hmmm, haaaa, ohhh", Bash is also fascinating.
Anyone complaining about 'my' in Perl should be forced to write Bash with 'local' and see how much fun it is. It's actually as much fun as writing with everything global, because sometime it does not do what I expect it to do. I ended up removing local!
I learned a other things, I won't call functions some-name, with a dash, ever again in my life as ctags refused to parse them. Why would one ever want to do that? tmux functions are all some-name in format. Following like a sheep is comfortable but not always a good idea.
I also needed two characters to tag panes, that lead to a miserable period trying to find some unicode characters that would not mess konsole's display. It's also frustrating to find a unicode character that looks like one wants. Larry Wall made it look sooo easy with Chinese characters, I find it difficult to find a house symbol and even more difficult to find one that displays correctly. but I also found this little gem of a web page http://shapecatcher.com/index.html which let's you draw what you are looking for.
I learned two things. I love-hate Bash as much as I love-hate make (and all its derivated small mongrels). I also understand better how much of a liberation Perl must have been for people programming in Bash. It may look a bit the same but while Bash is a charming, banjo playing, crazy GranPa, Perl is a the uncle, with the huge tool shed, you want to be like when you grow up.
I also found out that I am happy to be part of a community that appreciate many languages, I may find Python boring to write code in but I can see how nice and useful it is, a great language actually; I grind my teeth at large Bash script but there's also beauty in it. All this I learned because the Perl developers I have been in contact use many languages, There is little to no dogmatism and we don't spend our time slandering at languages we are not intelligent enough to use, we fix stuff, we invent, we cooperate and if needed coerce other bits and pieces in our solutions. We are solution people.
We also have a great community, and I could see that clearly at YAPC::EU, and a great language and many other cool languages to play with. We live in a fascinating time.
Long live Perl and its GrandPa (and mongrels).