Les Journées Perl / French Perl Workshop 2017

Nobody seems to have blogged about Les Journées Perl so I thought I would blog about Les Journées Perl. Or maybe I haven't been paying attention, or it passed by me in a state of crossing timezones?

I was travelling back from the UK office the weekend the workshop happened, so stopped by. I also uploaded a couple of photos to the perl_events Instagram feed. The workshop was held in the Carrefour numérique in the Cité des Sciences et de l'Industrie in Paris - a very interesting space within a very interesting space. Possibly one …

The Perl Toolchain Summit 2017 + Perl Events Instagram Feed

I'm shortly going to drive back to Switzerland after attending the Perl Toolchain Summit in Lyon. I was mostly here on photo duties, but between taking photos did manage to file a bug against Time::HiRes, which seems to be down to some unexpected behaviour in File::Spec::Unix after testing the RC of 5.26 against our stack.

I also managed to provide a little help to the metacpan developers by getting to the bottom of an issue upgrading the VM for use w…

Emulating Just About Any RESTful JSON API

At YAPC::EU 2016 I gave a talk on my approach to developing code against RESTful services. The talk starts out a little silly, but my aim was to show some of the frustrations that can arise when developing aforementioned code. My conclusion is that you should write an emulator for any service you are developing against. Not just that but release an emulator for any RESTful APIs you are developing for others so they can trivially test their client code.

Of course I am a developer so inherently lazy, and being a perl developer I am especially lazy. Having done the emulation dance for at least three modules I've written I suggested I would write something to make this easier. I managed to find some time last week, amongst our annual developer's conference, to do this.

With the upload of JSON::Schema::ToJSON it is now trivial to emulate just about any RESTful JSON API with a combination of that module, Mojolicious, OpenAPI, and optionally OAuth2::Server. When I say trivial I mean it, the following generic emulator is included with the git repository (Note: requires an up to date version of Mojolicious::Plugin::OpenAPI, v1.09):

use Mojolicious::Lite; # "strict", "warnings", "utf8" and Perl 5.10 features
use JSON::Schema::ToJSON;

my $spec_uri    = shift
    || die "Need a spec URI: $0 <spec_uri> <base_path> [<example_key>]";
my $base        = shift
    || die "Need base path: $0 <spec_uri> <base_path> [<example_key>]";
my $example_key = shift;

plugin OpenAPI => {
    route => app->routes->under( $base )->to( cb => sub { 1; } ),
    url   => $spec_uri,
};

app->helper( 'openapi.not_implemented' => sub {

    if ( my $responses = shift->openapi->spec->{'responses'} ) {
        if ( my ( $response ) = grep { /^2/ } sort keys( %{$responses} ) ) {

            my $ret = $responses->{$response}{description} // '';
            if ( my $schema = $responses->{$response}{schema} ) {
                $ret = JSON::Schema::ToJSON->new->json_schema_to_json(
                    schema      => $schema,
                    example_key => $example_key,
                );
            }
            return {json => $ret, status => $response};
        }
    }

    return {
        status => 501,
        json   => {
            errors => [{message => 'Not implemented.', path => '/'}]
        }
    };
});


app->start;

Passing it an openapi (swagger) spec + base path on the command line allows you to call the endpoints defined in the spec. Let's try it with the Instagram spec, as listed at https://github.com/APIs-guru/openapi-directory

> morbo ./emulators/generic.pl \
    https://tinyurl.com/jls2km7 \
    /v1 \
    -l 'https://*:3000'
Server available at https://127.0.0.1:3000

Then call it:

> curl -s -k -X GET "https://127.0.0.1:3000/v1/tags/landscape" | jq .
{
  "data": {
    "media_count": 116,
    "name": "]yeq3@j^I'^v6kc)1i"
  },
  "meta": {
    "code": 206
  }
}

As you can see the emulator returns a representative data structure for the endpoint, whereas the actual data contained within that structure is random (it's possible to take example from the spec if you provide an example_key argument to the constructor). What about an endpoint that has a lot more data?

> curl -s -k -X GET "https://127.0.0.1:3000/v1/users/self/media/liked" | jq .
{
  "data": [
    {
      "attribution": "C($?wi\\z%m+x4O:l,m}73[2aBPliIZ$L",
      "comments": {
        "count": 820,
        "data": [
          {
            "created_time": "8c76W6RFv-5hpL3I",
            "from": {}
          }
        ]
      },
      "created_time": "qXrxma?Gnwx",
      "filter": "/YpDllbpBI6iekXw_\"}Ycp$6^&YMKG#C}dX+x`l\\?Z2[",
      "id": "{skN}W#GbTq_&J,)i:)4aB}mlgj6sLDcm/]J'yfcG)6A=",
      "images": {
        "low_resolution": {}
      },
      "likes": {
        "count": 225,
        "data": [
          {},
          {}
        ]
      },
      "location": {
        "id": "Df}lBZt*XU/<{.5!W}[\\~yN2C7J#%(:@mUCQum8)c-e",
        "latitude": 797.2,
        "longitude": 982.6,
        "name": "!<8NN!^|Pu#~_wRRve`xc\\IL~_'"
      },
      "tags": [
        "|!'uQzllb8y$jpU>axy^'nj5nFQ1=",
        "\"djh:JvJ?WK='WFb5nm<%suV-.Bh%#6;F%+iDx5eQ",
        "^a\\U9S5{c`qE-<=6*YKbIZ:w"
      ],
      "type": "video",
      "user": {
        "full_name": ";SdEk#fcN|%d#=ArCO`zm9*vH",
        "id": "ajoO4qVsp9~m(A;bY{|n|vQ|wb$Z^<>",
        "profile_picture": "'a6.7)f%SSfd&d[SB]6",
        "username": "=&_$}.G@XG&{^E6X0wg,F"
      },
      "user_has_liked": false,
      "users_in_photo": [
        {
          "user": {}
        }
      ]
    },
    {},
    {},
    {}
  ],
  "meta": {
    "code": 464
  },
  "pagination": {
    "next_max_id": "}tWCVq:KOd$m]Gdu4X%#FAG$5wo",
    "next_url": "F[7D$XuQF,KA[cH+'s8NNn[TMWAVc}2k8-s"
  }
}

You'll notice in the above example a few empty objects ({}). This is where we hit one of the first limitations of programatically generating example JSON from a JSON Schema - JSON Schema can be self referential, so any recursion can potentially become infinite. So the JSON::Schema::ToJSON module defaults to a max recursion depth of 10 levels. It is possible to increase this by passing an option to the object constructor. There are a couple of other limitations in generating example JSON structures from a JSON Schema - see the module's documentation for these.

What about adding OAuth2 emulation? Also trivial. In an aim to practice what I preach I uploaded an emulator for our WIP API, which features OAuth2 and CORS headers emulation. What about overriding emulated endpoints? Just tweak the route you pass to the OpenAPI plugin so you can install your own (much like you would if you were using the OpenAPI plugin to actually build an API):

my $route = app->routes->under( $base )->to( cb => sub { 1; } );

$route->get( '/users/self/media/liked' => sub {
    return shift->render( json => {
        overriden => Mojo::JSON::true,
    } );
} );

plugin OpenAPI => {
    route => $route,
    url   => $spec_uri,
};

I have tested the JSON::Schema::ToJSON module against several openapi specs and believe I have ironed out most of the bugs. If you find any more bugs in the implementation then do raise an issue on github.

With Mojolicious and an OpenAPI spec you can already do the following:

  • Generate the routes in your app
  • Automatic validation of incoming request data
  • Automatic validation of outgoing response data
  • Generate your API docs

Now with JSON::Schema::ToJSON we can add:

  • Prototype your API before you start fleshing out the details
  • Generate an emulator for dogfooding and client testing

If you're not already using OpenAPI for your RESTful API then I would seriously suggest evaluating it for your next RESTful API.

Paging TOBYINK

This week marks 2 years since TOBYINK seemingly dropped off the perl map, in terms of viewable open source contributions that is. Since then the author's modules have gone unmaintained, consequently they are no longer compatible with recent versions of perl. This is going to cause problems eventually.

I, and several others, have sent e-mails to TONYINK but have not received any response. So TOBYINK, where are you? Are you OK? You seem to be active in at least ="…

Alpine Perl Workshop Videos (Day 1)

I have uploaded videos from the first day of the Alpine Perl Workshop (Innsbruck, 2016): https://www.youtube.com/channel/UCLsVz9vprzs6U6PmcI4tnqA. The second day of videos should be done uploading by the same time tomorrow. Some observations from recording/processing/uploading the videos:

  • A shotgun microphone attached to the camera and set to mono seems to be a reasonable compromise between the near-unusable built-in microphone of the camera and the ideal case of an external microphone used by the speaker.…