Lee J

At YAPC::EU 2016 I gave a talk on my approach to developing code against RESTful services. The talk starts out a little silly, but my aim was to show some of the frustrations that can arise when developing aforementioned code. My conclusion is that you should write an emulator for any service you are developing against. Not just that but release an emulator for any RESTful APIs you are developing for others so they can trivially test their client code.

Of course I am a developer so inherently lazy, and being a perl developer I am especially lazy. Having done the emulation dance for at least three modules I've written I suggested I would write something to make this easier. I managed to find some time last week, amongst our annual developer's conference, to do this.

With the upload of JSON::Schema::ToJSON it is now trivial to emulate just about any RESTful JSON API with a combination of that module, Mojolicious, OpenAPI, and optionally OAuth2::Server. When I say trivial I mean it, the following generic emulator is included with the git repository (Note: requires an up to date version of Mojolicious::Plugin::OpenAPI, v1.09):

use Mojolicious::Lite; # "strict", "warnings", "utf8" and Perl 5.10 features
use JSON::Schema::ToJSON;

my $spec_uri    = shift
    || die "Need a spec URI: $0 <spec_uri> <base_path> [<example_key>]";
my $base        = shift
    || die "Need base path: $0 <spec_uri> <base_path> [<example_key>]";
my $example_key = shift;

plugin OpenAPI => {
    route => app->routes->under( $base )->to( cb => sub { 1; } ),
    url   => $spec_uri,
};

app->helper( 'openapi.not_implemented' => sub {

    if ( my $responses = shift->openapi->spec->{'responses'} ) {
        if ( my ( $response ) = grep { /^2/ } sort keys( %{$responses} ) ) {

            my $ret = $responses->{$response}{description} // '';
            if ( my $schema = $responses->{$response}{schema} ) {
                $ret = JSON::Schema::ToJSON->new->json_schema_to_json(
                    schema      => $schema,
                    example_key => $example_key,
                );
            }
            return {json => $ret, status => $response};
        }
    }

    return {
        status => 501,
        json   => {
            errors => [{message => 'Not implemented.', path => '/'}]
        }
    };
});


app->start;

Passing it an openapi (swagger) spec + base path on the command line allows you to call the endpoints defined in the spec. Let's try it with the Instagram spec, as listed at https://github.com/APIs-guru/openapi-directory

> morbo ./emulators/generic.pl \
    https://tinyurl.com/jls2km7 \
    /v1 \
    -l 'https://*:3000'
Server available at https://127.0.0.1:3000

Then call it:

> curl -s -k -X GET "https://127.0.0.1:3000/v1/tags/landscape" | jq .
{
  "data": {
    "media_count": 116,
    "name": "]yeq3@j^I'^v6kc)1i"
  },
  "meta": {
    "code": 206
  }
}

As you can see the emulator returns a representative data structure for the endpoint, whereas the actual data contained within that structure is random (it's possible to take example from the spec if you provide an example_key argument to the constructor). What about an endpoint that has a lot more data?

> curl -s -k -X GET "https://127.0.0.1:3000/v1/users/self/media/liked" | jq .
{
  "data": [
    {
      "attribution": "C($?wi\\z%m+x4O:l,m}73[2aBPliIZ$L",
      "comments": {
        "count": 820,
        "data": [
          {
            "created_time": "8c76W6RFv-5hpL3I",
            "from": {}
          }
        ]
      },
      "created_time": "qXrxma?Gnwx",
      "filter": "/YpDllbpBI6iekXw_\"}Ycp$6^&YMKG#C}dX+x`l\\?Z2[",
      "id": "{skN}W#GbTq_&J,)i:)4aB}mlgj6sLDcm/]J'yfcG)6A=",
      "images": {
        "low_resolution": {}
      },
      "likes": {
        "count": 225,
        "data": [
          {},
          {}
        ]
      },
      "location": {
        "id": "Df}lBZt*XU/<{.5!W}[\\~yN2C7J#%(:@mUCQum8)c-e",
        "latitude": 797.2,
        "longitude": 982.6,
        "name": "!<8NN!^|Pu#~_wRRve`xc\\IL~_'"
      },
      "tags": [
        "|!'uQzllb8y$jpU>axy^'nj5nFQ1=",
        "\"djh:JvJ?WK='WFb5nm<%suV-.Bh%#6;F%+iDx5eQ",
        "^a\\U9S5{c`qE-<=6*YKbIZ:w"
      ],
      "type": "video",
      "user": {
        "full_name": ";SdEk#fcN|%d#=ArCO`zm9*vH",
        "id": "ajoO4qVsp9~m(A;bY{|n|vQ|wb$Z^<>",
        "profile_picture": "'a6.7)f%SSfd&d[SB]6",
        "username": "=&_$}.G@XG&{^E6X0wg,F"
      },
      "user_has_liked": false,
      "users_in_photo": [
        {
          "user": {}
        }
      ]
    },
    {},
    {},
    {}
  ],
  "meta": {
    "code": 464
  },
  "pagination": {
    "next_max_id": "}tWCVq:KOd$m]Gdu4X%#FAG$5wo",
    "next_url": "F[7D$XuQF,KA[cH+'s8NNn[TMWAVc}2k8-s"
  }
}

You'll notice in the above example a few empty objects ({}). This is where we hit one of the first limitations of programatically generating example JSON from a JSON Schema - JSON Schema can be self referential, so any recursion can potentially become infinite. So the JSON::Schema::ToJSON module defaults to a max recursion depth of 10 levels. It is possible to increase this by passing an option to the object constructor. There are a couple of other limitations in generating example JSON structures from a JSON Schema - see the module's documentation for these.

What about adding OAuth2 emulation? Also trivial. In an aim to practice what I preach I uploaded an emulator for our WIP API, which features OAuth2 and CORS headers emulation. What about overriding emulated endpoints? Just tweak the route you pass to the OpenAPI plugin so you can install your own (much like you would if you were using the OpenAPI plugin to actually build an API):

my $route = app->routes->under( $base )->to( cb => sub { 1; } );

$route->get( '/users/self/media/liked' => sub {
    return shift->render( json => {
        overriden => Mojo::JSON::true,
    } );
} );

plugin OpenAPI => {
    route => $route,
    url   => $spec_uri,
};

I have tested the JSON::Schema::ToJSON module against several openapi specs and believe I have ironed out most of the bugs. If you find any more bugs in the implementation then do raise an issue on github.

With Mojolicious and an OpenAPI spec you can already do the following:

• Generate the routes in your app
• Automatic validation of incoming request data
• Automatic validation of outgoing response data
• Generate your API docs

Now with JSON::Schema::ToJSON we can add:

• Prototype your API before you start fleshing out the details
• Generate an emulator for dogfooding and client testing

If you're not already using OpenAPI for your RESTful API then I would seriously suggest evaluating it for your next RESTful API.