Clinton Gormley

RFC: Single or multiple instances of ORM objects?

2012-02-21T15:25:17Z

In our homegrown ORM we have an in-memory cache, which enables us to ensure that only one instance of any object is live in memory at any one time.

In other words:


    $one = MyObject->get(123);
    $two = MyObject->get(123);

    refaddr($one) == refaddr($two)

I find this setup useful because:

if you update one copy of the object, all other copies automatically update
get’ing the object again is cheap

When I do a search against the DB, it returns a list of objects, which I can then retrieve (in bulk) from:

-> the in memory cache
  -> memcached
    -> the DB

No DB-based object contains another DB-based object, to avoid circular references. Instead, it just contains the ID of the object. Retrieving the actual object is cheap (assuming it has already been loaded) because we can just request the single instance of that object from the in-memory cache.

The in-memory cache is cleared at the end of each web-request.

The above is pretty similar to how KiokuDB works.

THE FUTURE AND BEYOND:

I’m currently working on an “ORM” that uses ElasticSearch as its backend. (“ORM” is in quotes because ES functions as a Lucene-powered document store, rather than being a relational DB).

I’d like to replicate the current functionality, because I think it has merits, but there is a complication:

Time doesn’t necessarily flow forwards

To explain:

ES has real-time GET. In other words, as soon as a document has been indexed (saved), it is available to be retrieved by it’s unique ID
When searching for documents, the full document is returned (by default), which means that you don’t have to do a second request to GET the document, but:
ES has NEAR-real-time SEARCH. Once a second (by default), the search view is refreshed to include changes that have occurred during the last second

What this means is that I could:


    GET doc 123        -> returns version 6
    SEARCH for doc 123 -> returns version 5

This would normally never happen in a traditional DB, because updates are atomic, and indexes are updated as the document is indexed. But it could happen in a master-slave setup where there is replication lag.

Also, I’m guessing this is a common scenario in NoSQL datastores.

Note:

This is an issue just for the current request, not for writes to ES. Every doc in ES has a _version number, and if you try to update the wrong version, it will throw a Conflict error, in which case you can:

get the latest version, reapply your changes and save, or
instruct ES to ignore the version and to update the doc regardless

So where might this be a problem:

Scenarios:


    $a = get     -> version 1
    $b = search  -> version 1

This one is easy. $b can just reuse the object in $a.


    $a = get     -> version 1
    $b = search  -> version 1
    $a->change()
    $a->save()   -> version 2

Potentially, the object no longer matches the search that you did, so you may be displaying incorrect results. (eg you search for name == ‘Joe’, then change name to ‘Bob’). But this looks like a reasonable process to me.


    $a = get     -> version 2
    $b = search  -> version 1

Our search has returned an older version of the object. The newer version might or not match the search parameters. Do we display the old results? or the new results?


    $a = get     -> version 1
    $a->change()
    $b = search  -> version 1

We have a changed (but as yet unsaved) object in the cache. Should $b contain the changed object, or the pristine object?


    $a = get     -> version 1
    $a->change()
    $b = search  -> version 2

We have an old (and changed) version in $a. We know that a newer version already exists in the DB, so we’ll get a conflict error if we try to save $a. What do we do?

Proposal:

I think my logic will look something like this:


    my ($class,$id,$version,$data) = @_;

    if (my $cached = $cache->{$id}) {

        return $cached
            if $version <= $cached->{version};

        return $cache->re_new($data);
            unless $cached->has_changed;

    }
    return $cache->{$id} = $class->new($data);

In other words, all instances of the object are always updated to the latest version, EXCEPT if the current instance has been edited and not yet saved. (Saving will throw a conflict error later on anyway).

Also, if you wanted to “detach” an object, then you could clone it and update it independently.

The only issue is that search results may contain a newer object which no longer matches the search parameters. Personally, I’m probably happy to live with this, but I probably need (a) a default setting and (b) a dynamic flag which the user can use to control this behaviour.

Thanks for getting to the bottom of this.

What do you think? See any obvious (or not-so-obvious) flaws?

(Also posted to PerlMonks )

ElasticSearch::Sequence - a blazing fast ticket server

2011-10-22T13:00:19Z

I'm considering ditching my RDBM for my next application and using ElasticSearch as my only data store.

My home-grown framework uses unique IDs for all objects, which currently come from a MySQL auto-increment column, and my framework expects the unique ID to be an integer.

ElasticSearch has its own unique auto-generated IDs, but:

they look like this 'KpSb_Jd_R56dH5Qx6TtxVA' and I'd say are less human-readable than an integer
I would need to change a fair bit of legacy code to migrate to non-integer IDs

Initially I thought I could keep MySQL around as a ticket server, as described by Flickr but then I wondered if I could achieve the same thing by abusing ElasticSearch's built-in versioning, allowing me to ditch MySQL completely, and give me a distributed ticket server with high availability into the bargain.

The logic is simple: when you index a document in ElasticSearch, it returns a new version number for the document, which is always incrementing and is guaranteed to be unique across the cluster.


# MAIL ID
curl -XPUT 'http://127.0.0.1:9200/sequence/sequence/mail_id?pretty=1' 

# {
#    "ok" : true,
#    "_index" : "sequence",
#    "_id" : "mail_id",
#    "_type" : "sequence",
#    "_version" : 1         # note: version number
# }

We can have multiple distinct sequences by storing a document with a different ID for each sequence.


curl -XPUT 'http://127.0.0.1:9200/sequence/sequence/other_id?pretty=1' 

# {
#    "ok" : true,
#    "_index" : "sequence",
#    "_id" : "other_id",    # note: different ID
#    "_type" : "sequence",
#    "_version" : 1
# }

ElasticSearch enables a bunch of features by default, which are very useful for using it as a document store and a full text search server, but aren't relevant in this situation, and will just slow it down.

The amount of data will tiny, so our index only needs one primary shard, not the 5 that are created by default in ElasticSearch. But for high-availability purposes, we'd like this shard to be replicated across all nodes in our cluster. So the index settings look like this:


   "settings" : {
      "number_of_shards" : 1,           
      "auto_expand_replicas" : "0-all"  
   },

For the type mapping (like a schema in a database) we want to turn off the _all field and _source field, disable _type indexing, and disable indexing for the document (which is only ever going to be an empty hashref):


   "sequence" : {
      "_source" : { "enabled" : 0 },
      "_all"    : { "enabled" : 0 },
      "_type"   : { "index" : "no" },
      "enabled" : 0
   }

So the full command to create the index and set the type mapping looks like this:


curl -XPUT 'http://127.0.0.1:9200/sequence/?pretty=1'  -d '
{
   "settings" : {
      "number_of_shards"     : 1,           
      "auto_expand_replicas" : "0-all"  
   },
   "mappings" : {
      "sequence" : {
         "_source" : { "enabled" : 0 },
         "_all"    : { "enabled" : 0 },
         "_type"   : { "index" : "no" },
         "enabled" : 0
      }
   }
}
'

Requesting a single ID (indexing the doc to get a new version) at a time is going to be relatively slow, as there is a fair bit of HTTP latency per request. This is fine for normal use, but our ticket server has to be super fast.

So instead, I'm going to request several new version numbers at once using the bulk API, and buffer them.


curl -XPOST 'http://127.0.0.1:9200/_bulk?pretty=1'  -d '
{"index":{"_index":"sequence","_type":"sequence","_id":"mail_id"}}
{}
{"index":{"_index":"sequence","_type":"sequence","_id":"mail_id"}}
{}
[*** SNIP ***]
'

# {
#    "items" : [
#       {
#          "index" : {
#             "ok" : true,
#             "_index" : "sequence",
#             "_id" : "mail_id",
#             "_type" : "sequence",
#             "_version" : 1
#          }
#       },
#       {
#          "index" : {
#             "ok" : true,
#             "_index" : "sequence",
#             "_id" : "mail_id",
#             "_type" : "sequence",
#             "_version" : 2
#          }
#       },
[*** SNIP ***]

ElasticSearchX::Sequence

I've wrapped up all of the above and released it as ElasticSearchX::Sequence


use ElasticSearch();
use ElasticSearchX::Sequence();
 
my $es  = ElasticSearch->new();
my $seq = ElasticSearchX::Sequence->new( es => $es );
 
$seq->bootstrap();   # setup the index and type mapping
 
my $it  = $seq->sequence('mail_id');
 
my $mail_id = $it->next;

Benchmarks

I wrote a small benchmark script which compares:

MySQL, using the ticket method described by Flickr
this module, using the httptiny backend
this module, using the curl backend
this module, using the curl backend but only requesting blocks of 10 IDs at a time

The results (run on my laptop) are pretty startling:


               Rate es_curl_10  db_ticket    es_tiny    es_curl
es_curl_10  38760/s         --       -48%       -55%       -72%
db_ticket   74627/s        93%         --       -13%       -47%
es_tiny     85470/s       121%        15%         --       -39%
es_curl    140845/s       263%        89%        65%         --

If you are already using ElasticSearch as your search server, (and if you're not, you should be - it's fantastic), and you're currently using your DB as a ticket server, I'd consider moving this function over to ElasticSearch instead.

Perlish concise query syntax for ElasticSearch

2011-07-03T21:43:08Z

Announcing ElasticSearch::SearchBuilder

In Perl, we like to put important things first, so the ElasticSearch query language has always felt a bit wrong to me. For instance, to find docs where the content field contains the text keywords:

    # op        field       value
    { text => { content => 'keywords' } }

To me, the important part of this is the field that we’re operating on, so this feels more natural:

    # field        op       value
    { content => { text => 'keywords' }}

So, in the spirit of SQL::Abstract I am proud to announce ElasticSearch::SearchBuilder, which is tightly integrated into the latest ElasticSearch.pm version 0.38.

Any method which takes a query or filter param (eg search() now also accepts a queryb or filterb parameter instead, whose value will be parsed via SearchBuilder:

Do a full text search of the _all field for 'my keywords':

    $es->search( queryb=> 'my keywords' );

Find docs whose title field contains the text apple but not orange, whose status field contains the value active:

$es->search(
    queryb => {
        title => {
            '='  => 'apple',
            '!=' => 'orange'
        },
        -filter => {
            status => 'active'
        }
    }
)

If you have suggestions to improve the API or the documentation, please get in touch.

You can try out ElasticSearch::SearchBuilder here.

And finally, a more complex example, to demonstrate how much more concisely you can write queries:

Out of all docs published in 2010 and tagged with either “perl” or “ruby”, find those whose title field contains”my keywords”, in which case consider this doc to be particularly relevant (boost: 2) or the title field is missing but the body field contains 'my keywords':

$es->search(
    queryb => {
        -or => [
            {
                title => {
                    '=' => {
                        query => 'my keywords',
                        boost => 2
                }}
            },
            {
                body    => 'my_keywords',
                -filter => {
                    -missing => 'title'
                }
            },
        ],
        -filter => {
            tags => [ 'perl','ruby' ],
            date => {
                '>=' => '2010-01-01',
                '<'  => '2011-01-01'
            },
        }
    }
)

is the equivalent of:

    $es->search(
        query => {
            filtered => {
                filter => {
                    and => [
                        { 
                            terms => { 
                                tags => ["perl", "ruby"] 
                            } 
                        },
                        { 
                            numericrange => { 
                                date => { 
                                    gte => "2010-01-01", 
                                    lt => "2011-01-01" 
                                }
                            }
                        }
                    ],
                },
                query  => {
                    bool => {
                        should => [
                            { 
                                text => { 
                                    title => { 
                                        boost => 2, 
                                        query => "my keywords" 
                                    } 
                                } 
                            },
                            { 
                                filtered => {
                                    filter => { 
                                        missing => { 
                                            field => "title" 
                                        } 
                                    },
                                    query  => { 
                                        text => { 
                                            body => "mykeywords" 
                                        } 
                                    },
                                }
                            }
                        ],
                    }
                }
            }
        }
    )

Which looks better to you?

ElasticSearch.pm v0.37 released, with a small breaking change

2011-05-01T12:14:04Z

Just released ElasticSearch.pm v 0.37 which has a small breaking change.

In version 0.36, $scrolled_search->next() returned the next $size results. Now, by default it returns the next one result, which makes it easier to write:



     while ( my $result = $scroller->next ) {...}

ElasticSearch.pm v0.36, now with extra sugar

2011-04-24T19:16:16Z

ElasticSearch v 0.16.0 was released yesterday with a long list of new features, enhancements and bug fixes.

ElasticSearch.pm v 0.36 is on its way to CPAN as we speak.

Besides adding support for the new stuff in v 0.16, I've also added a few features:

scrolled_search()

It is possible to scroll through a long list of results in ElasticSearch, but this required a bit of repetitive code, which is now nicely packaged up in scrolled_search. So you can do:



    $scroll = $es->scrolled_search( 
        search_type => 'scan',   # efficient search type for scrolling
        scroll => '2m', # cache search results for the next 2 minutes
    );

    while (my $doc = $scroll->next(1)) {
         # do something
    }

reindex()

Users on the mailing list are always asking how to reindex their data, either from one index to another on the same cluster, or from one cluster to another.

Now, scrolled_search() and reindex() make it easy to do this in a single command.

For example:

To copy the ElasticSearch website index locally, you could do:



    my $local = ElasticSearch->new(
        servers => 'localhost:9200'
    );
    my $remote = ElasticSearch->new(
        servers    => 'search.elasticsearch.org:80',
        no_refresh => 1
    );

    my $source = $remote->scrolled_search(
        search_type => 'scan',
        scroll      => '5m'
    );
    $local->reindex(source=>$source);

To copy one local index to another, make the title upper case,
exclude docs of type boring, and to preserve the version numbers
from the original index:



    my $source = $es->scrolled_search(
        index       => 'old_index',
        search_type => 'scan',
        scroll      => '5m',
        version     => 1
    );

    $es->reindex(
        source      => $source,
        dest_index  => 'new_index',
        transform   => sub {
            my $doc = shift;
            return if $doc->{_type} eq 'boring';
            $doc->{_source}{title} = uc( $doc->{_source}{title} );
            return $doc;
        }
    );

no_refresh

By default, ElasticSearch.pm retrieves a list of live nodes from the ElasticSearch cluster, and round-robins around them.

However, if you are talking to a remote ES cluster, or a cluster behind a proxy, this may not be desirable behaviour. The no_refresh parameter turns off the discovery of live nodes. Instead ES.pm round robins through the list of servers passed to new(), and can fail over between this list:



    my $es = ElasticSearch->new(
        servers => ['es1.search.com:80', 'es2.search.com:80'],
        no_refresh => 1
    );

Firefox 4 bug adding inline script tag - last tag doesn't fire

2011-03-25T19:02:45Z

I came across a nasty bug in Firefox 4 today, which will break a lot of AJAX.

All script tags fire, except for the last one, eg:





function test(num) {
    var temp = document.createElement('div');
    var src  = '