A major milestone in WebGUI 8 development was reached this week: A dry-run of the WebGUI 8 upgrade was successfully run against the plainblack.com database. This means the only thing remaining from releasing an alpha 8.0.0 is updating all the custom code on http://plainblack.com and http://webgui.org. As always, plainblack.com and webgui.org will be the first sites running the latest bleeding-edge version of WebGUI (unless one of you wants to beat me to the punch).

This month, I also gave a presentation to Madison.PM about building applications in WebGUI 8, a quick introduction to Assets and an overview of the most important changes to how they work. The slides are available at http://preaction.github.com/ and the code samples are linked at the end.

On an unrelated topic, I really enjoyed using S5 to build my slides, SHJS to highlight the code inside, and Github Pages to host the whole thing. I plan on doing the same for all my presentations: They look good, readable without a special program, editable without a special program, anyone can fork and update my presentations, and they're served by a nice, fast, free host.

By far the biggest change we've made in WebGUI 8 is the new Admin Console. Though parts of it may look familiar, it has been completely rewritten from the ground up to be a flexible, extensible, responsive JavaScript application making calls to JSON services in Perl.

I could talk about how to use the admin interface, but I don't think that's why you would read this blog, so instead I'm going to talk about how you can add functionality to it.

Asset Services

Since Assets are the basic unit of both application and content in WebGUI, much of the Admin Console is spent interacting with Assets. It does so by calling out to Asset Helpers.

By default, every asset has a helper to Cut, Copy, Duplicate, Delete, and more. When a helper gets called, it returns a JSON data structure explaining to the Admin Console what to do next.

We can simply show the user a message:

message     => 'The work is done, here's what happened.'
error       => 'Something went wrong.'

Or we can open up new dialogs or tabs to allow the user to give us more data:

openDialog  => '/helper/get_input'
openTab     => '/helper/get_input'

We can let the user know their command is running in a forked process:

forkId      => '...' # GUID for WebGUI::Fork object

We can even load and run any external JS file:

scriptFile  => '/extras/newscript.js',  # Load a new script file
scriptFunc  => 'myFunction',            # Call a function in that script
scriptArgs  => [ "arg1", "arg2", ],     # Pass some arguments to that func

To write an Asset Helper, we inherit from WebGUI::AssetHelper and override the process() method to send back one of the message types from above.

package MyHelper;
use base 'WebGUI::AssetHelper';

sub process {
    my ( $self ) = @_;

    return { error => 'Cry Havoc!' } if !$self->asset->canEdit;

    # Do some work

    return { message => 'Work is done!' };
}

If our Asset Helper needs to get some input from the user, we can open a dialog. Like most everything in WebGUI, Asset Helpers can also have www_ methods.

package MyFormHelper;
use base 'WebGUI::AssetHelper';

sub process {
    my ( $self ) = @_;
    my $url = $self->getUrl( "showForm" );
    return { openDialog => $url };
}

sub www_showForm {
    my ( $self ) = @_;
    my $form    = $self->getForm( 'processForm' ); # WebGUI::FormBuilder
    $form->addField( "text", name => 'why' );
    return $form->toHtml;
}

sub www_processForm {
    my ( $self ) = @_;
    my $input = $self->session->form->get( 'why' ); # input from the form
    return { message => $input }; # Why not?
}

But our asset helpers are not only useful inside of the Admin Console. Because they're all built on a simple JSON API, you can call them from anywhere. For example, the Asset Helper to resize and rotate images could be used by anyone with edit privileges to the Image.

Because we already have these Asset Helpers, the new Asset Manager (now called the Tree view) uses them to perform all of its tasks. This means, again, more code reuse and less code in WebGUI.

Side note: I love deleting code much more than writing it.

Adding Helpers

What would a plugin point be without a way to override what already exists? In our case, if you want another helper to handle the "cut" operation, you can make it happen.

If you have your own asset, you can override the getHelpers method, which returns a hashref of helper descriptions:

package MyAsset;

around getHelpers => sub {
    my ( $orig, $self ) = @_;
    my $helpers = $self->$orig;
    $helpers->{ "cut" } = {
        className   => 'MyCutHelper',
        label       => 'SuperCuts',
    };
    return $helpers;
};

Or if you don't want to edit the asset's code, you could add your helpers to the configuration file:

{
    "assets" : {
        "WebGUI::Asset::Snippet" : {
            "helpers" : {
                "cut" : {
                    "className" : "MyCutHelper",
                    "label"     : "SuperCuts"
                }
            }
        }
    }
}

Side Note: Deep data-structure is deep.

A Helper doesn't have to be its own class, it could be any URL at all:

$helpers->{ "edit" } = {
    url     => './edit',
    label   => 'Edit',
};

So Asset Helpers are the new way to add related tasks to your assets. Come back next time when I introduce WebGUI::FormBuilder.

Caching is a tricky business. Having just one kind of cache won't work, because the production environment will greatly determine the most efficient caching system. A distributed production environment would be best-served with a distributed cache. A smaller, single-server environment could use a simple shared memory cache.

Enter Jonathan Swartz's CHI module, the greatest Perl module to provide a unified caching interface. CHI is the DBI of caching: It presents an API, and delegates to CHI::Driver modules to perform the heavy lifting. It provides a layered caching system, allowing you to have a faster, more volatile cache in front of a slower, more persistent cache. It also provides a variable expiration time, preventing a "miss stampede" where all processes try to recompute an expired cache item at the same time.

By integrating CHI cache into WebGUI, we have the ability to provide any caching strategy that CHI can provide. We get Memcached, FastMmap, and DBI drivers (and more drivers can be written).

I wrote a CHI cache driver for WebGUI 7.9 that we've been using on many of our shared hosting servers. The performance increase using FastMmap through CHI over the old Storable+DBI cache module is dramatic: 2-5 times faster with CHI and FastMmap.

Using CHI in WebGUI

The fewer wrappers that WebGUI has around CPAN modules we use, the less code I have to write, and the more features will be available to our users without having to change WebGUI to use them.

To that end, you can write a section of the configuration file that gets passed directly to CHI->new. Some massaging occurs to make sure a DBI cache driver gets the right $dbh, but otherwise you can fully configure CHI directly from the WebGUI config file:

# The new default cache for WebGUI, FastMmap
{
     cache : {
         driver : 'FastMmap',
         root_dir : '/tmp/WebGUICache',
         expires_variance : 0.5
     }
 }

 # Set up a memcached cache with local memory in front
 {
     cache : {
         driver : 'Memcached::libmemcached',
         servers : [ '10.0.0.100:11211', '10.0.0.110:11211' ],
         l1_cache : {
            driver : 'Memory'
         }
     }
 }

When you want to use the cache in your code, you can get a CHI object with $session->cache. CHI's interface is sufficiently simple, with some fun tricks:

my $cache = $session->cache; # as read
my $value = $cache->get('cache_key');
if ( !$value ) {
    $value = compute_value();
    $cache->set( 'cache_key', $value );
}

# Combine get and set with intelligence
my $value = $cache->compute( 'cache_key', \&compute_value );

Future Plans

With a single unified cache that performs well and layers like CHI, we can take our current stow and scratch APIs and move them to the cache. In the case of stow, we remove a redundant API. In the case of scratch, we remove database hits.

We've also been exploring cache-only sessions, instead of updating the session every time a page is requested, updating the cache only, flushing to the database (or not). The fewer DB calls we make per page, the better performance will be.

Special thanks go out to Jonathan Swartz for such a wonderful solution.

Stay tuned for next time when I explore our new Admin Interface. Lots of pretty and screenshots!

Following The Path

If you installed WebGUI 0.9.0 back in August of 2001 (the first public release), you've had a stable upgrade path through WebGUI 7.10.8 (January 2011) and beyond. Plainblack.com has been through every upgrade for the last 10 years, a shining bastion to our upgradability.

A WebGUI 7.10 user would not even recognize a WebGUI 6.0 database, much less the database used by the 1.x series, but slowly, gradually, our upgrade system brought new features to every WebGUI site that wanted them.

The Ancient Way

Our old upgrade system was quite simple:

docs/upgrade_2.9.0-3.0.0.pl
docs/upgrade_3.0.0-3.0.1.sql
docs/upgrade_3.0.0-3.0.1.pl

Our upgrade.pl script would check for docs/upgrade_*, compare version numbers, and then execute the .sql and .pl scripts in order until there were no more upgrades left.

Because each .pl script was executed individually, there was a considerable amount of boilerplate in each script (123 lines). Because there was only one script per version, some scripts could get quite long. We had conventions to manage these limitations, but it was still a bit of a mind-twist to write an upgrade routine.

Later, when we moved to simultaneous beta and stable trees, it became even more difficult to manage these huge upgrade scripts. Collecting the new features from the beta tree to apply to the stable tree was a time-consuming manual task that some poor coder had to perform, back hunched over a dimly-lit screen in the wee hours of the night, testing and re-testing the upgrade to make sure stable lived up to its expectations.

Though our upgrade system had performed admirably, it was time for a fresh look at the problem.

The Modern Vision

The individual files for upgrades was working quite well, but didn't go far enough. Our new upgrade system has one file per upgrade step. Each sub from an old upgrade script would be one file in the new upgrade system. What's more, additional file types would be supported:

$ ls share/upgrades/7.10.4-8.0.0/
addNewAdminConsole.pl
admin_console.wgpkg
facebook_auth.sql
migrateToNewCache.pl
moveMaintenance.pl
moveRequiredProfileFields.pl

So now, instead of a single file for an upgrade, we have an entire directory. In this directory, the .pl files are scripts to be run, the .wgpkg files are WebGUI assets to add to the site, the .sql files are SQL commands to run, and any .txt files will be shown as a confirmation message to the user for gotchas like "All your users have been logged out as a result of this upgrade. Deal with it.".

So now, if you want to add your own custom upgrade routine, you just add another file to the directory which means less worrying about conflicts. When we need to build another new stable version release, we can just move the unique upgrade files from beta to the new upgrade.

The best part of the new upgrade system is how the .pl scripts are written. When you are in a .pl, you have a bunch of sugar to make the basic tasks much easier.

# Old upgrade routine. Just another day in a session
sub migrateToNewCache {
    my $session = shift;
    print "\tMigrating to new cache " unless $quiet;

    use File::Path;
    rmtree "../../lib/WebGUI/Cache";
    unlink "../../lib/WebGUI/Workflow/Activity/CleanDatabaseCache.pm";
    unlink "../../lib/WebGUI/Workflow/Activity/CleanFileCache.pm";

    my $config = $session->config;
    $config->set("cache", {
        driver              => 'FastMmap',
        expires_variance   => '0.10',
        root_dir            => '/tmp/WebGUICache',
    });

    $config->set("hotSessionFlushToDb", 600);
    $config->delete("disableCache");
    $config->delete("cacheType");
    $config->delete("fileCacheRoot");
    $config->deleteFromArray("workflowActivities/None", "WebGUI::Workflow::Activity::CleanDatabaseCache");
    $config->deleteFromArray("workflowActivities/None", "WebGUI::Workflow::Activity::CleanFileCache");

    my $db = $session->db;
    $db->write("drop table cache");
    $db->write("delete from WorkflowActivity where className in ('WebGUI::Workflow::Activity::CleanDatabaseCache','WebGUI::Workflow::Activity::CleanFileCache')");
    $db->write("delete from WorkflowActivityData where activityId in  ('pbwfactivity0000000002','pbwfactivity0000000022')");

    print "DONE!\n" unless $quiet;
}

If you're familiar with WebGUI session, this is pretty standard, but still much boilerplate and convention. The new scripts remove boilerplate and enforce what was once merely convention.

# New upgrade routine. migrateToNewCache.pl
use WebGUI::Upgrade::Script;
use Module::Find;

start_step "Migrating to new cache";

rm_lib
    findallmod('WebGUI::Cache'),
    'WebGUI::Workflow::Activity::CleanDatabaseCache',
    'WebGUI::Workflow::Activity::CleanFileCache',
;

config->set("cache", {
    'driver'            => 'FastMmap',
    'expires_variance'  => '0.10',
    'root_dir'          => '/tmp/WebGUICache',
});

config->set('hotSessionFlushToDb', 600);
config->delete('disableCache');
config->delete('cacheType');
config->delete('fileCacheRoot');
config->deleteFromArray('workflowActivities/None', 'WebGUI::Workflow::Activity::CleanDatabaseCache');
config->deleteFromArray('workflowActivities/None', 'WebGUI::Workflow::Activity::CleanFileCache');

sql 'DROP TABLE IF EXISTS cache';
sql 'DELETE FROM WorkflowActivity WHERE className in (?,?)',
    'WebGUI::Workflow::Activity::CleanDatabaseCache',
    'WebGUI::Workflow::Activity::CleanFileCache',
;
sql 'DELETE FROM WorkflowActivityData WHERE activityId IN (?,?)',
    'pbwfactivity0000000002',
    'pbwfactivity0000000022',
;

done;

The first thing we do in our new upgrade script is use WebGUI::Upgrade::Script. Now, instead of using the session for everything, we have subs imported for various tasks. This means that many times we can run an entire upgrade script without opening a WebGUI session, or creating a version tag unnecessarily.

If we do need a session, or a version tag, they will be automatically assigned relevant information describing what we're doing. When we're done, they will be automatically cleaned up and committed. What once was done with boilerplate, and subject to random deletion or subversion, is now enforced policy.

In all other respects, a WebGUI upgrade script is a Perl script. You can add modules, write subroutines, and do anything necessary to move WebGUI into the future.

The Internet is always evolving. With the WebGUI 8 upgrade system, we've made it easier to evolve with it.

Stay tuned for next time where I'll show off our CHI-based caching system.

Auth changes

• Auth predates Facebook, which was founded in 2004.

• Since Auth, there have been two Summer Olympiads.

• Auth was written when I was still in college.

Since then, it has not fundamentally changed, though everything about the Internet surely has.

We began our planning for 8.0 with the idea to completely rebuild Auth from scratch, but that quickly got scrapped when we realized both the scope of th…