This post is written mainly as a personal note to remind a future me.

So first: Just a I thought I am done with the Rinci transaction protocol and framework, another idea popped up in my head and pulled me back in. I have to admit, the v1 protocol and the framework are too complicated. Ever since the realization that the undo protocol is not safe for use with transaction, and thus another protocol is needed, I ended up with the formalization of the steps concept, which of course broke down the minute things get a bit more complex, and also with passing the transaction manager object to functions and requiring them to invoke methods on it directly. Don't know how I came up with all that mess, but thankfully it's been simplified now. Gone is the concept of steps, everything is function again (this I'm very happy about). And also gone is the undo protocol (deprecated to be exact); undo is now implemented through the transaction protocol, instead of the other way around. Dry-run is also now somewhat based on transaction, for transactional functions. But due to all of these, I had to spend a good chunk of August rewriting the specification and discarding the framework and rewriting all the released modules containing transactional functions. Hopefully I won't have to do another major rewrite of the specification, unless I can simplify it again significantly. You know, the famous Einstein quote: "Everything should be made as simple as possible, but not simpler."

Transactional function

So what is a transactional function (in the Rinci sense)? It is a function to which we can say, "If I ask you to do this later, what will be needed to undo it?" In other words, the function always knows how to undo its own action, before performing the action. If it does not know how to do that, or know that an action cannot be undone, it will refuse to do the action.

The asking beforehand is called the check_state phase, and the actual performing of action is called the fix_state phase. During a transaction, we call one or several transactional functions (and the functions can also "call other functions", we'll discuss this later) through the transaction manager (TM). For each function, TM will call the function twice. First by passing along a special argument -tx_action => 'check_state' to ask for the undo actions from the function. After recording the undo actions to the journal, TM will call the function again, this time with -tx_action => 'fix_state' to actually perform the action. If the second call fails or crashes, we can rollback using the undo actions already recorded in the journal. The specification describes the process in all its gory details, so you can read it first if you want more information.

Basic structure

So the basic structure of the function is as follow:

sub txfunc {
    my %args = @_;
    my $tx_action = $args{-tx_action} // "";
    if ($tx_action eq 'check_state') {
        ...
        return ...;
    } elsif ($tx_action eq 'fix_state') {
        ...
        return ...;
    }
    [400, "Invalid -tx_action"];
}

The check_state phase

Now what should we do in the check_state and fix_state phase? Let's use an actual example first: trashing file.

In the check_state phase, we should check whether the state of things is already to our liking. In other words, whether we need to fix_state at all. By the way, remember that all transactional functions must also be idempotent. So we cannot make a transactional function for, say, appending a text to a file, because that action is not idempotentable (well, actually we can make it idempotent by checking whether the string to be appended is already at the end of the file and if so refusing to append the same thing twice in a row, but that is not exactly called a pure append). So anyway, our trash() function should check whether the file is already deleted (note that I simplify things in this example). If so, this is what we call the (already) fixed state.

    ...
    my $path = $args{path};
    defined($path) or return [400, "Please specify path"];

    if ($tx_action eq 'check_state') {

        return [304, "File $path is already deleted"] unless -e $path;

        ...

    }

Notice that we return 304 status along with a descriptive message saying how the state is already fixed ("the file so and so is already deleted", "the database so and so is already created").

If the file still exists, this is called a fixable state. We should return 200 status with a descriptive message saying how the state can be fixed:

    return [200, "File $path needs to be deleted", undef, 
        {undo_actions=>[
            [untrash=>{path=>$path}],
        ]}];

BTW, I settled on the "X needs to be Y" sentence pattern. TM will log these messages at the debug level. You should make the message pretty informative, not too technical, and consistent. If you want additional log messages that are technical, like some data dump, you should log those using the trace level.

Note the undo_actions. It is a list of actions, where each action is a pair or function name and arguments hash.

Also note that there is another possible state called the unfixable state, where we know that things cannot be fixed or done by the function. In this example I choose not to regard any such state, but we could have. For example, if we check that $path will not be unlinkable (e.g. it is on a read-only filesystem, or the directory is owned by other user and we are not the superuser, etc), we can return:

    [return 412, "File $path is not unlinkable"] if !$unlinkable;

Other examples of unfixable state: an append() function to append string to a file encountering $path to be a directory. A create_user() function instructed to create a user with a specific UID, when the user to be created already exists but with a different UID.

The fix_state phase

For our example:

    ...
    } elsif ($tx_action eq 'fix_state') {
        $log->infof("Trashing %s ...", $path);
        eval { $trash->trash($path) };
        if ($@) {
            return [500, "Can't trash: $@"];
        } else {
            return [200, "OK"];
        }
    }

Note that I added a logging message with the sentence pattern "X-ing Y ..." at the info level. User can see this when they run the program using --verbose to see what actions are being performed. In the error message, I no longer choose to include $path as it is already logged before.

If you need to return result, for example the path to the file in the trash, simply put it in the result envelope:

...
        my $tfile;
        eval { $tfile = $trash->trash($path) };
        if ($@) {
            return [500, "Can't trash: $@"];
        } else {
            return [200, "OK", $tfile];
        }
...

Calling other transactional functions (a.k.a. merging check_state and fix_state, a.k.a. do_actions)

So far so good. But what if our function/action grows more complex and needs to call/be composed of other simpler functions? We should not call other transactional functions directly (unless we want to replace the TM's job of calling to check state, recording undo actions into journal, and performing rollback/roll forward/recovery). What we can do is return do_actions in the check_state phase. Here's a simplified example from Setup::Unix::User:

sub setup_unix_user {
    ...
    my (@do, @undo);
    if (!$user_exists) {
        push @do, [adduser=>{user=>$user}];
        unshift @undo, [deluser=>{user=>$user}];
    }
    if (!$user_exists || !(-d $home)) {
        push @do, 
            [mkdir=>{path=>$home}], 
            [chmod=>{path=>$home, mode=>0700}],
            [chown=>{path=>$home, owner=>$user, group=>$user];
        unshift @undo,
            [trash=>{path=>$home}];
    }
    
    if (@do) {
        return [200, "", undef, {do_actions=>\@do, undo_actions=>\@undo}];
    } else {
        return [304, "User $user has been created"];
    }

If you notice the above, the function doesn't bother to check $tx_action, as it does not do any state fixing by itself. What it does is just stating that to fix the state, please do the actions specified in do_actions. When the TM sees this, it will skip calling the function with -tx_action=>'fix_state' but instead will call the functions in do_actions (each with their own check_state and fix_state phases, and the undo actions recorded are from those functions too).

You'll also notice that we return empty envelope message. This is because the check_state message(s) will also be provided by the functions in do_actions.

Dry-run message

One last thing. Since in check_state we check state but do not actually do anything, we might as well implement dry-run using it. We can do something like:

    ...
    my $dry_run = $args{-dry_run};
    if ($tx_action eq 'check_state') {
        return [304, "File $path already deleted"] if $exists;
        $log->infof("(DRY) Trashing %s ...", $path) if $dry_run;
        return [200, "File $path needs to be deleted", undef, 
        {undo_actions=>[
            [untrash=>{path=>$path}],
        ]}];
    }
    ...

Note the $log->infof() statement. If user activates dry-run mode using --dry-run, verbose (level info) mode is also activated and user will see something like this in the output:

% trash-u a b c --dry-run
(DRY) Trashing a ...
(DRY) Trashing b ...
(DRY) Trashing c ...
%

Dry-run mode is implemented by calling the transactional function only for the check_state phase and skipping the fix_state phase.

I also settled on using the log prefix (DRY) in the log message to differentiate it from other messages, like the one in the fix_state:

% trash-u a b c --verbose
Trashing a ...
Trashing b ...
Trashing c ...
%

The messages above are from the fix_state phase.

So what's all this for?

You might wonder about the use of transactional function and Rinci transaction. Well, my main use case for now is software installation and uninstallation. If installation or upgrade fails in the middle, it can be rolled back. I guess most installers in Windows (still) have this capability. It has been abandoned in RPM (and has never been supported by dpkg?) due to buggy (or perhaps non-transaction-caring) shell scripts.

But anyway, anywhere you need undo/redo facility, the transaction protocol can be used.