Writing tests by non-programmers

I constantly see requests from companies to allow non-programmers to write automated tests. I also see products that allow this on various levels of sophistication. Every such tool also requires some programming in a high level language. Sometimes that is VB Script or .NET or Java. It is rarely Perl.

By the time the customer understand that they need a lot of extra programming they have already invested a lot of money and time so they won't switch to another solution even if that saved them a lot of money and time.

So I though I might try to reinvent this wheel. The idea would be to create a GUI where a user can select "tools" (e.g. "web browser", "telnet client", "database client") and add these tools to the test script by clicking on a button. Once a "browser" was added the user can do various things with that browser: fetch a url, check if
content has certain string in it, click on a link defined by the text on the link etc.

Behind the scene these steps are saved in a json file and when the user clicks on "run" the json file gets translated to perl code and gets executed. A json file would look something like this:


{
"name": "Simple web test",
"format" : 0.01,
"steps": [
{
"id" : 1,
"action" : "new",
"tool" : "Mechanize",
"name" : "User foo"
},
{
"id" : 1,
"action" : "get_ok",
"params" : {
"url" : "http://www.google.com/"
}
},
{
"id" : 1,
"action" : "content_like",
"params" : {
"regex": "Google"
}
}
]
}

while the generated perl code something like this:


use strict;
use warnings;

use Test::More;
plan tests => 2;
use Test::WWW::Mechanize;
my %tools;
$tools{1} = Test::WWW::Mechanize->new();
$tools{1}->get_ok('http://www.google.com/');
$tools{1}->content_like(qr{Google});

Once people need higher level abstractions that have steps such as
"login to application" That would require some programming in order
implement the new tool and to integrate it into the GUI which is the
same with the other products except that one could write this in Perl.

To me this whole thing looks very similar to Fit though I think currently this style of testing is called "Keyword Driven Testing" or KDT.

How bad is this idea?

Are there similar solutions on CPAN already?

What other route would you suggest for this?

5 Comments

This sort of thing has indeed already been done, and its implementation is rather good! Take a look at Selenium, particularly the Selenium IDE browser plugin.


With it, a non-technical user can essentially "record" their interaction with the site, including content and some DOM checks. They can then save the resulting 'script' as an HTML document with a very simple table structure that can be read by the plugin, or it can output code that would drive the Selenium RC server in several languages including perl, python and java.


Last year I wrote a fairly crude but effective testing tool that essentially does what you described above. After giving a lightning talk at YAPC::NA showing it off (including a successful live demo!) Schwern actually told me about Selenium. I tried it and have been in love with it ever since!

How about a perl version of Cucumber? While it's mostly used for web testing, it's by no means limited to that -- and I'd _kill_ for a perl version....

Wow, I haven't logged into this site in a while, but I noticed a list of previous comments I've made so I checked it out. I'll have to see if there's a feature to email me when someone responds to something I've commented on...


Anyhow... No, the code isn't online anywhere. I think I may have tossed it into the bit-bucket when I found Selenium did everything I needed and did it better. HOWEVER - you make a very good point! Selenium is a heavy-weight, comprehensive solution that is really applies when you have (and need to test) the full stack. There are certainly times when lighter-weight, and more flexible things are a much better tool for the job.


Like your description above, what I wrote was data-driven, and I even hacked together a means of recording tests through a browser...


I will see if I have the code somewhere on my hard drive, or on my backup disk and at least get it on github!

Leave a comment

About Gábor Szabó - גאבור סבו

user-pic I blog about Perl.