3Party - cpan modules third party testing
Last time I have been deeply involved into test automation development. So some projects was born like swat, outthentic and finally sparrowhub.
After some mediation on the future of my projects I am still committed to the idea of sharing such a tests with the rest of perl/cpan community.
An issues arise on my way though:
people used to stick to convenient unit tests kept under their cpan distributions and are half-heartedly for third party tests alternative which of course is respected by me.
(IMHO) integration tests are hard to write and maintain, this is probably why people prefer to write unit tests, and as swat is kind of integration test tool, it could be "neglected" for the reason above. I believe still that integration tests are essential part of development.
Taking into account all of this I am introducing an idea of third party tests service - 3Party ( working name, just can't invent something better right now ) with the following features:
tests are written on swat or outthentic frameworks
cpan modules to test are randomly chosen and tests are provided
if one want to force your cpan module to get tested via 3Party - welcome! we could discuss it
a cpan modules author receive test reports via email or through 3Party web interface
tests triggered at every cpan module release so they could be considered as regression base
no warranty for full coverage of cpan module API, but test base could be increased eventually as far as tests contributors have a free time on it ( or new tests contributors join the team )
now tests contributors team is me only but new developers are welcome
tests relies on cpan module public API provided by it's documentation, in case false test issue a cpan module author is encouraged to help test author to correct tests behavior, but it is only guaranteed that test author will follow cpan module documentation and try to implement things based on his understanding of such a documentation
This only an informal draft of the idea.
I need a community feedback before to dive into this project.
Thanks
Alexey Melezhik
Dev & Devops
---
Then I beheld all the work of God, that a man cannot find out the work that is done under the sun: because though a man labour to seek it out, yet he shall not find it; yea further; though a wise man think to know it, yet shall he not be able to find it.
(Ecclesiastes 8:17)
Parts of this sound very similar to the CPAN Testers project. What makes this different? Could it be added to CPAN Testers?
Hi!
CPAN Testers is a great service undoubtedly. But the idea behind 3Party is _integration_ testing. CPAN tests are mostly unit tests, which has pros and cons. Consider testing some code requiring complex integration environment which is not easy to achieve during unit test run. Another difference is WHO writes a tests. Cpan module unit tests are usually written by module author and often such a tests are coupled with some internal stuff ( like objects, functions, methods ) which is ok when we talk about unit testing. Integration testing provided by 3Party is intended to be external black box testing and probably done by people not involved in cpan module development process, such an approach obviously has some advantages in way of testing.
Another possible interesting feature of 3Party is specification by example or BDD development where 3Party tests could be in some sense a cpan module API examples, they are:
* simple
* illustrative
* based on end user point of view
* ( integration testing is of much importance for web applications, where unit tests often are not enough )
* black box and are not coupled with internal module structure
Consider swat documentation for the implementation of this idea.
In contrast with unit tests which are mostly for developer and tends to be complicated and hard to read by unprepared user.
Regards