Of course I still use Perl

Unit testing and continuous integration testing are ways to make sure that every change to our code won't break stuffs. We execute our tests after each commit and when we want to release our software.

On the other side of the spectrum, there are things outside our code that change and break our non-changing code. Platforms (like new versions of Perl) and dependencies, for example. Integration testing and CPAN Testers are some of the ways to make sure that dependencies and different versions of Perl still work with our software. We must do this testing regularly even when we're not changing our code, since the changing part is the other party.

There is a specific class of scenarios along this end of the spectrum when we're developing scraping libraries and API clients: source websites and API services change all the time, breaking our otherwise working code. For lack of a better term, I'm going to call this third party testing.

To do this for my Perl distribution, I plan to put the tests in xt-3rd/ (to differentiate it with xt/ which are mostly author tests and related to release testing). There will also be an environment flag variable THIRD_PARTY_TESTING, analogous to RELEASE_TESTING or AUTOMATED_TESTING. The tests in xt-3rd/ should only run when THIRD_PARTY_TESTING is true.

We can run third party testing regularly, say once or twice a day. The results can be collected and tabularized much like CPAN Testers.

Web scraping or API access can fail in two general cases: either the website/service changes their format/interface, or there is a network problem (the website/service blacklisting us is categorized as the latter). In the former case, we want to find out right away. In the latter case, we'll allow for some failures, say for a week, to pass. If the failures persist, then there is probably something wrong and the system can notify the developer. We want to differentiate these two cases to ignore false positives. This is where the testing report service differs with CPAN Testers: CPAN Testers reports every failure to the developer, but we want only the failures of the first case. The second case failures should only be reported if they persist after a week.

Unfortunately, prove only exits with exit code of either 0 or 1, so we'll need to do some parsing of its output. For example, if we detect a scraping/parsing failure or an API interface changes, our script can exit with another non-zero exit code (e.g. 2, or 99).