Hi,
In my team we are using Simpletests extensively both on local workstations and on our continuous integration server.
Our workflow resembles something like this:
* before committing code to the git repo, developers run all tests pertaining the test group "my-website" and make sure they pass. This is often done in the GUI, as the GUI provides nicely themed green and red test report with links to verbose messages.
* when a change in git is detected by the continuous integration, tests are run on the command line either with drush test-run my-website or php scripts/run-tests.sh.
I modified Drush slightly as documented here so that, when the tests are run on the command line, the themed test reports and verbose messages (captured html code) can be stored in a given directory as a build artifact. This is very useful to quickly figure out what is going wrong with a particular build.
Moshe Weitzman suggested in this issue that it might be worth considering the possibility of adding this functionality to php scripts/run-tests.sh.
Would there be interest in including this functionality to Drupal, as an option for php scripts/run-tests.sh and as a checkbox on the test GUI page? When the option is selected, the themed test results and corresponding verbose messages would be saved to a given directory, for example, public://simpletest/artifacts/ or public://simpletest/artifacts/1234 (the ID of the test).
Other options would be adding this functionality to drush_extras or to a dedicated module.
Comments
Comment #1
sunAdding 4 very closely related issues.
I've thought a lot about this in my recent work, and I agree that we should clean up and improve the current situation of test results/artifacts.
We already have an
--xmlparameter, which produces JUnit XML files in the specified path, containing test results.But even without the
--xmlparameter, if you run PHPUnit tests via run-tests.sh, then PHPUnit gets configured to create JUnit XML test result files inpublic://simpletest/phpunit-$testId.xmlIf you pass the
--verboseflag, then verbose test browser results will be written topublic://simpletest/verbose/$classname.$requestId.htmlLastly, the new
--sqliteparameter in D8 creates a new SQLite database using the specified pathname, in order to execute the test runner without a pre-installed Drupal parent site.This quick wrap-up clarifies that the test runner currently writes into half a dozen of different locations. That's not helpful in any way.
As also outlined more in-depth in #2248845: [run-tests.sh] simpletest_phpunit_xml_filepath() depends on public:// stream wrapper:
All of this pretty much cries to be consolidated into a single new
--outparameter, which would be used for (1) the SQLite test runner db, (2) XML/TAP/whatever test results, (3) verbose browser output, etc.pp.If omitted, we could try to use the global temporary directory filesystem path of the OS - possibly using a unique subdirectory that is a shortened
hash(__DIR__)of the document root directory of the Drupal installation; e.g.,/tmp/drupal-test-a1b2c3d4/However, if you'd pass the
--outparameter, then you'd get all of the results + artifacts in a predictable/desired location, which you can use to perform additional operations.For example, when passing
--out build --xml --verbose, then I'd imagine/expect to get:Thoughts?
Comment #2
alberto56 commented@sun thanks for the extra details, and the larger perspective on this. I would like to see this made extensible, so that contirb modules (or add-ons to the test script, would they be modules if a host installation is no longer required?) could also access the "output" directory.
For example, I like to take screenshots of various pages using PhantomJS, and then capturing a screen diff with the previous build. One could also capture the build time of each page, the number and duration of database queries, test coverage or myriad other metrics during testing. I'm wondering if we could make it easy to plug into the output directory to save all these things as artifacts.
Lots of food for thought for the week-end!
Cheers,
Albert.
Comment #3
moshe weitzman commented--out sounds good to me. in addition to the *.html files, it would be nice IMO to get a results page at index.html which is like what Simpletest UI gives today. This chronology is really easy to follow and would have links to the individual .html files. This would be a big jump in run-tests.sh usability. Drush knows how to launch a browser with its --browser option on various commands. Perhaps we could even add it to run-tests.sh so folks get that nice results overview quickly.
Comment #4
alberto56 commentedRelated: #2324955: In run-tests.sh, add an --xml-verbose argument allowing saving screen captures as artifacts, a less ambitions feature request which is more of a stopgap measure for those needing this functionality in D7.
Comment #5
sunJust to manage expectations here:
My goal is to get rid of Simpletest entirely for Drupal 8.0.0, replacing all test suites with PHPUnit. I personally attacked kernel tests, others are actively working on web tests. This is not a pipe-dream anymore. We're close to completion.
(The only thing that may prevent me from accomplishing that mission is running out of available time/money/resources.)
Like many other issues in the queue, this issue will most likely become obsolete with PHPUnit, too. PHPUnit has a much more sophisticated, pluggable API for test loggers and test result printers. — Even better, in most cases we don't need to invent anything custom at all, because there are existing PHPUnit extensions already.
So, back to expectations, I currently do not plan to waste any more time on Simpletest, and instead, focus 100% on getting rid of it as soon as possible.
Comment #16
quietone commentedThe simpletest module is obsolete in core.
Based on the last comment I am closing this as outdated.