Change Simpletest UI from a test runner to a CLI snippet generator

I really like the idea! Do you think it would make sense to also show the phpunit test command in case you run one of those?

@dawehner yes if only phpunit tests are selected.

what i think of this is irrelevant but most definitely get YesCT and webchick approval before doing this.

Until I can reliably see the HTML output of pages via the command line, I don't think this will work.

For example, here's what I got via the CLI:

Drupal test run
---------------

Tests to be run:
  - Drupal\link\Tests\LinkFieldUITest

Test run started:
  Saturday, August 2, 2014 - 10:25

Test summary
------------

Drupal\link\Tests\LinkFieldUITest                             17 passes   4 fails   1 exceptions             

Test run duration: 19 sec

Here's what I got from qa.drupal.org:

And here's what I got from the simpletest UI results:

Guess which one of these helped me find the bug fastest?

@tim.plunkett
Doesn't --verbose gives you the link when things failed? If not we should certainly add it independent from this issue.

Nope.

$ php core/scripts/run-tests.sh --verbose --url http://d8 --color --class "Drupal\link\Tests\LinkFieldUITest"

Drupal test run
---------------

Tests to be run:
  - Drupal\link\Tests\LinkFieldUITest

Test run started:
  Saturday, August 2, 2014 - 12:44

Test summary
------------

Drupal\link\Tests\LinkFieldUITest                             17 passes   4 fails   1 exceptions   5 messages

Test run duration: 1 min 12 sec

Detailed test results
---------------------


---- Drupal\link\Tests\LinkFieldUITest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      LinkFieldUITest.p   27 Drupal\link\Tests\LinkFieldUITest->
    Enabled modules: node, link, field_ui
Pass      Role       LinkFieldUITest.p   29 Drupal\link\Tests\LinkFieldUITest->
    Created role ID 'i7k8zfbt' with name '?JsYFl7<'.
Pass      Role       LinkFieldUITest.p   29 Drupal\link\Tests\LinkFieldUITest->
    Created permissions: administer content types, administer node fields,
    administer node display
Pass      User login LinkFieldUITest.p   29 Drupal\link\Tests\LinkFieldUITest->
    User created with name meDFdHG6 and pass zxDkk9H8kj
Pass      Browser    LinkFieldUITest.p   30 Drupal\link\Tests\LinkFieldUITest->
    GET http://d8/user returned 200 (3.53 KB).
Pass      Browser    LinkFieldUITest.p   30 Drupal\link\Tests\LinkFieldUITest->
    Valid HTML found on "http://d8/user"
Pass      Browser    LinkFieldUITest.p   30 Drupal\link\Tests\LinkFieldUITest->
    GET http://d8/user/2 returned 200 (3.88 KB).
Pass      Browser    LinkFieldUITest.p   30 Drupal\link\Tests\LinkFieldUITest->
    Valid HTML found on "http://d8/user/2"
Pass      User login LinkFieldUITest.p   30 Drupal\link\Tests\LinkFieldUITest->
    User meDFdHG6 successfully logged in.
Pass      Other      LinkFieldUITest.p   38 Drupal\link\Tests\LinkFieldUITest->
    Created content type rujiuex7.
Pass      Browser    LinkFieldUITest.p   49 Drupal\link\Tests\LinkFieldUITest->
    GET http://d8/admin/structure/types/manage/rujiuex7/fields returned 200
    (10.31 KB).
Pass      Browser    LinkFieldUITest.p   49 Drupal\link\Tests\LinkFieldUITest->
    Valid HTML found on "http://d8/admin/structure/types/manage/rujiuex7/fields"
Exception Notice     FormSubmitter.php  160 Drupal\Core\Form\FormSubmitter->red
    Undefined offset: 0
Pass      Browser    LinkFieldUITest.p   49 Drupal\link\Tests\LinkFieldUITest->
    GET http://d8/user/2 returned 200 (5.07 KB).
Pass      Browser    LinkFieldUITest.p   49 Drupal\link\Tests\LinkFieldUITest->
    Valid HTML found on "http://d8/user/2"
Fail      Other      LinkFieldUITest.p   51 Drupal\link\Tests\LinkFieldUITest->
    Found the Save field settings button
Fail      Other      LinkFieldUITest.p   51 Drupal\link\Tests\LinkFieldUITest->
    Found the requested form fields at
Fail      Other      LinkFieldUITest.p   53 Drupal\link\Tests\LinkFieldUITest->
    Found the Save settings button
Fail      Other      LinkFieldUITest.p   53 Drupal\link\Tests\LinkFieldUITest->
    Found the requested form fields at
Pass      Browser    LinkFieldUITest.p   58 Drupal\link\Tests\LinkFieldUITest->
    GET http://d8/admin/structure/types/manage/rujiuex7/display returned 200
    (18.17 KB).
Pass      Browser    LinkFieldUITest.p   58 Drupal\link\Tests\LinkFieldUITest->
    Valid HTML found on
    "http://d8/admin/structure/types/manage/rujiuex7/display"
Pass      Other      LinkFieldUITest.p   59 Drupal\link\Tests\LinkFieldUITest->
    "Link text trimmed to 80 characters" found

Yep we need to be able to use run-tests.sh and end up with a browser window open with the test results and the ability to see any pages visited by the simpletest browser. I think #2253999-3: Provide a way to keep test results and verbose messages as artifacts is the issue to do that. So postponing this on that issue.

In general, I fully agree with the premise of this issue. Especially the parent/child site environment leakage is a fundamental problem of our current testing framework.

That said, I'd like to throw an alternative option into the discussion, which still achieves the same, but retains the UI as-is:

---
Change the (front-end) batch.js to request a custom and minimal test front-controller PHP file, which does NOT boot Drupal, executes a test, and returns test results in a response in the format expected by Batch API.

That will execute the test in a clean and cleanly isolated environment, identically to CLI, merely invoked by a different SAPI.
---

The idea was born when I worked on a cleanup of run-tests.sh some months ago, with the goal of removing all Drupal dependencies from the script. Truth is, even the run-tests.sh CLI script environment leaks, because it boots Drupal prior to executing a test. The prototype successfully worked, but wasn't complete, and it (obviously) revealed that many tests are currently "relying" on leaked environment aspects. Need to grep my sources to find that code.

Of course, the web-based front-controller script would not be run-tests.sh, because that's explicitly designed for CLI. But the task is the same: Bootstrap the most minimal environment possible to execute a test.

This most minimal environment is expressed/resembled by /core/tests/bootstrap.php already, which thus presents the benchmark: (1) ClassLoader, (2) Register namespaces, (3) Minimally required constants + PHP INI settings, (4) Done (Run test).

Existing Web UIs for PHPUnit are using the identical approach to run tests from a browser. Obviously, as a standalone application, their unique benefit is that they cannot run into the trap of unintentionally priming the environment with application-specific state/aspects, because they're not aware of your custom code in the first place.

However, it should be easily possible to mimic that behavior in a custom UI. In fact, that's relatively simple and not the challenge. The challenge is to fix our existing tests to no longer rely on the environment primed by the test runner.

If we were to pursue this approach, then the necessary roadmap would roughly be:

1. Enhance Batch API to allow to specify a custom batch processing endpoint URI. (unless that is possible already)
2. Fix run-tests.sh to no longer boot Drupal in any way.
3. Fix WebTestBase to no longer assume a parent site.
4. Fix (plenty of) tests that are unintentionally/unknowingly relying on the test runner environment currently.
5. Based on 2), create a test front-controller script (batch endpoint) that executes one test (or ideally multiple, depending on test type).

The test front-controller script simply outputs the test results into the response. We can choose from any serialization format natively supported by PHPUnit (JSON, XML/JUnit, TAP) and re-use its code to generate test result responses for legacy Simpletest tests (until its gone).

The UI simply consumes and processes the response data, purely in the front-end. When running tests through the UI, test results will not be written to any file nor stored anywhere — The test runner/dispatcher/logger is the browser (page) only.

Consequently, the test results page of the UI would be dynamically composed via JS; de facto a micro app in the front-end. Quite possibly, it would even make sense to remove Batch API altogether from the equation, in favor of asynchronous requests to execute multiple tests and/or test suites in parallel. We have plenty of JS/front-end talent that can be leveraged.

I'm fairly confident that this presents a viable alternative solution that still resolves the problems, while retaining a nice UX for newcomers/developers who are intimidated by CLI operations or those who just simply prefer to look at rich HTML output. Thoughts?

Indeed, good point. In essence, it's a new phpunit_ui.module that strictly focuses on providing a UI for executing PHPUnit. It may happen to have a temporary BC layer for legacy Simpletest tests, but that should be an afterthought.

However, I'm not sure whether it's possible to achieve this outside of core — unless we'd independently remove the current Simpletest UI(!) upfront + adjust run-tests.sh + WebTestBase accordingly.

We now have --browser and --verbose which gives you pretty much all you need already. The result page certainly has all the information you'll ever need.

OMG PLEASE YES

Here is a start.

Good idea as a shortcut around the complexity of maintaining the UI testing page.

Good idea in that there are (at least) two testing systems people use regularly, and there should only be one.

Bad idea in that run-tests.sh is really cumbersome, and this issue locks us into its arguments and options, which really need some improvement.

In a better universe, we'd refactor both run-tests.sh and simpletest module to share test-running infrastructure.

In the money-where-my-mouth-is department: #2624926: Refactor run-tests.sh for Console component.

Moving to Simpletest contrib issue space.