[Plan] Roadmap for Simpletest

Simpletest is a legacy testing framework which comes out of the Drupal 6 days, was in core in Drupal 7, and is supported in Drupal 8 alongside PHPUnit-based frameworks.

Defining Some Terms

When we refer to 'simplest', we could be referring to a few distinct-but-overlapping things:

1. The simpletest module, AKA 'Testing'. This module is in core, and is a hard dependency of all tests, PHPUnit or otherwise. It also provides a way to run tests in the user interface, through a form. We call this feature the 'simpletest UI.'
2. The simpletest framework, AKA TestBase and WebTestBase (aka WTB). This framework lives in the simpletest module. Most WTB tests have been converted to their PHPUnit equivalents. The remaining core tests which use this framework are tests of the framework itself, marked with @group WebTestBase.
3. The test runner, AKA run-tests.sh. This is a script which can run PHPUnit-based tests as well as WTB ones.

Moving forward to Drupal 9, the simpletest framework and the simpletest UI should be phased out in a reasonable way.

We currently have an initiative to convert core's tests to use PHPUnit-based frameworks, and not use Simpletest-based ones. This initiative has completed. #2807237: PHPUnit initiative

This still leaves us with many legacy tests in contrib for Drupal 8, which we must still support, but which we'd rather not maintain.

The Testbot

DrupalCI testbot currently runs all the tests by invoking the run-tests.sh script.

The obvious reason for this is that there should be as few points of contact as possible between the test runner and the system under test. But there are other reasons as well:

• PHPUnit has an undefined behavior if the test process has a fatal error. That is, if a test crashes, the phpunit tool does not continue running other tests, and in fact does not generate a report of the tests that did run.
• PHPUnit does not support concurrent test runs. This means that all tests must be run sequentially, leading to long CI times.

These factors mean that we should keep the behavior of run-tests.sh. It provides us with features we can't easily get from other libraries, and will be consistent for backwards-compatibility.

The Users

Some users want to keep the simpletest UI. Others want to remove it, including a general consensus from maintainers. The current issue for this is #3028663: Split up simpletest into simpletest and testing_ui to enable easier decisions for Drupal 9 where we are teasing the UI portions out of the simpletest module into a testing_ui module, so that we can decide later.

The Maintainers

Core maintainers and others who wish to work on the testing subsystem are frequently frustrated with the fragile nature of the simpletest testing system.

It is difficult to test the testing system without refactoring, and is difficult to refactor without testing, and the quality of the rest of the code depends on the code that is demonstrated to work. Thus it is difficult to work on the testing subsystem from both the old-crusty-code perspective, as well as the reluctance-to-change perspective.

Note that most of the simpletest module is not marked as deprecated. There are two categories of things which have been deprecated:

• The simpletest framework: TestBase and its children. These are the test types we wish to quit supporting.
• Global-level functions which have been replaced by the test_discovery service.

This means a deprecation process will be needed if any modernizing changes are to happen in simpletest, or if it is to be removed once and for all.

The Internals

run-tests.sh and the simpletest module contain many shims that convert one thing to another in order to provide output that is known to be necessary. This includes converting PHPUnit JUnit test results to the simpletest db table schema and then from that table to JUnit again, and similar operations.

The path of execution for these processes extend between global-level functions in run-tests.sh and simpletest.module. Note that neither of these files has a .php extension. That aside, this creates a dependency between the script and the module, which makes it increasingly unmaintainable.

Suffice it to say that our goal is to keep run-tests.sh for the foreseeable future, and so we should refactor towards that end.

The Plan

So those are some of the problems in this space.

The solution seems to look like the following:

Consistency in Drupal 8

For the rest of the Drupal 8 lifecycle, we should provide the same behaviors from simpletest and run-tests.sh as we have now. We should provide the normal BC layer that we'd expect from any deprecated Drupal 8 code.

Drupal 8 should still be able to run Simpletest-based tests. While there is an initiative to phase out these tests in core, we're still promising to be able to run them. We need to fulfill this promise so that the testbot can still run contrib tests.

• #2807237: PHPUnit initiative

Drupal 8 should still have the Simpletest test runner UI. This form should be present and should be useable as it is now. It might be enhanced in some ways, but this should not be our focus. Whether to keep this UI in Drupal 9 can be an open conversation.

• #3028663: Split up simpletest into simpletest and testing_ui to enable easier decisions for Drupal 9
• #2750461: Remove Simpletest UI because we don't want to maintain a graphical test runner
• #2301481: Mark test groups as belonging to modules in UI
• #2858652: Support multiple @group test annotations in Simpletest UI

For Drupal 8, top-level functions in simpletest.module and run-tests.sh should be deprecated. Their functionality should be moved to helper classes which can be unit-tested with relative ease.

• #2863044: [plan] Remove simpletest dependencies from PHPUnit-based tests
• #2863055: Move TestDiscovery out of simpletest module, minimize dependencies
• #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate
• #2800267: Turn simpletest_clean*() functions into a helper class
• #2798273: simpletest_phpunit_configuration_filepath() is dead code. Deprecate it.

Drupal 8's simpletest hook APIs should be deprecated. They are only currently supported for the Simpletest UI at this point, which is an inconsistency that will need to be maintained for the D8 life cycle. We can examine whether these hooks should be implemented within run-tests.sh, but since we're trying to move away from simpletest-isms, and since PHPUnit-based test listeners and alternate test suites are better available workarounds, this seems unlikely.

• #2460521: Deprecate hook_simpletest_alter()
• #2234479: Deprecate hook_test_* hooks in simpletest

Radical changes to Drupal 8, with BC

In the near future, the simpletest UI form should be refactored into the testing_ui module: #3028663: Split up simpletest into simpletest and testing_ui to enable easier decisions for Drupal 9

All test-running behavior should be factored into a core namespace. This would mean taking all behaviors out of the simpletest module and placing them into the \Drupal\Tests\ namespace.

All of run-tests.sh's behavior would also be refactored into this namespace, except the script itself which would then use these classes to run the tests.

This namespace should not be autoloaded unless it's test-running time. This means the code should be refactored into the core/tests/ directory. Whatever test-running UI is available from testing_ui can autoload test-running code from this directory if it is enabled and running tests.

Once the code is moved in a consistent process into the core/tests/ directory, it could be turned into Composer-based subtree split. This would allow the testing infrastructure to be a dev dependency of core, removing it from the codebase of production sites.

This is the ideal goal for the simpletest refactor: That one could remove it from a production codebase using Composer, by issuing this command: composer install --no-dev Any point along the line between where things stand now, and this complete removal would be a step in the right direction.

Drupal 9

The Drupal 9 process would remove the deprecated top-level functions from the simpletest infrastructure itself.

The conversation over whether to keep the test runner UI would hopefully happen before the 9.0.0 release.

Since the other refactoring would have already occurred, the decision to remove the UI would be the same as deciding to remove the module from core. All other testing infrastructure provided by core would remain.