[iOS] Add doc around creating EG2 tests.
Includes step by step instructions of writing a new test, talking with
app from test, add new tests to suites, running tests.
Bug: None
Change-Id: I39124524fae4bc56c9f3fbe02bca7d9387bdfefb
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/2503809
Commit-Queue: Zhaoyang Li <[email protected]>
Auto-Submit: Zhaoyang Li <[email protected]>
Reviewed-by: Eugene But <[email protected]>
Cr-Commit-Position: refs/heads/master@{#821486}
diff --git a/docs/ios/testing.md b/docs/ios/testing.md
index eeab38b..e761bd92 100644
--- a/docs/ios/testing.md
+++ b/docs/ios/testing.md
@@ -11,7 +11,125 @@
## Integration testing
-[EarlGrey] is the integration testing framework used by Chromium for iOS.
+[EarlGrey] (EG2) is the integration testing framework used by Chromium for iOS.
+
+### Writing EarlGrey tests
+
+#### Before you start
+
+* Just write a unit test if the purpose of your test does not involve UI.
+* Learn about EarlGrey test framework principles and APIs in [EarlGrey].
+* Learn about [Defining Test Cases and Test Methods] from Apple.
+
+#### Creating test files and writing EG2 tests
+
+1. EG2 test files are ended with _egtest.mm, and usually located within the same
+directory of the UI code you wish to test.
+2. Basic imports of a EG2 test file:
+
+ * You’ll have to include:
+ ```
+ #import "ios/chrome/test/earl_grey/chrome_test_case.h
+ ```
+ * You’ll most likely find util functions in these files helpful.
+ ```
+ #import "ios/chrome/test/earl_grey/chrome_earl_grey.h"
+ #import "ios/chrome/test/earl_grey/chrome_earl_grey_ui.h"
+ #import "ios/chrome/test/earl_grey/chrome_matchers.h"
+ ```
+ * Beside these, directly import an EG2 header for an EG2 API you are using.
+
+3. TestCase/testMethods definitions. Create `SomeGreatTestCase` as a subclass of
+`ChromeTestCase`. Create test methods, eg `-(void)testMyGreatUIFeature {...}`,
+and put UI actions within the test methods.
+ * Put your setup and tear down code for the *TestCase* in
+`+(void)setUpForTestCase` and `+tearDown`. These will run once before and
+after all tests for the test class.
+ * Put your setup and tear down code for each *test method* in `-(void)setUp`
+and `-(void)tearDown`. These will run before and after every
+`-(void)testMethod` in the file.
+4. Writing test contents. See the chrome helpers (imports in 2.) as well as
+[EarlGrey APIs] to write a UI action/assertion in your testMethod.
+
+#### Interacting with the app in a test
+
+##### Relaunch app with different flags
+
+In EG2 tests, the test process launches the host app process at the beginning,
+then runs UI actions/assertions in the app. To pass args or feature flags to the
+app at initial launching, or relaunch the app in the middle of your test, see
+[this AppLaunchManager API].
+
+##### Accessing app internals
+
+EG2 test targets are built with test-related code but without app code.
+
+To access anything from the app side, use an "app interface". App interface is
+implemented as a class that lives in the app process, but can be accessed in the
+test process through [eDO]. You can include the header in your test side code
+and call class methods of the interface class. The methods will execute code in
+the app process and can return basic Objective-C types. See this [Example of App
+ Interface].
+
+See `eg_test_support+eg2` (test side utilities) and `eg_app_support+eg2` (app
+side utilities) targets in `BUILD.gn` files to learn how test utilities are
+organized in targets. If you added an app side helper (app interface), you’ll
+also need to include your new `eg_app_support+eg2` target in
+`//ios/chrome/test/earl_grey/BUILD.gn`’s `eg_app_support+eg2` target. ([Example
+ CL adding App Interface]).
+
+Note that if you create an App interface, you can’t build the app interface
+class in your eg2_tests target, but you need to include and refer to it. If you
+see a “Undefined symbols for architecture… MyTestAppInterface”, add
+```
+#if defined(CHROME_EARL_GREY_2)
+// TODO(crbug.com/1015113) The EG2 macro is breaking indexing for some reason
+// without the trailing semicolon. For now, disable the extra semi warning
+// so Xcode indexing works for the egtest.
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wc++98-compat-extra-semi"
+GREY_STUB_CLASS_IN_APP_MAIN_QUEUE(HandoffManagerAppInterface);
+#pragma clang diagnostic pop
+#endif // defined(CHROME_EARL_GREY_2)
+```
+to the top of your foo_egtest.mm file.
+
+#### Creating test targets and adding the target to test suites
+
+1. Create a test target. Add a target(`source_set`) named "eg2_tests" into the
+closest `BUILD.gn` file. Put the test file into the `sources` array and put the
+targets containing headers used in your test file into `deps` array. This is to
+organize test source files and dependencies so that the GN build system can
+correctly build the test module. The skeleton of the target:
+```
+source_set("eg2_tests") {
+ configs += [
+ "//build/config/compiler:enable_arc",
+ "//build/config/ios:xctest_config",
+ ]
+ testonly = true
+ sources = [
+ "some_egtest.mm"
+ ]
+ deps = [
+ "//ios/chrome/test/earl_grey:eg_test_support+eg2",
+ "//ios/testing/earl_grey:eg_test_support+eg2",
+ "//ios/third_party/earl_grey2:test_lib",
+ ]
+ libs = [ "UIKit.framework" ]
+}
+```
+2. Include your test target in the `deps` array of a suitable suite in
+`//src/ios/chrome/test/earl_grey2/BUILD.gn`.
+3. Optional: If you feel like your new test should be in a new suite, or you
+want to delete an existing suite to make tests better organized, you’ll need to
+change the suites in `//src/ios/chrome/test/earl_grey2/BUILD.gn` in the format
+of existing ones. (Do not forget to [config the bots] so the new suite can run
+in infra.)
+4. Ensure your dependencies are correct.
+```
+$ gn gen --check out/Debug-iphonesimulator
+```
### Running EarlGrey tests
@@ -19,27 +137,74 @@
#### Running tests from Xcode
-An entire suite of tests can be run from Xcode.
-1. Select the *egtest target you wish to run.
-2. ⌘+U to run all the tests. Note: ⌘+R, which is normally used to run an
-application, will simply launch the app under test, but will not run the
-XCTests.
-
-A subset of tests can be run by selecting the test or test case from the
-XCTest navigator on the left side of the screen.
+1. If you added a new test file / suite, run `gclient runhooks` to sync for the
+list of tests in Xcode.
+2. Change the scheme to "ios_chrome_eg2test". Create and select the simulator
+you wish to use.
+3. You may run a test suite(module), TestCase or testMethod in test navigator.
+Xcode will build the targets and run the test(s) you choose. Alternatively,
+use ⌘+U to run all the tests. See Apple's [Running Tests and Viewing Results].
#### Running from the command-line
-When running from the command-line, it is required to pass in the *.xctest
-target, in addition to the test application.
+EG2 tests can run in the command line with test runner scripts. You’ll need to
+build the targets before running tests in cmd. This is used by continuous
+integration infra and thus not user friendly. Running UI tests directly in Xcode
+is recommended.
+
+Important notes:
+* The test runner can invoke mac_toolchain to install a new Xcode of the version
+specified to the path specified. You may want to choose a different path from
+your daily use Xcode.
+* If test_cases is empty in --args-json, all tests will run. Specifying a
+testMethod to run is currently not supported in the test runner.
+
Example:
```
-./out/Debug-iphonesimulator/iossim -d "iPad Retina" -s 8.1 \
-out/Debug-iphonesimulator/ios_chrome_integration_egtests.app \
-out/Debug-iphonesimulator/ios_chrome_integration_egtests_module.xctest
+src/ios/build/bots/scripts/run.py
+ --app
+ src/out/Debug-iphonesimulator/ios_chrome_ui_eg2tests_module-Runner.app
+ --host-app
+ src/out/Debug-iphonesimulator/ios_chrome_eg2tests.app
+ --args-json
+ {"test_args": [], "xctest": false, "test_cases": ["ReadingListTestCase"],
+ "restart": false, "xcode_parallelization": true, "xcodebuild_device_runner":
+ false}
+ --out-dir
+ path/to/output/dir
+ --retries
+ 3
+ --shards
+ 1
+ --xcode-build-version
+ 11c29
+ --mac-toolchain-cmd
+ path/to/mac_toolchain
+ --xcode-path
+ path/to/Xcode.app
+ --wpr-tools-path
+ NO_PATH
+ --replay-path
+ NO_PATH
+ --iossim
+ src/out/Debug-iphonesimulator/iossim
+ --platform
+ iPad (6th generation)
+ --version
+ 13.3
```
+The invocation args are logged. You can find the latest arg format at the
+beginning of stdout from an infra test shard if the above doesn't work.
-[EarlGrey]: https://github.com/google/EarlGrey
+[config the bots]: https://chromium.googlesource.com/chromium/src/testing/+/refs/heads/master/buildbot/README.md#buildbot-testing-configuration-files
+[Defining Test Cases and Test Methods]: https://developer.apple.com/documentation/xctest/defining_test_cases_and_test_methods?language=objc
+[EarlGrey]: https://github.com/google/EarlGrey/tree/earlgrey2
+[EarlGrey APIs]: https://github.com/google/EarlGrey/blob/master/docs/api.md
+[eDO]: https://github.com/google/eDistantObject
+[Example of App Interface]: https://cs.chromium.org/chromium/src/ios/chrome/browser/metrics/metrics_app_interface.h
+[Example CL adding App Interface]: https://chromium-review.googlesource.com/c/chromium/src/+/1919147
[instructions]: ./build_instructions.md
+[Running Tests and Viewing Results]: https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/05-running_tests.html
+[this AppLaunchManager API]: https://source.chromium.org/chromium/chromium/src/+/master:ios/testing/earl_grey/app_launch_manager.h;drc=d0889865de20c5b3bc59d58674eb2dcc02dd2269;l=47
[XCUITest]: https://developer.apple.com/documentation/xctest
