Tests are identified by names like login.Chrome or platform.ConnectToDBus. The portion before the period, called the category, is the final component of the test's package name, while the portion after the period is the name of the exported Go function that implements the test.
Test function names should follow Go's naming conventions, and acronyms should be fully capitalized. Test names should not end with Test, both because it's redundant and because the _test.go filename suffix is reserved in Go for unit tests.
Test names are automatically derived from tests' package and function names and should not be explicitly specified when defining tests.
Public tests built into the default cros local and remote test bundles are checked into the tast-tests repository under the src/go.chromium.org/tast-tests/cros/local/bundles/cros/ and src/go.chromium.org/tast-tests/cros/remote/bundles/cros/ directories (where src/go.chromium.org/tast-tests/cros/ can also be accessed by the cros symlink at the top of the repository). Private tests are checked into private repositories such as the tast-tests-private repository, and built into non-cros test bundles.
Tests are categorized into packages based on the functionality that they exercise; for example, the ui package contains local tests that exercise the ChromeOS UI. The category package needs to be directly under the bundle package. Thus the category package path should be matched with go.chromium.org/tast/core/(local|remote)/bundles/(?P<bundlename>[^/]+)/(?P<category>[^/]+).
A local test named ui.DoSomething should be defined in a file named src/go.chromium.org/tast-tests/cros/local/bundles/cros/ui/do_something.go (i.e. convert the test name to lowercase and insert underscores between words).
Support packages used by multiple test categories located in src/go.chromium.org/tast-tests/cros/local/ and src/go.chromium.org/tast-tests/cros/remote/, alongside the bundles/ directories. For example, the chrome package can be used by local tests to interact with Chrome.
If there‘s a support package that’s specific to a single category, it‘s often best to place it underneath the category’s directory. See the Scoping and shared code section.
Tast-tests-private repository go.chromium.org/tast-tests-private/... should not import packages bundle in go.chromium.org/tast-tests/cros/local/bundles/... and go.chromium.org/tast-tests/cros/remote/bundles/...
Packages outside go.chromium.org/tast-tests/cros/local/... should not import packages in go.chromium.org/tast-tests/cros/local/..., and packages outside go.chromium.org/tast-tests/cros/remote/... should not import packages in go.chromium.org/tast-tests/cros/remote/.... If local and remote packages should share the same code, put them in go.chromium.org/tast-tests/cros/common/....
A test needs to be registered by calling testing.AddTest() in the test entry file, which is located directly under a category package. The registration needs to be done in init() function in the file. The registration should be declarative, which means:
testing.AddTest() should be the only statement of init()'s body.testing.AddTest() should take a pointer of a testing.Test composite literal.Each field of testing.Test should be constant-like. Fields should not be set using the invocation of custom functions (however, append() is allowed), or using variables. In particular, we say constant-like is any of these things:
The test registration code will be similar to the following:
// Copyright 2018 The ChromiumOS Authors // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. package ui import ( "context" "go.chromium.org/tast/core/testing" ) func init() { testing.AddTest(&testing.Test{ Func: DoSomething, Desc: "Does X to verify Y", Contacts: []string{"[email protected]", "[email protected]"}, BugComponent: "b:12345", Attr: []string{"group:mainline", "informational"}, SoftwareDeps: []string{"chrome"}, Timeout: 3 * time.Minute, }) } func DoSomething(ctx context.Context, s *testing.State) { // The actual test goes here. }
Tests have to specify the descriptions in Desc, which should be a string literal.
Tests have to specify email addresses of persons and groups who are familiar with those tests in Contacts. The first element of the slice should be a group alias for the team ultimately responsible for the test. Subsequent elements should be individuals or groups who can be contacted for code reviews, bugs, and any issue with the test‘s usage. To help aid triage and on-call rotations, partner owned tests must specify a Google email contact that can be reached by on-call rotations. Any google.com or chromium.org groups listed should accept email posts from non-members within the organization. Users who no longer work on Chrome OS or with test’s owning team should remove themselves as a contact.
Tests have to specify a BugComponent, which should be a string with a prefix indicating the bug tracker. The string's contents point to the location where bugs regarding the test should initially be filed. A prefix is used to distinguish between different bug trackers. For Buganizer, use “b:” plus the componentid, e.g. “b:1034625”. Ensure that a componentid is used, not a specific bug id.
Tests have to specify attributes to describe how they are used in ChromeOS testing. A test belongs to zero or more groups by declaring attributes with group:-prefix. Typically functional tests belong to the mainline group by declaring the group:mainline attribute. New mainline tests should have the informational attribute, as tests without this attribute will block the Commit Queue on failure otherwise. The Attr fields should be an array literal of string literals.
The SoftwareDeps field lists software dependencies that should be satisfied in order for the test to run. Its value should be an array literal of string literals or (possibly qualified) identifiers which are constant value.
Tests should always set the Timeout field to specify the maximum duration for which Func may run before the test is aborted. If not specified, a reasonable default will be used, but tests should not depend on it.
If a test has no group:* attribute assigned it will be effectively disabled, it will not be run by any automation. If a test needs to be disabled leave a comment in the test source with the reason. If applicable, create a bug explaining under what circumstances the test can be enabled.
When adding a new test category, you must update the test bundle's imports.go file (either local/bundles/cros/imports.go or remote/bundles/cros/imports.go) to underscore-import the new package so its init functions will be executed to register tests.
Test code should be formatted by gofmt and checked by go vet, staticcheck and tast-lint. These tools are configured to run as pre-upload hooks, so don't skip them.
Tast code should also follow Go's established best practices as described by these documents:
The Go FAQ may also be helpful. Additional resources are linked from the Go Documentation page.
Packages and exported identifiers (e.g. types, functions, constants, variables) should be documented by Godoc-style comments. Godoc comments are optional for test functions, since the Test.Desc field already contains a brief description of the test.
Support packages should be exercised by unit tests when possible. Unit tests can cover edge cases that may not be typically seen when using the package, and they greatly aid in future refactorings (since it can be hard to determine the full set of Tast-based tests that must be run to exercise the package). See How to Write Go Code: Testing and Go's testing package for more information about writing unit tests for Go code. The Best practices for writing ChromeOS unit tests document contains additional suggestions that may be helpful (despite being C++-centric).
Setting FEATURES=test when emerging a test bundle package (tast-local-tests-cros or tast-remote-tests-cros) will run all unit tests for the corresponding packages in the tast-tests repository (i.e. go.chromium.org/tast-tests/cros/local/... or go.chromium.org/tast-tests/cros/remote/..., respectively).
During development, the fast_build.sh script can be used to quickly build and run tests for a single package or all packages.
Entries in import declaration must be grouped by empty line, and sorted in following order.
In each group, entries must be sorted in the lexicographical order. For example:
import (
"context"
"fmt"
"github.com/godbus/dbus/v5"
"golang.org/x/sys/unix"
"go.chromium.org/tast/core/errors"
"go.chromium.org/tast-tests/cros/local/chrome"
)
Note that, although github.com and golang.org are different domains, they should be in a group.
This is how goimports --local=chromiumos/ sorts. It may be valuable to run the command. Note that, 1) the command preserves existing group. So, it may be necessary to remove empty lines in import() in advance, and 2) use the command to add/remove import entries based on the following code. The path resolution may require setting GOPATH properly.
As seen in the test declaration above, each test is comprised of a single exported function that receives a testing.State struct. This is defined in the Tast testing package (not to be confused with [Go's testing package] for unit testing) and is used to log progress and report failures.
If a test requires the system to be in a particular state before it runs, it should include code that tries to get the system into that state if it isn‘t there already. Previous tests may have aborted mid-run; it’s not safe to make assumptions that they undid all temporary changes that they made.
Tests should also avoid performing unnecessary de-initialization steps on completion: UI tests should leave Chrome logged in at completion instead of restarting it, for example. Since later tests can‘t safely make assumptions about the initial state of the system, they’ll need to e.g. restart Chrome again regardless, which takes even more time. In addition to resulting in a faster overall running time for the suite, leaving the system in a logged-in state makes it easier for developers to manually inspect it after running the test when diagnosing a failure.
Note that tests should still undo atypical configuration that leaves the system in a non-fully-functional state, though. For example, if a test needs to temporarily stop a service, it should restart it before exiting.
Use defer statements to perform cleanup when your test exits. defer is explained in more detail in the Defer, Panic, and Recover blog post.
Put more succintly:
Assume you‘re getting a reasonable environment when your test starts, but don’t make assumptions about Chrome‘s initial state. Similarly, try to leave the system in a reasonable state when you go, but don’t worry about what Chrome is doing.
Tast uses context.Context to implement timeouts. A test function takes as its first argument a context.Context with an associated deadline that expires when the test‘s timeout is reached. The default timeout is 2 minutes for local tests and 5 minutes for remote tests. The context’s Done function returns a channel that can be used within a select statement to wait for expiration, after which the context's Err function returns a non-nil error.
The testing.Poll function makes it easier to honor timeouts while polling for a condition:
if err := testing.Poll(ctx, func (ctx context.Context) error {
var url string
if err := MustSucceedEval(ctx, "location.href", &url); err != nil {
return testing.PollBreak(errors.Wrap(err, "failed to evaluate location.href"))
}
if url != targetURL {
return errors.Errorf("current URL is %s", url)
}
return nil
}, &testing.PollOptions{Timeout: 10 * time.Second}); err != nil {
return errors.Wrap(err, "failed to navigate")
}
Return a testing.PollBreak error to stop the polling. Useful when you get an unexpected error inside the polling.
Sleeping without polling for a condition is discouraged, since it makes tests flakier (when the sleep duration isn't long enough) or slower (when the duration is too long). If you really need to do so, use testing.Sleep to honor the context timeout.
Any function that performs a blocking operation should take a context.Context as its first argument and return an error if the context expires before the operation finishes.
Several blog posts discuss these patterns in more detail:
Note: there is an old equivalent “golang.org/x/net/context” package, but for consistency, the built-in “context” package is preferred.
As a rule of thumb, a timeout should be double of the expected worst case performance. If you're unsure, measure time multiple times in the worst case scenario and double that. Do not use timeouts to catch performance regressions. Instead consider writing a performance test. When a test hits a timeout that was sufficient before, investigate why it hit the timeout before increasing it.
The performance and worst case scenario can be obtained using the time calculation script. It parses the test result logs to obtain the average and max time from various executions.
For any function with a corresponding clean-up function, prefer using the defer statement to keep the two function calls close together (see the Startup and shutdown section for detail).
Create a separate clean up context using ctxutil.Shorten to make sure to reserve enough time for your deferred functions to run even when the test context timed out.
cleanupCtx := ctx
ctx, cancel := ctxutil.Shorten(ctx, timeForCleanup)
defer cancel()
...
service := createService(ctx, ...)
defer func(ctx context.Context) {
if err := service.Close(ctx); err != nil {
s.Error("Failed to close service: ", err)
}
}(cleanupCtx)
Make sure you catch and report errors in cleanup functions. You can use s.Error to continue running the current deferred function and return if your cleanup function needs to exit early. Avoid using s.Fatal in deferred functions.
Concurrency is rare in integration tests, but it enables doing things like watching for a D-Bus signal that a process emits soon after being restarted. It can also sometimes be used to make tests faster, e.g. by restarting multiple independent Upstart jobs simultaneously.
The preferred way to synchronize concurrent work in Go programs is by passing data between goroutines using a channel. This large topic is introduced in the Share Memory by Communicating blog post, and the Go Concurrency Patterns talk is also a good summary. The Go Memory Model provides guarantees about the effects of memory reads and writes across goroutines.
Global variables in Go are scoped at the package level rather than the file level:
The scope of an identifier denoting a constant, type, variable, or function ... declared at top level (outside any function) is the package block.
As such, all tests within a package like platform or ui share the same namespace. It is ok to declare top level unexported symbols (e.g. functions, constants, etc), but please be careful of conflicts. Also, please avoid referencing identifiers declared in other files; otherwise repo upload will fail with lint errors.
If you need to share functionality between tests in the same package, please introduce a new descriptively-named subpackage; see e.g. the chromecrash package within the ui package, used by the ui.ChromeCrashLoggedIn and ui.ChromeCrashNotLoggedIn tests. Subpackages are described in more detail later in this document. Importing a subpackage is allowed only in the category package containing it; otherwise repo upload will fail with lint errors.
Much praise has been written for verifying just one thing per test. A quick sampling of internal links:
While this is sound advice for fast-running, deterministic unit tests, it isn't necessarily always the best approach for integration tests:
If you need to verify multiple related aspects of a single feature that requires a time-consuming setup process like logging in to Chrome, starting Android, or launching a container, it's often preferable to write a single test that just does the setup once and then verifies all aspects of the feature. As described in the Errors and Logging section, multiple errors can be reported by a single test, so coverage need not be reduced when tests are consolidated and an early expectation fails.
For lightweight testing that doesn‘t need to interact with Chrome or restart services, it’s fine to use fine-grained tests — there's almost no per-test overhead in Tast; the overhead comes from repeating the same slow operations within multiple tests.
A Tast test either passes (by reporting zero errors) or fails (by reporting one or more errors, timing out, or panicking). If a test requires functionality that isn't provided by the DUT, the test is skipped entirely.
Avoid writing tests that probe the DUT's capabilities at runtime, e.g.
// WRONG: Avoid testing for software or hardware features at runtime. func CheckCamera(ctx context.Context, s *testing.State) { if !supports720PCamera() { s.Log("Skipping test; device unsupported") return } // ... }
This approach results in the test incorrectly passing even though it actually didn‘t verify anything. (Tast doesn’t let tests report an “N/A” state at runtime since it would be slower than skipping the test altogether and since it will prevent making intelligent scheduling decisions in the future about where tests should be executed.)
Instead, specify software dependencies when declaring tests:
// OK: Specify dependencies when declaring the test. func init() { testing.AddTest(&testing.Test{ Func: CheckCamera, SoftwareDeps: []string{"camera_720p", "chrome"}, // ... }) }
The above document describes how to define new dependencies.
Also, there is an API Features which allows tests to get information regarding DUT features. However, it is purely used for informational purpose only. Do not use it to alter test behavior. Use it for only for informational purpose. Use parameterized tests for tests to have different behavior for different DUT features.
If a test depends on the DUT being in a specific configurable state (e.g. tablet mode), it should put it into that state. For example, chrome.ExtraArgs can be passed to chrome.New to pass additional command-line flags (e.g. --force-tablet-mode=touch_view) when starting Chrome.
The tast-users mailing list is a good place to ask questions about test dependencies.
Sometimes a lengthy setup process (e.g. restarting Chrome and logging in, which takes at least 6-7 seconds) is needed by multiple tests. Rather than running the same setup for each of those tests, tests can declare the shared setup, which is named “fixtures” in Tast.
Tests sharing the same fixture run consecutively. A fixture implements several lifecycle methods that are called by the framework as it executes tests associated with the fixture. SetUp() of the fixture runs once just before the first of them starts, and TearDown() is called once just after the last of them completes unless s.Fatal or s.Error are called during SetUp(). Reset() runs after each but the last test to roll back changes a test made to the environment.
SetUp()Reset()Reset()Reset()TearDown()Reset() should be a light-weight and idempotent operation. If it fails (returns a non-nil error), framework falls back to TearDown() and SetUp() to completely restart the fixture. Tests should not leave too much change on system environment, so that the next Reset() does not fail.
Fixtures also have PreTest() and PostTest() methods, which run before and after each test. They get called with testing.FixtTestState with which you can report errors as a test. It's a good place to set up logging for individual test for example.
For details of these fixture lifecycle methods, please see the GoDoc testing.FixtureImpl.
Each test can declare its fixture by setting testing.Test.Fixture an fixture name. The fixture's SetUp() returns an arbitrary value that can be obtained by calling s.FixtValue() in the test. Because s.FixtValue() returns an interface{}, type assertion is needed to cast it to the actual type. However, s.FixtValue() will always return nil when local tests/fixtures try to access values from remote fixtures because Tast does not know the actual types of fixture values to deserialize them. Therefore, there is another function s.FixtFillValue(v, any) which requires user to pass in a pointer, and it will store the deserialized result in the value pointed to by pointer.
Fixtures are composable. A fixture can declare its parent fixture with testing.Fixture.Parent. Parent‘s SetUp() is executed before the fixture’s SetUp() is executed, parent‘s TearDown() is executed after the fixtures’s TearDown(), and so on. Fixtures can use the parent's value in the same way tests use it.
Local tests/fixtures can depend on a remote fixture if they live in test bundles with the same name (e.g. local cros and remote cros).
Fixtures are registered by calling testing.AddFixture with testing.Fixture struct in init(). testing.Fixture.Name specifies the fixture name, testing.Fixture.Impl specifies implementation of the fixture, testing.Fixture.Parent specifies the parent fixture if any, testing.Fixture.SetUpTimeout and the like specify methods' timeout, and the other fields are analogous to testing.Test.
Fixtures can be registered outside bundles directory. It's best to initialize and register fixtures outside bundles if it is shared by tests in multiple categories.
Rather than calling chrome.New at the beginning of each test, tests can declare that they require a logged-in Chrome instance by setting testing.Test.Fixture to “chromeLoggedIn” in init(). This enables Tast to just perform login once and then share the same Chrome instance with all tests that specify the fixture. See the chromeLoggedIn documentation for more details, and example.ChromeFixture for a test using the fixture.
If you want a new Chrome fixture with custom options, call testing.AddFixture from chrome/fixture.go with different options, and give it a unique name.
On designing composable fixtures, understanding the theory behind fixtures might help.
Let us think of a space representing all possible system states. A fixture‘s purpose is to change the current system state to be in a certain subspace. For example, the fixture chromeLoggedIn‘s purpose is to provide a clean environment similar to soon after logging into a Chrome session. This can be rephased that there’s a subspace where “the system state is clean similar to soon after logging into a Chrome session” and the fixture’s designed to change the system state to some point inside the subspace.
To denote these concepts a bit formally: let U be a space representing all possible system states. Let f be a function that maps a fixture to its target system state subspace. Then, for any fixture X, f(X) ⊆ U. Note that f(F) is a subspace of U, not a point in U; there can be some degrees of freedom in a resulting system state.
A fixture's property is as follows: if a test depends on a fixture F directly or indirectly, it can assume that the system state is in f(F) on its start. This also applies to fixtures: if a fixture depends on a fixture F directly or indirectly, it can assume that the system state is in f(F) on its setup.
To fulfill this property, all fixtures should satisfy the following rule: if a fixture X has a child fixture Y, then f(X) ⊇ f(Y). Otherwise, calling Y's reset may put the system state to one not accepted by X, failing to fulfill the aforementioned property.
Similar to tests, parameterized fixtures are also supported. They can be used to define multiple similar fixtures with different features/options.
To parameterize a fixture, specify a slice of testing.FixtureParam in the Params field on fixture registration. If Params is non-empty, testing.AddFixture will expand the fixture into one or more tests corresponding to each item in Params by merging testing.Fixture and testing.FixtureParam.
Here is an example with normal parameterized fixtures and an example of a test using the normal parameterized fixtures.
FixtureParam should be a literal in general since fixture registration should be declarative. However, a fixture may need to support a different combination of features or options. For fixtures that support a lot of different features/options, fixture writers and users may have a hard time to figure out what parameters have been declared for a particular combination of features/options. To ease the effort for composing a fixture with a lot of features/options, factory-like functions are allowed to be used for FixtureParam declaration and reference to the parameterized fixture. Even with the exception, the fixture writers and test writers should always make sure the function always returns the same fixture name for the same build. Otherwise, the generated metadata will produce the wrong data and cause issues in the test executions. Also, the factory function should be simple and the name of the parameter should allow users to identify the combination of features/options easily.
Here is an example of parameterized fixtures with a factory, and an example of a test use a parameterized fixture with a factory.
Preconditions, predecessor of fixtures, are not recommended for new tests.
Tast remote root fixture which allows test writers to add hooks that affect every test (except test with Pre-Condition) in a Tast session. Those hooks can perform different functions such as checking DUT state between tests.
All tests and fixtures will depend on the remote root fixture directly or indirectly. The Setup phase of the remote root fixture which will be run at the beginning of the execution of a Tast bundle. It can allow different hooks to be run during different phases of the remote root fixture: Setup, PreTest, Reset, PostTest, and TearDown. Hook writers can do something applicable to all tests.
Tast's fixture tree is constructed separately while running tests for different bundles. Since a Tast session may include tests from different bundles, the setup for the root fixture may be invoked multiple times during a Tast session. Also, if the retries flag of Tast is set to be greater than zero, Tast will need to re-construct and run the fixture tree. Hence, the setup of the root fixture will be invoked again.
The root fixture will allow us to add framework level setups and monitors. For example, we can start servod on the labstation at the beginning of the running a bundle so that all tests in the bundle can use servo. It will also allow test writers to create hooks that affect every test in a Tast session. It will allow users to create a customized environment for each test suite.
All hooks should be in the hooks directory. Hooks can depend on remote and common Tast libraries. On the other hand, tests or fixtures should not have any dependencies on hooks.
The function addHook for users to add a hook to the root fixture. It is intentionally not to export this function’s symbol so that all hooks and the remote root fixture will be created in the same package. This will allow Tast team to have better control what hooks are allowed.
HookImpl is an interface that users have to implement so that the root fixture can invoke the corresponding action in each step of a fixture process.
type HookImpl interface { // SetUp will be called during root fixture Setup. SetUp(ctx context.Context, s *HookState) error // Reset will be called during root fixture Reset. Reset(ctx context.Context) error // PreTest will be called during root fixture PreTest. PreTest(ctx context.Context, s *HookTestState) error // PostTest will be called during root fixture PostTest. PostTest(ctx context.Context, s *HookTestState) error // TearDown will be called during root fixture TearDown. TearDown(ctx context.Context, s *HookState) error }
To avoid non-deterministic order of the execution, a variable “orderHooks” will be used to maintain the order of execution. To add a new hook, the author needs to add the name of the hook to the list. Otherwise, the hook will not be executed. The order of the execution On the other hand, PostTest and TearDown will be executed in the reverse order of this list.
var orderedHooks []string
HookState includes certain fixture state information that is available for each hook to be used during Setup and Teardown.
HookTestState includes fixture state information that is available for each hook to be used during PreTest and PostTest.
Since the remote root fixture is a remote fixture, all hooks are running on the host side. Some hooks may need to use GPRC services that are running on the DUT to perform actions on the DUT. Users can add ServiceDeps to the root fixture if their hooks need GPRC services.
Since all hooks affect the entire Tast session, they affect all tests. Therefore, any additions and changes of hooks will require approval from a member of the Tast team. Also, at least one expert reviewer will be needed for each hook related CL.
Here is a hook example.
All hooks should be added in the “init” function of a file.
func init() {
addHook(&Hook{
Name: "exampleHook",
Desc: "Demonstrate how to use hook",
Contacts: []string{"[email protected]"},
BugComponent: "b:1034522",
Impl: &testhook{},
})
}
In the hook example, the hook will use a Tast global variable to determine whether actions should be performed. This variable is for this example only.
var shouldRun = testing.RegisterVarString(
"hooks.example.shouldrun",
"",
"A variable to decide whether example hook should run",
)
The implementation of a hook needs to include five functions: Setup, PreTest, PostTest, Reset, and Teardown. In this example, all five functions will only perform action if the variable hooks.example.shouldrun is set to true.
type testhook struct { shouldRun bool } // SetUp will be called during root fixture Setup. func (h *testhook) SetUp(ctx context.Context, s *HookState) error { h.shouldRun = shouldRun.Value() == "true" if !h.shouldRun { return nil } testing.ContextLog(ctx, "testhook Setup") return nil } // Reset will be called during root fixture Reset. func (h *testhook) Reset(ctx context.Context) error { if !h.shouldRun { return nil } testing.ContextLog(ctx, "testhook Reset") return nil } // PreTest will be called during root fixture PreTest. func (h *testhook) PreTest(ctx cont