Test Coverage API #123713
Description
📣 2024-03 update: test coverage support has been finalized. You can find the finished, usable version of the API in vscode.d.ts and a guide in our docs.
Original discussion has been preserved below.
Notes researching of existing formats:
Clover
- Contains timestamp and total lines of code, files, classes, coverage numbers (see below)
- Also can include complexity information
- Lines are covered as XML simply
<line num="24" count="5" type="stmt"/> - Conditionals are represented as
<line num="12" count="0" type="cond" truecount="0" falsecount="3"/>. Either truecount/falsecount being 0 indicates incomplete branch coverage.
Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows:
TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100% where BT - branches that evaluated to "true" at least once BF - branches that evaluated to "false" at least once SC - statements covered MC - methods entered B - total number of branches S - total number of statements M - total number of methods
gcov
Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php
- Organized per test, which are associated with one or more files. Indicates functions and how many times each function was called.
- Lines have number of times they were hit,
DA:<line number>,<execution count>[,<checksum>] - Has similar branch coverage, but has indexed branches instead of true/false. A branch with 0 taken is uncovered.
BRDA:<line number>,<block number>,<branch number>,<taken>, declared multiple times for line, one for each branch
- No timestamp
cobertura
- General coverage information/count
- Number of times each line was hit.
- Has conditions in a slightly less-strict way,
<line number="21" hits="58" branch="true" condition-coverage="50% (1/2)"/> - Normal lines look like
<line number="20" hits="5" branch="false"/> - Has timestamp and metadata
- Has method call count (everything is organized in classes)
This results in the following API. The TestCoverageProvider is given as an optional provider on the managed TestRun object. This is similar to the SourceControl interface in vscode.d.ts. The provider is then pretty standard; it'll only be examined when the run finishes, so all its data is static.
It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.
I refer to the atomic unit of coverage as "statement coverage". The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it's called statement coverage, but it can be used as line coverage.
The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don't like the method names on the TestCoverageProvider, but have not thought of better ones yet.
declare module 'vscode' {
// https://github.com/microsoft/vscode/issues/123713
export interface TestRun {
/**
* Test coverage provider for this result. An extension can defer setting
* this until after a run is complete and coverage is available.
*/
coverageProvider?: TestCoverageProvider;
// ...
}
/**
* Provides information about test coverage for a test result.
* Methods on the provider will not be called until the test run is complete
*/
export interface TestCoverageProvider<T extends FileCoverage = FileCoverage> {
/**
* Returns coverage information for all files involved in the test run.
* @param token A cancellation token.
* @return Coverage metadata for all files involved in the test.
*/
// @API - pass something into the provide method:
// (1) have TestController#coverageProvider: TestCoverageProvider
// (2) pass TestRun into this method
provideFileCoverage(token: CancellationToken): ProviderResult<T[]>;
/**
* Give a FileCoverage to fill in more data, namely {@link FileCoverage.detailedCoverage}.
* The editor will only resolve a FileCoverage once, and only if detailedCoverage
* is undefined.
*
* @param coverage A coverage object obtained from {@link provideFileCoverage}
* @param token A cancellation token.
* @return The resolved file coverage, or a thenable that resolves to one. It
* is OK to return the given `coverage`. When no result is returned, the
* given `coverage` will be used.
*/
resolveFileCoverage?(coverage: T, token: CancellationToken): ProviderResult<T>;
}
/**
* A class that contains information about a covered resource. A count can
* be give for lines, branches, and functions in a file.
*/
export class CoveredCount {
/**
* Number of items covered in the file.
*/
covered: number;
/**
* Total number of covered items in the file.
*/
total: number;
/**
* @param covered Value for {@link CovereredCount.covered}
* @param total Value for {@link CovereredCount.total}
*/
constructor(covered: number, total: number);
}
/**
* Contains coverage metadata for a file.
*/
export class FileCoverage {
/**
* File URI.
*/
readonly uri: Uri;
/**
* Statement coverage information. If the reporter does not provide statement
* coverage information, this can instead be used to represent line coverage.
*/
statementCoverage: CoveredCount;
/**
* Branch coverage information.
*/
branchCoverage?: CoveredCount;
/**
* Function coverage information.
*/
functionCoverage?: CoveredCount;
/**
* Detailed, per-statement coverage. If this is undefined, the editor will
* call {@link TestCoverageProvider.resolveFileCoverage} when necessary.
*/
detailedCoverage?: DetailedCoverage[];
/**
* Creates a {@link FileCoverage} instance with counts filled in from
* the coverage details.
* @param uri Covered file URI
* @param detailed Detailed coverage information
*/
static fromDetails(uri: Uri, details: readonly DetailedCoverage[]): FileCoverage;
/**
* @param uri Covered file URI
* @param statementCoverage Statement coverage information. If the reporter
* does not provide statement coverage information, this can instead be
* used to represent line coverage.
* @param branchCoverage Branch coverage information
* @param functionCoverage Function coverage information
*/
constructor(
uri: Uri,
statementCoverage: CoveredCount,
branchCoverage?: CoveredCount,
functionCoverage?: CoveredCount,
);
}
// @API are StatementCoverage and BranchCoverage etc really needed
// or is a generic type with a kind-property enough
/**
* Contains coverage information for a single statement or line.
*/
export class StatementCoverage {
/**
* The number of times this statement was executed, or a boolean indicating
* whether it was executed if the exact count is unknown. If zero or false,
* the statement will be marked as un-covered.
*/
executed: number | boolean;
/**
* Statement location.
*/
location: Position | Range;
/**
* Coverage from branches of this line or statement. If it's not a
* conditional, this will be empty.
*/
branches: BranchCoverage[];
/**
* @param location The statement position.
* @param executed The number of times this statement was executed, or a
* boolean indicating whether it was executed if the exact count is
* unknown. If zero or false, the statement will be marked as un-covered.
* @param branches Coverage from branches of this line. If it's not a
* conditional, this should be omitted.
*/
constructor(executed: number | boolean, location: Position | Range, branches?: BranchCoverage[]);
}
/**
* Contains coverage information for a branch of a {@link StatementCoverage}.
*/
export class BranchCoverage {
/**
* The number of times this branch was executed, or a boolean indicating
* whether it was executed if the exact count is unknown. If zero or false,
* the branch will be marked as un-covered.
*/
executed: number | boolean;
/**
* Branch location.
*/
location?: Position | Range;
/**
* Label for the branch, used in the context of "the ${label} branch was
* not taken," for example.
*/
label?: string;
/**
* @param executed The number of times this branch was executed, or a
* boolean indicating whether it was executed if the exact count is
* unknown. If zero or false, the branch will be marked as un-covered.
* @param location The branch position.
*/
constructor(executed: number | boolean, location?: Position | Range, label?: string);
}
/**
* Contains coverage information for a function or method.
*/
export class FunctionCoverage {
/**
* Name of the function or method.
*/
name: string;
/**
* The number of times this function was executed, or a boolean indicating
* whether it was executed if the exact count is unknown. If zero or false,
* the function will be marked as un-covered.
*/
executed: number | boolean;
/**
* Function location.
*/
location: Position | Range;
/**
* @param executed The number of times this function was executed, or a
* boolean indicating whether it was executed if the exact count is
* unknown. If zero or false, the function will be marked as un-covered.
* @param location The function position.
*/
constructor(name: string, executed: number | boolean, location: Position | Range);
}
export type DetailedCoverage = StatementCoverage | FunctionCoverage;
}