crates/create_update_cl.md

# Updating Rust crates used by Chromium

This document describes how Chromium updates crates.io Rust crates that Chromium
depends on.

## Staffing

We have a
[weekly rotation](https://goto.google.com/chromium-crates-update-rotation) of
Google engineers responsible for creating and landing CLs that update Rust
crates.

Google engineers can join the rotation by emailing
[chrome-safe-coding@google.com](mailto:chrome-safe-coding@google.com).

## Initial setup

The "Rust: periodic update of 3rd-party crates" rotation requires access to an
up-to-date Chromium repo.  One way to start a shift is to run `git fetch`,
`git checkout origin/main`, and `gclient sync` (but other workflows should also
work - e.g. ones based on `git-new-workdir`).

## Checking the state of the world

Before creating a CL stack, check for open CLs with the [`cratesio-autoupdate`
tag](https://chromium-review.googlesource.com/q/hashtag:%22cratesio-autoupdate%22+(status:open%20OR%20status:merged)).
Such CLs tend to conflict, so coordinate with owners of any open CLs.

You may also check a doc with notes from previous rotations, where we may note
known issues and their workarounds.  See (Google-internal, sorry):
https://docs.google.com/document/d/1S7gsrJFsgoU5CH0K7-X_gL55zIIgd6UsFpCGrJqjdAg/edit?usp=sharing

## Automated step: `create_update_cl.py`

The first actual step of the rotation is running `create_update_cl.py`. You must
invoke it from within the `src/` directory of a Chromium repository checkout,
and it depends on `depot_tools` and `git` being present in the `PATH`.

```sh
$ cd ~/chromium/src  # or wherever you have your checkout
$ tools/crates/create_update_cl.py auto
```

In `auto` mode, it runs `gnrt update` to discover crate updates and then for
each update creates a new local git branch (and a Gerrit CL unless invoked with
`--no-upload`). Each branch contains an update created by `gnrt update <old
crate id>`, `gnrt vendor`, and `gnrt gen`. Depending on how many crates are
updated, the script may need 10-15 minutes to run.

The script should Just Work in most cases, but sometimes it may fail when
dealing with a specific crate update.  See [Recovering from script
failures](#recovering-from-script-failures) below for what to do when that
happens.

Before the auto-generated CLs can be landed, you will need to get an LGTM from
`//third_party/rust/OWNERS`.  A review checklist can be found at
`//third_party/rust/OWNERS-review-checklist.md`.

## New transitive dependencies

Notes from `//third_party/rust/OWNERS-review-checklist.md` apply:

* The dependency will need to go through security review.
* An FYI email should be sent to
  [chrome-atls-discuss@google.com](mailto:chrome-atls-discuss@google.com)
  in order to record the addition.

### Optional: Adding the transitive dependency in its own CL

If the new crate is non-trivial, it's possible to split the
additional crate into its own CL, however then it will default to global
visibility and allowing non-test use.
* `gnrt add` and `gnrt vendor` can add the dependency to a fresh checkout.
* Mark the crate as being for third-party code only by setting
  `allow_first_party_usage` to `false` for the crate in
  `third_party/rust/chromium_crates_io/gnrt_config.toml`.
* If the crates making use of the transitive dependency are only allowed
  in tests, then set `group = 'test'` for the crate in
  `third_party/rust/chromium_crates_io/gnrt_config.toml`. This reduces
  the level of security review required for the library.
* `gnrt gen` will then generate the GN rules.
* Rebase the roll CL on top of the changes to make sure the choices made above
  are correct. `gn gen` will fail in CQ if the crate was placed in the `'test'`
  group but needs to be visible outside of tests.

## Potential additional steps

* The `create_update_cl.py` script may stop early if it detects that `gnrt
  vendor` or `gnrt gen` have reported any warnings or errors (e.g. a "License
  file not found for crate foo" warning).  In this case, manual intervention is
  needed to finish the update CL.  It's probably best to finish and land the CLs
  created so far before trying to restart the script in order to create the
  remaining CLs.

## Landing the CL

Other than the above, the CL can go through the normal, plain-vanilla, manual
review and landing process.

1. `git cl upload`
1. Get a review from one of `//third_party/rust/OWNERS`
1. Land the CL using CQ+2

## Checking for new major versions

Note that `create_update_cl.py auto` will by default only handle minor version
updates (e.g.  123.1 => 123.2, or 0.123.1 => 0.123.2).  Major version changes
(e.g. 1.0 => 2.0, which may include breaking API changes and other breaking
changes) need to be handled separately - this section describes what to do.

### Detecting available major version updates

As part of the rotation, one should attempt to check for new major versions of
_direct_ Chromium dependencies (i.e. dependencies directly listed in
`third_party/rust/chromium_crates_io/Cargo.toml`).  To discover direct _and_
transitive dependencies with a new major version, you can use the command below
(running it in the final update CL branch - after all the minor version
updates):

```sh
$ tools/crates/run_gnrt.py update -- --verbose --dry-run
...
   Unchanged serde_json_lenient v0.1.8 (latest: v0.2.0)
   Unchanged syn v1.0.109 (latest: v2.0.53)
...
```

### Major version update: Workflow A: Single update CL

If updating to a new major version doesn't require lots of Chromium changes,
then it may be possible to land the update in a single CL.  This is typically
possible when the APIs affected by the major version's breaking change either
weren't used by Chromium, or were used only in a handful of places.

**Warning**: Sometimes a new major version may be API compatible, but may
introduce breaking changes in the _behavior_ of the existing APIs.

To update:

1. `tools/crates/create_update_cl.py auto -- some_crate_name --breaking`
1. Follow the manual steps from the minor version update rotation for
   review, landing, etc.

### Major version update: Workflow B: Incremental transition

When lots of first-party code depends on the old major version, then the
transition to the new major version may need to be done incrementally.  In this
case the transition can be split into the following steps:

1. Open a new bug to track the transition
    - TODO: Figure out how to tag/format the bug to make it easy to discover
      in future rotations
1. Land the new major version, so that the old and the new versions coexist.
   To do this follow the process for importing a new crate as described in
   [`docs/rust.md`](../../docs/rust.md#importing-a-crate-from-crates_io)
   (i.e. edit `Cargo.toml` to add the new version, run `gnrt vendor`, and so
   forth).
1. Incrementally transition first-party code to the new major version
1. Remove the old major version.  To do this follow a similar process as above
   (i.e. edit `Cargo.toml` to remove the old version, run `gnrt vendor`, and so
   forth).  Any leftover files in `//third_party/rust/<crate>/<old epoch>`
   should also be removed.

Note that the following `Cargo.toml` syntax allows two versions of a crate to
coexist:

```toml
[dependencies.serde_json_lenient_old_epoch]
package = "serde_json_lenient"
version = "0.1"

[dependencies.serde_json_lenient]
version = "0.2"
```

## Other ways to use `create_update_cl.py`

### `auto` mode

Extra arguments passed to `create_update_cl.py auto` end up being passed to
`cargo update`.  For a complete list of available options, see
[Cargo documentation here](https://doc.rust-lang.org/cargo/commands/cargo-update.html#update-options)),
but the most common scenarios are covered in the sections below.

#### Updating all crates during the weekly rotation

`tools/crates/create_update_cl.py auto` with no extra arguments will attempt to
discover **minor** version updates for **all** crates that Chromium depends on
and for their transitive dependencies.

#### Updating the minor version of a single crate

`tools/crates/create_update_cl.py auto -- some_crate_name` can be used to
trigger a **minor** version update of a single crate.

#### Updating the major version of a single crate

`tools/crates/create_update_cl.py auto -- some_crate_name --breaking` can be
used to trigger a **major** version update of a single crate

### `manual` mode

For maximal control, the script can be used in `manual` mode:

1. Prepare `Cargo.toml` change:
    1. `git checkout origin/main`
    1. `git checkout -b manual-update-of-foo`
    1. Edit `third_party/rust/chromium_crates_io/Cargo.toml` to change the crate
       version of the crate (or crates) you want to update.
       **Important**: Do not edit `Cargo.lock` (e.g. don't run `gnrt vendor`
       etc.).
    1. `git add third_party/rust/chromium_crates_io/Cargo.toml`
    1. `git commit -m "Manual edit of Cargo.toml"`
    1. `git cl upload -m "Manual edit of Cargo.toml" --bypass-hooks --skip-title --force`
1. Run the helper script as follows:
   `tools/crates/create_update_cl.py manual
   --title "Roll foo crate to new version X"`
    - This will run `gnrt vendor` to discover and execute updates that were
      requested by the manual edits of `Cargo.toml` in the previous steps.
    - This will automatically add more details to the CL description
    - To make the review easier, one of the patchsets covers just the path
      changes.  For example - see [the delta here](https://crrev.com/c/5445719/2..7).

<a id="recovering-from-script-failures"></a>
## Recovering from script failures

Sometimes the `create_update_cl.py` script will fail when dealing with
a specific crate update.  The general workflow in this case is to
1) fix the issue in a separate CL, and 2) restart the tool from the middle
by using `--upstream-branch` that points to the last successful update branch
(or to the fix CL) rather than defaulting to `origin/main`.

Examples of a few specific situations that may lead to script failure:

* An update brought in a new crate, but `gnrt` didn't recognize new crate's
  license kind or license file.  In that case a prerequisite CL needs to be
  landed first, teaching `gnrt` about the new license kinds/files
  ([in readme.rs](https://source.chromium.org/chromium/chromium/src/+/main:tools/crates/gnrt/lib/readme.rs;l=264-290;drc=c838bc6c6317d4c1ead1f7f0c615af353482f2b3)).
  You can see [an example CL with such a fix](https://crrev.com/c/6219211).
* Patches from `//third_party/rust/chromium_crates_io/patches/` no longer
  apply cleanly to the new version of a crate.  In that case the crate update CL
  needs to 1) first update the patches, and then 2) update the crate as usual.
  This is not very well supported by the script... But something like this
  should work:
    - Checkout a new branch:
        ```sh
        $ git checkout rust-crates-update--last-successful-update
        $ git checkout -b fix-patches-for-foo
        $ git branch --set-upstream-to=rust-crates-update--last-successful-update
        ```
    - Fix the patches and upload as a temporary / throw-away CL
      (this CL can't be landed on its own - it needs to be combined
      with the actual update CL):
        ```sh
        $ # Fix the patches
        $ git commit -a -m ...
        $ git cl upload
        ```
    - Restart the script (the CL created by the script can't be landed
      as-is / on its own - it needs to be combined with the fixed patches
      in the step below) with `--upstream-branch` parameter:
        ```sh
        $ tools/crates/create_update_cl.py auto \
            --upstream-branch=fix-patches-for-foo \
            -- name-of-failed-crate
        ```
    - Combine the branches:
        ```sh
        $ git map-branches -v # to orient yourself
        $ git checkout rust-crates-update--new-successful-update
        $ git branch --set-upstream-to=rust-crates-update--last-successful-update
        $ git cl upload -m Rebasing... # --bypass-hooks as needed
        ```
* `//third_party/rust/chromium_crates_io/gnrt_config.toml` needs to be updated
  to work with a new crate version.  The same workflow should work as for fixing
  `//third_party/rust/chromium_crates_io/patches/` (see the item above).