docs/transcripts/wuwt-e05-build-gn.md

# What’s Up With BUILD.gn

This is a transcript of [What's Up With
That](https://www.youtube.com/playlist?list=PL9ioqAuyl6ULIdZQys3fwRxi3G3ns39Hq)
Episode 5, a 2023 video discussion between [Sharon (yangsharon@chromium.org)
and Nico (thakis@chromium.org)](https://www.youtube.com/watch?v=NcvJG3MqquQ).

The transcript was automatically generated by speech-to-text software. It may
contain minor errors.

---

Building Chrome is an integral part of being a Chrome engineer. What actually
happens when you build Chrome, and what exactly happens when you run those
build commands? Today, we have Nico, who was responsible for making Ninja the
Chrome default build system, to tell us more.

Notes:
- https://docs.google.com/document/d/1iDFqA3cZAUo0TUFA69cu5wEKL4HjSoIGfcoLIrH3v4M/edit

---

00:00 SHARON: Hello, and welcome to "What's Up With That," the series that
demystifies all things Chrome. I'm your host, Sharon, and today, we're talking
about building Chrome. How do you go from a bunch of files on your computer to
running a browser? What are all the steps involved? Our special guest today is
Nico. He's responsible for making Ninja, the Chrome default build system, and
he's worked on Clang and all sorts of areas of the Chrome build. If you don't
know what some of those things are, don't worry. We'll get into it. Welcome,
Nico.

00:29 NICO: Hello, Sharon, and hello, internet.

00:29 SHARON: Hello. We have lots to cover, so let's get right into it. If I
want to build Chrome at a really quick overview, what are all the steps that I
need to do?

00:41 NICO: It's very easy. First, you download `depot_tools` and add that to
your path. Then you run fetch Chromium. Then you type `cd source`, run `gclient
sync`, `gn gen out/GN`, and `ninja -C out/GN chrome`. And that's it.

00:53 SHARON: Wow. Sounds so easy. All right. We can wrap that up. See you guys
next time. OK. All right. Let's take it from the start, then, and go over in
more detail what some of those things are. So the first thing you mentioned is
`depot_tools`. What is that?

01:11 NICO: `depot_tools` is just a collection of random utilities for - like,
back in the day, for managing subversion repositories, nowadays for pulling
things from git. It contains Ninja and GN. Just adds a bunch of stuff to your
path that you need for working on Chrome.

01:25 SHARON: OK. Is this a Chrome-specific thing, or is this used elsewhere,
too?

01:33 NICO: In theory, it's fairly flexible. In practice, I think it's mostly
used by Chromium projects.

01:39 SHARON: OK, all right. And there, you mentioned Ninja and GN. And for
people - I think most people who are watching this have built Chrome at some
point. But what is the difference between Ninja and GN? Because you have your
build files, which are generally called Build.gn, and then you run a command
that has Ninja in it. So are those the same thing? Are those related?

01:57 NICO: Yes. So GN is short for Generate Ninja. So Ninja is a build system.
It's similar to Make. It basically gets a list of source files and a list of
build outputs. And then when you run Ninja, Ninja figures out which build steps
do I have to run, and then it runs them. So it's kind of like Make but simpler
and faster. And then GN - and Ninja doesn't have any conditionals or anything,
so GN is - just a built - it describes the build. And then it generates Ninja
files.

02:34 SHARON: OK.

02:34 NICO: So if you want to do, like, add these files only if you're building
for Windows, this is something you can do, say, in GN. But then it only
generates a Windows-specific Ninja file.

02:46 SHARON: All right. And in terms of when you mention OS, so there's a
couple places that you can specify different arguments for how you build
Chrome. So you have your gclient sync - sorry, your gclient file, and then you
have a separate args.gn. And in both of these places, you can specify different
arguments. And for example, the operating system you use - that can be
specified in both places. There's an OS option in both. So what is the purpose
of the gclient file, and what is the purpose of the args.gn file?

03:25 NICO: Yes. So gclient reads the steps file that is at the root of the
directory, and the DEPS file basically specifies dependencies that Chrome pulls
in. It's kind of similar to git submodules, but it predates git, so we don't
use git submodules also for other reasons. And so if you run gclient sync, that
reads the DEPS file, the Chrome root, and that downloads a couple hundred
repositories that Chrome depends on. And then it executes a bunch of so-called
hooks, which are just Python scripts, which also download a bunch of more
stuff. And the hooks and the dependencies are operating system dependent, so
gclient needs to know the operating system. But the build also needs to know
the operating system. And GN args are basic things that are needed for the
builds. So the OS is something that's needed in both places, but many GN args
gclient doesn't need to know about. For example, if you enable DCHECKs, like
Peter discussed a few episodes ago, that's a GN-only thing.

04:26 SHARON: All right. That sounds good. So let's see. When you actually -
OK. So when you run Chrome and you - say you build Chrome, right? A typical
example of a command to do that would be, say, `autoninja -C out/default
content`, right? And let's just go through each part of that and say what each
of those things is doing and what happens there. Because I think that's just an
example from one of the starter docs. That's just the copy and paste command
that they give you. So autoninja seems like it's based on Ninja. What is the
auto they're doing for us?

05:15 NICO: Yeah. So autoninja is also one of the things that's just
`depot_tools`. It's a very - or it used to be a very thin wraparound Ninja. Now
it's maybe a little thicker, but it's optional. You don't have to use autoninja
if you don't want to. But what it does is basically - like, it helps - So
Chrome contains a lot of code. So we have this system called Goma, which can
run all the C++ compilations in a remote data center. And if you do use the
system, then you want to build with a very high build parallelism. You want to,
say, `-j 1000` or what and run, like, a thousand bit processes in parallel. But
if you're building locally, you don't want to do that. So what autoninja
basically does - it looks at your args.gn file, sees if you have enabled Goma,
and if so, it runs Ninja with many processes, and else, it runs it with just
one process per core, or something like that. So that's originally all that
autoninja does. Nowadays, I think it also uploads a bunch of stuff. But you can
just run which autoninja, and that prints some path, and you can just open that
in the editor and read it. I think it's still short enough to fairly quickly
figure out what it does.

06:17 SHARON: OK. What does `-C` do? Because I think I've been using that this
whole time because I copied and pasted it from somewhere, and I've just always
had it.

06:28 NICO: It says - it changes the current directory where Ninja runs, like
in Make. So it basically says, change the current directory to out/GN, or
whatever you build directory is, and then run the build from there. So for
Chrome, the build always - the current directory during the build is always the
build directory. And then Ninja looks for a file called build.ninja in the
current directory, so GN writes build.ninja to out/GN, or whatever you build
directory is. And then Ninja finds it there and reads it and does its thing.

06:57 SHARON: All right. So the next part of this would be out/default, or out
slash something else. So what are out directories, and how do we make use of
them?

07:11 NICO: An out directory - it's just a build directory. That's where all
the build artifacts go to, all the generated objects files, executables, random
things that are generated during the build. So it can be any directory, really.
You can make up any directory name that you like. You can build your Chrome in,
I don't know, fluffy/kitten, or whatever. But I think most people use out just
because it's in the global `.gitignore` file already. Then you want to use
something that's two directories deep so that the path from the directory to
the source is always `../..`. And that makes sure that this is deterministic.
We try to have a so-called deterministic build, where you get exactly the same
binary when you build Chrome at the same revision, independent of the host
machine, more or less. And the path from the build directory to the source file
is something that goes into debug info. So if you want to have the same build
output as everyone else, you want a build directory path that's two directories
deep. And the names of those two directories doesn't really matter. So what
some people do is they use out/debug for the debug builds and out/release for
their release builds. But it's really up to you.

08:26 SHARON: Right. Other common ones are, like, yeah. ASan is a common one,
different -

08:33 NICO: Right.

08:33 SHARON: OSes. Right. So you mentioned having a deterministic build. And
assuming you're on the same version of Chrome, at the same checkout,
tip-of-tree, or whatever as someone else, I would have expected that all of the
builds are just deterministic, but maybe that's because of work that people
like you and the build team have done. But what are things that could cause
that to be nondeterministic? Because you have all the same files. Where is the
actual nondeterminism coming from? Or is it just different configurations and
setups you have on your local machine?

09:09 NICO: Yeah, that's a great question. I always thought this would be very
easy to - but turns out it mostly isn't. We wrote a very long blog post that we
can link to it from the show notes about this. But there's many things that can
go wrong. Like for example, in C++, there's the preprocessor macro `__DATE__`,
which embeds the current date into the build output. So if you do that, then
you're time dependent already. By default, I think you end up with absolute
paths to everything in debug information. So if you build under
`/home/sharon/blah`, then that's already different from all the people who are
not called Sharon. Then there's - we run tools as part of the build that
produce output. For example, the protobuf compiler or whatnot. And so if that
binary iterates over some map, some hash map, and that doesn't have
deterministic iteration order, then the output might be different. And there's
a long, long, long, long, long list of things. Making the build deterministic
was a very big project, and there's still a few open things.

10:08 SHARON: OK, cool. So I guess it's - yeah, it's not true nondeterminism,
maybe, but there's enough factors that go into it that to a typical person
interacting with it, it does seem -

10:21 NICO: Yeah, but there's also true nondeterminism. Like, every now and
then, when we update the compiler, the compiler will write different object
files on every run just because the compiler internally iterates about some -
over some hash map. And then we have to complain upstream, and then they fix
it.

10:34 SHARON: OK. Oh, wow. OK. That's very cool. Well, thank you for dealing
with this kind of stuff so people like us don't have to worry about it. OK. And
the last part of our typical build thing is content. So what is content in this
context? If you want to learn about content more in general, check out
episode 3. But in this case, what does that mean?

10:58 NICO: So just a build target. So I think people - at least I usually
build some executable. I usually build, I don't know, `base_unittests` or
`unit_tests` or Chrome or content shell or what. And it's just - so in the
Ninja files, there's basically - there's many, many lines that go, if you want
to build this file, you need to have these inputs and then run this command. If
you want to build this file, instead, you need these other files. You need to
run this other command. So for example, if you want to build `base_unittests`,
you need a couple thousand object files, and then you need to run the linkers,
what's in there. And so if you tell Ninja - the last thing you give it -
basically, it tells Ninja, what do you want to build? So if you say, `ninja -C
out/GN content_shell` or what, then Ninja is like, let's look at the line that
says `content_shell`. And then it checks - I need these files, so it builds all
the prerequisites, which usually means compiling a whole bunch of files. And
then it runs the final command and runs the linker. So Ninja basically decides
what it needs to do and then invokes other commands to do the actual work.

12:08 SHARON: OK, makes sense. So say I run the build - so say I built the
target Chrome, which is the one that actually is an executable, and that's
what - if you run that, the browser is built from it. So say I've built the
Chrome build target. How do I run that now?

12:31 NICO: Well, it's written - so normally, the thing you give to Ninja is
actually a file name. And the `-C` change current directory. So if you say, `-C
out/release chrome`, then this creates the file `out/release/chrome`. It just
creates that file in the out directory. So to run that, you just run
`out/release/chrome`, and hopefully it'll start up and work.

12:54 SHARON: Great. Sounds so easy. So you mentioned earlier something called
Goma, which had remote data centers and stuff. Is this something that's
available to people who don't work at Google, or is this one of the
Google-specific things? Because I think so far, everything mentioned is anyone,
anywhere can do all this. Is that the case with Goma, also?

13:14 NICO: Yeah. For the other things - so Ninja is actually something that
started in Chrome land, but that's been fairly widely adopted across the world.
Like, that's used by many projects. But yeah, Goma - I think it's kind of like
distcc. Like, it's a distributed compiler thing. I think the source code for
both the client and the server are open source. And we can link to that. But
the access to the service, I think, isn't public. So they have to work at
Google or at a partner company. I think we hand out access to a few partners.
And as far as I know, there's a few independent implementations of the
protocol, so other people also use something like Goma. But as far as I know,
these other services also aren't public.

13:53 SHARON: OK. Right. Yeah, because I think one of the main things is - I
mean, as someone who did an internship on Chrome, after, I was like, I'll
finish some of these remaining to do items once I go back to school, right? And
then I started to build Chrome on my laptop, just a decent laptop, but still a
laptop, and I was like, no, I guess I won't be doing that.

14:17 NICO: No, it's doable. You just need to be patient and strategic. Like, I
used to do that every now and then. You have to start the build at night, and
then when you get up, it's done. And if you only change one or two CC files,
it's reasonably fast. It's just, full builds take a very long time.

14:29 SHARON: Yeah, well, yeah. There was enough stuff going on that I was
like, OK. We maybe won't do this. Right. Going back to another thing you
mentioned is the compiler and Clang. So can you tell us a bit more about Clang
and how compiling fits into the build process?

14:50 NICO: Yeah, sure. I mean, compiling just means - almost all of Chrome
currently is written in C++, and compiling just means taking a CC file, like a
C++ file, and turning it into - turning that into an object file. And there are
a whole bunch of C++ compilers. And back in the day, we used to use many, many
different C++ compilers, and they're all slightly different, so that was a
little bit painful. And then the C++ language started changing more frequently,
like with C++ 11, 14, 17, 20, and so on. And so that was a huge drain on
productivity. Updating compilers was always a year-long project, and we had to
update, like, seven different compilers, one on Android, iOS, Windows, macOS,
Android, Fuchsia, whatnot. So over time, we moved to - we moved from using
basically the system compiler to using a hermetically built Clang that we
download as a gclient DEPS hook. So when you run gclient sync, that downloads a
prebuilt Clang binary. And we use that Clang binary to build Chrome on all
operating systems. So if one file builds for you on your OS, then chances are
it'll build on all the other OSes because it's built by the same compiler. And
that also enables things like cross builds, so you can build Chrome for Windows
on Linux if you want to because your compiler is right there.

16:11 SHARON: Oh, cool. All right. I didn't know that. Is there any reason,
historically, that Clang beat out these other compilers as the compiler of
choice?

16:24 NICO: Yes. So it's basically - I think when we looked at this - so Clang
is basically the native compiler on macOS and iOS, and GCC is kind of the
system compiler on Linux, I suppose. But Clang has always had very good GCC
compatibility. And then on Windows, the default choice is Visual Studio. And we
still want to link against the normal Microsoft library, so we need a compiler
that's ABI-compatible with the Microsoft ABI. And GCC couldn't do that. And
Clang also couldn't do that, but we thought if we teach Clang to do that, then
Clang basically can target all the operating systems we care about. And so we
made Clang work on Windows, also with others. But there was a team funded by
Chrome that worked on that for a few years. And also, Clang has pretty good
tooling interface. So for code search, we also use Clang. So we now use the
same code to compile Chrome and to index Chrome for code search.

17:28 SHARON: Oh, cool. I didn't know that either, so very interesting. OK.
We're just going to keep going back. And as you mention more things, we'll
cover that, and then go back to something you previously mentioned. So next on
the list is gclient sync. So I think for everyone who's ever worked on Chrome,
ever, especially at the start, you're like, I'll build Chrome. You build your
target, and you get these weird errors. And you look at it, and you think, oh,
this isn't some random weird spot that I definitely didn't change. What's going
on? And you ask a senior team member, and they say to you, did you run gclient
sync? And you're like, oh, I did not. And then you run it, and suddenly, things
are OK. So what else is going - you mentioned a couple of things that happen.
So what exactly does gclient sync do?

18:13 NICO: Yeah. So as I - that's this file at the source root called DEPS,
D-E-P-S, all capital letters. And when you update - if you git pull the Chrome
repository, then that also updates the DEPS file. And then this DEPS file
contains a long list of revisions of dependent projects. And then when you run
gclient sync, it basically syncs all these other git repositories that are
mentioned in the DEPS file. And after that, it runs so-called hooks, which like
do things download a new Clang compiler and download a bunch of other binaries
from something called the CIPD, for example, GN. But yeah, basically makes sure
that all the dependencies that are in Chrome but that aren't in the Chrome
repository are also up to date. That's what it does.

19:06 SHARON: OK. Do you have a rough ballpark guess of how many dependencies
that includes?

19:13 NICO: Its operating system dependent. I think on Android we have way
more, but it's on the order of 200. Like, 150 to 250.

19:25 SHARON: Sounds like a lot. Very cool. OK. In terms of - speaking of other
dependencies, one of the top-level directories in Chrome is `//third_party`,
and that seems in the same kind of direction. So how does stuff in
`//third_party` work in terms of building? Can you just build them as targets?
What kind of stuff is in there? What can you and can you not build? Like, for
example, Blink is one of the things in `//third_party`, and lots of people -
that's a big part of it, right? But a lot of things in there are less active
and probably less big of a part of Chrome. So does `//third_party` just build
anything else, or what's different about it?

20:09 NICO: And that's a great question. So Blink being in `//third_party` is a
bit of a historical artifact. Like, most things - almost all of the things in
`//third_party` is basically code that's third-party code. That's code that we
didn't write ourselves. And Chrome's secret mission is to depend on every other
library out there in the world. No, we depend on things like libpng for reading
PNG files, libjpeg for reading all of - libjpeg-turbo these days, I guess, for
reading JPEG files, libxml for reading XML, and so on. And, well, that's many
dependencies. I won't list them all. And some of these third-party dependencies
are just listed in the DEPS file that we talked about. And so they basically -
like, when gclient sync runs, it pulls the code from some git repository that
contains the third-party code and puts it into your source tree. And for other
third-party code, we actually check in the code into the Chrome main repository
instead of DEPSing it in. There are trade-offs, which approach to choose. We do
both from time to time. But yeah. Almost no third-party dependency has a GN
file upstream, so usually what you do is you have to write your own BUILD.gn
file for the third-party dependency you want to add. And then after that, it's
fairly normal. So for a library, if you want to add a dependency on libfoo,
usually what we do is you add - you create third-party libfoo, and you put
BUILD.gn in there. And then you add a DEPS entry that syncs the actual code to
a third-party libfoo source or something. Yes.

21:37 SHARON: All right. Sounds good. Again, you mentioned BUILD.gn files, and
that's, as, expected a big part of how building works. And that's probably the
part that most people have interacted more with, outside of just actually
running whatever command it is to build Chrome. Because if you create, delete,
rename any files, you have to update it in some BUILD.gn file. So can you walk
us through the different things contained in a BUILD.gn file? What are all the
different parts?

22:12 NICO: Sure. So there's a great presentation by Brett, who wrote GN, that
we can link to. But in a summary, it's - BUILD.gn contains build targets, and
the build target normally is like - it doesn't have to be, but usually, it's a
list of CC files that belong together and that either make up a static library
or a shared library on executable. So those are the main target types for CC
code. But then you can also have custom build actions that run just arbitrary
Python code, which, for example, if you compile a protobuf - proto files into
CC and H - into C++ and header files, then we basically have a Python script
that runs protoc, the proto compiler, to produce those. And so in that case,
the action generates C++ files, and then those get built. But the other, simple
answer is libraries or executables.

23:11 SHARON: OK. One part of GN files that has caused me personally some
confusion and difficulty - which I think is maybe, depending on the part of
Chrome you work on, less of an issue - is DEPS. So you have DEPS in your GN
files, and there's also something called external DEPS. And then you have
separate DEPS files that are just called capital D-E-P-S.

23:30 NICO: Yes. Yes, there, that's some redundant - that's, again, I guess for
historical reasons. So in gclient, DEPS just means to build this target, you
first have to build these other targets. Like, this target depends - uses this
other code. And in different contexts, it kind of means different things. So
for example - I think if an executable depends on some other target, then that
external executable is linked - that other target is also linked in. If base
unit test depends on the base library, which in a normal build is a static
library - like in a normal build? Like in a release build, by default, it's a
static library. And so if base unit test is built, it first creates a static
library and then links to it. And then base itself might depend on a bunch of
third-party things, libraries, which means when base unit tests is linked, it
links base, but then it also links against basis dependencies. So that's one
meaning of DEPS. Another meaning, like these capital DEPS files, that's
completely distinct. Has nothing to do with GN, I'm sad to say. And that's just
for enforcing layering. Those predate GN, and they are for enforcing layering
at a fairly coarse level. They say, code in this directory may include code
from this other directory but not from this third directory. For example, a
third - like, Blink must not - may include stuff from base, but must not
include anything from, I don't know, the Chrome layer or something.

25:18 SHARON: Right, the classic content Chrome layering, where Chrome -

25:18 NICO: Right. And I think -

25:18 SHARON: content, but -

25:18 NICO: Right. And there's a step called check-deps, and that checks the
capital DEPS files.

25:24 SHARON: OK. Yeah, because before, I worked on some Fuchsia stuff, and
because we're adding a lot of new things, you're messing around with different
DEPS and stuff a lot more than I think if you worked in a typical part. Like,
now, I mostly just work within content. Unlikely that you're changing any
dependencies. But that was always a bit unclear because, for the most part, the
targets have very similar names - not exactly the same, but very similar. And
if you miss one, you get all these weird errors. And it was, yeah, generally
quite confusing.

25:55 NICO: Yeah, that's pretty confusing. One thing of the capital DEPS things
that they can do that the GN DEPS can't is if someone adds a DEPS on your
library and they add an entry to their DEPS file, that means that now at code
review time, you need to approve that they depend on you. And that's not
something we can do at the GN level. And the advantage there is, I don't know,
if you have some library and then 50 teams start depending on it without
telling you, and now you're on the hook for keeping all these 50 things
working, then with this system, you at least have to approve every time someone
adds a dependency on you, you have to say, this is fine with me. Or you can
say, actually, this is - we don't want this to be used by anyone else.

26:45 SHARON: Is there an ideal state where we don't have these DEPS files and
maybe that functionality is built into the BUILD.gn files, or is this something
that's probably going to be sticking around for a while?

26:52 NICO: That's a great question. I don't know. It seems weird, right? It's
redundant. So I think the current system isn't ideal, but it's also not
horrible enough that we have to fix it immediately. So maybe one day we'll get
around to it.

27:10 SHARON: Yeah. I think I've mostly just worked on Chrome, so I've gotten
pretty used to it. But a common complaint is people who work in Google internal
things or other, bigger - the main build system of whatever company they work
on, they come to Chrome and they're like, oh, everything's so confusing. But if
you - you just got to get used to it, but -

27:27 NICO: Right. I think if you're confused by anything, it's great if you
come to us and complain. Because you kind of become blind to these problems,
right? I've been doing this for a long time. I'm used to all the foot guns. I
know how to dodge them. And yeah. So if you're confused by anything, please
tell me personally. And then if enough people complain about something, maybe
we'll fix it.

27:55 SHARON: All right. Yeah. That's what you said. The outcome of that -
we'll see. We'll see how that goes. We'll see how many complaints you suddenly
get. Right. OK. So another thing I was interested in is right now there's a lot
of work around Rust, getting more Rust things, introducing that, memory safety,
that's good. We like it. What is involved from a build perspective for getting
a whole other language into Chrome and into the build? Because we have most of
the things C++. There's some Java in all of the Android stuff. And in some
areas, you see - you'll see a list of - you'll see a file name, and then you'll
see file name underscore and then all the different operating systems, right?
And most those are some version of C++. The Mac ones are .mm. And you have Java
ones for Android. But if you want to add an entirely different language and
still be able to build Chrome, at a high level, what goes into that?

29:00 NICO: Yeah, there's also some Swift on iOS. It's many different things.
So at first, you have to teach GN how to generate Ninja files for that
language. So when a CC file is built, then basically the compiler writes out a
file that says, here are all the header files I depend on. So if one of them
gets touched, the compiler - or Ninja knows how to rebuild those. So you need
to figure out how the Rust compiler or the Swift compiler track dependencies.
You need to get that information out of the compiler into the build system
somehow. And C++ is fairly easy to build. It's like a per-file basis. I think
most languages are more on a module or package base, where you build a few
files as a unit. Then you might want to think about, how can I make this work
with Goma so that the compilation can work remotely instead of locally? So
that's the build system part. Then also, especially for us, we want to use this
for some performance critical things, so it needs to be very fast. And we use a
bunch of toolchain optimization techniques to make Chrome very fast with
three-letter acronyms, such as PGO and LTO and whatnot. And LTO in particular,
that means a Link Time Optimization. That means the C++ or the Rust code is
built - is compiled into something called "bitcode." And then all the bitcode
files at link time are analyzed together so you can do cross-file in-lining and
whatnot. And for that work, the bitcodes - all the bitcode versions need to be
compatible, which means Clang and Rust need to be built against the same
version of LLVM, which is some - it's some internal compiler machinery that
defines the bitcode. So that means you have to - if you want to do
cross-language LTO, you have to update your C++ compiler and your Rust compiler
at the same time. And you have to build them at the same time. And when you
update your LLVM revision, it must break neither the C++ compiler nor the Rust
compiler. Yeah. And then you kind of want to build the Rust library from
source, so you have bit code for all of that. So it's a fairly involved - but
yeah, we've been doing a lot of work on that. Not me, but other people.

31:24 SHARON: Right. Sounds hard. And what does LTO stand for, since you used
it?

31:30 NICO: Link Time Optimization.

31:30 SHARON: All right.

31:30 NICO: And there's a blog post on the Chromium blog about this that we can
link to in the show notes that has a fairly understandable explanation what
this does.

31:43 SHARON: Yeah, all right. That sounds good. So linking, that was my next
question. As you build stuff, you sort out all of your just compile errors, you
got all your spelling mistakes out. The next type of error you might get is
linking error. So how does - can you tell us a bit more about linking in
general and how that fits into the build process?

32:01 NICO: I mean, linking - like, for C++, the compiler basically produces
one object file for every CC file. And then the linker takes, like, about
50,000 to 100,000 object files and produces a single executable. And every
object file has a list of functions that are defined in that object file and a
list of functions that are undefined in that object file that it calls that are
needed from elsewhere. And then the linker basically makes one long list of all
the functions it finds. And at the end, all of them should be defined, and all
the non-inline ones should be defined in exactly one object file. And if
they're not - if that doesn't happen, then it emits an error, and else, it
emits a binary. And the linker is kind of interesting because the only thing
you really care about is that it does its job very quickly. But it has to read
through gigabytes of data before it writes the executable. And currently, we
use a linker called `ld`, which was also written by people on the Chrome team,
and which is also fairly popular outside of Chrome nowadays. And so we wrote on
ELF linker, which is the file format used on Linux and Android, and on COFF
linker, which is the file system used on Windows, and our own Mach-O linker,
which is the file system on Apple - macOS and iOS. And our linkers are way,
way, way faster than the things that they replace. On Windows, we were, like,
10 times faster than the Windows linker. And on Mac, we're, like, four times
faster than the system linker and whatnot. The other linker vendors have caught
up a little bit, but we - I feel like Chrome has really advanced the state and
performance of linking binaries across the industry, which I think is really
cool.

33:44 SHARON: Yeah, that is really cool. And in a kind of similar vein to the
different OSes and all that kind of stuff is 32- versus 64-bit. There's some
stuff happening. I've seen people talk about it. It seems pretty important. Can
you just tell us a bit more about this in general?

34:04 NICO: Well, I guess most processors sold in the last decade or so are
64-bit. So I think on some platforms, we only support 64-bit binaries, like -
and the bit just means how wide is a pointer and has some implications on which
instructions can the compiler use. But it's fairly transparent too, I think, at
the C++ level. You don't have to worry about it all that much. On macOS, we
only support 64-bit builds. Same on iOS. On Windows, we still have 32-bit and
64-bit builds. On Linux, we don't publicly support 32-bit, but I think some
people try to build it. But it's really on Windows where you have both 32-bit
and 64-bit builds. But the default bits is 64-bit, and you can say, if you say
target CPU equals x86, I think, in your args.gn, then you get a 32-bit build.
But it should be fairly transparent to you as a developer, unless you write
assembly.

35:02 SHARON: How big of an effort would it be to get rid of 32-bit on Windows?
Because Windows is probably the biggest Chrome-using platform, and also,
there's a lot of versions out there, right? So -

35:15 NICO: Oh, yeah.

35:15 SHARON: How doable?

35:15 NICO: I think that the biggest platform is probably Android. But yeah,
Android is also 32-bit, at least on some devices at the moment. That's true. I
don't know. I think we've looked into it and decided that we don't want to do
that at the moment. But I don't know details.

35:33 SHARON: And you mentioned ARM. So is there any - how much does the Chrome
build team - are they concerned with the architecture of these processors? Is
that something that, at the level that you and the build team have to worry
about, or is it far enough - a few layers down that that's -

35:47 NICO: It's something we have to worry about at the toolchain team. So we
update the scaling compiler every two weeks or so, which means we pull in all -
around 1,000 changes from upstream contributors that work on LVM spread across
many companies. And we have to make sure this doesn't break from on 32-bit ARM,
64-bit ARM, 32-bit Intel, 64-bit Intel, across seven different operating
systems. And so fairly frequently, when we try to update Clang tests start
failing on, I don't know, 32-bit Windows or on 64-bit iOS or some very specific
configuration. And then we have to go and debug and dissect and figure out
what's going on and work with upstream to get that fixed. So yeah. That's
something we have to deal with at the toolchain team, but hopefully, it's -
hopefully, like the normal Chrome developer is isolated from that for the most
part.

36:45 SHARON: I think so. It's not - if I weren't asking all these other
questions, it's something that almost never crosses my mind, right? So that
means you're all doing a very good job of that. Thank you very much. Much
appreciated. And jumping way back, you mentioned earlier indexing the code
base, code search. So I make a change. I submit it. I upload it. It eventually
ends up in code search. So how does that process work? And what goes into
indexing? Because before, when I was working on Fuchsia all the Fuchsia code
wasn't indexed, so you couldn't do the handy thing of clicking a thing and
seeing where it was defined. You had to actually look it up. And once you got
that, it was like, oh my gosh, so much better. So can you just tell us a bit
more about that process?

37:30 NICO: Sure, yeah. The Chrome has a pretty good code search feature, I
think, codesearch.chromium.org or cs.chromium.org. Basically, we have a bot
that runs, I think, every six hours or so, pulls the latest code, bundles it
up, sends it to some indexer service that then also uses Clang to analyze the
code. Like, for C++, I think we also index Java. We probably don't index Rust
yet, but eventually we will. And then it generates - for every word, it
generates metadata that says, this is a class. This is an identifier. And so if
you click on it, if you click on a function, you have the option of jumping to
the definition of the function, to the declaration, to all the calls, all the
overrides, and so on. And that updates ideally several times a day and is
fairly up to date. And we built the index, I think, for most operating systems.
So you can see this is called here on Linux, here on Windows, and what not.

38:32 SHARON: OK. Sounds good. Very useful stuff. And I don't know if this is
part of the build team's jurisdiction, but when you are working on things
locally, you have some git commands, and then you have some git-cl commands.

38:43 NICO: Mm-hmm.

38:48 SHARON: So the git commands are your typical ones - git pull, git rebase,
git stash, that kind of thing. And then you have git-cl commands, which relate
more to your actual CL in Gerrit. So git-cl upload, git-cl status. That'll show
you all your local branches and if they have a Gerrit change associated with
them. So what's the difference between git and git-cl commands?

39:18 NICO: I'm sorry. So this is basically a git feature. If you call git-foo,
then git looks for git-foo on your path. So you can add arbitrary commands to
git if you want to. And git-cl is just something that's in `depot_tools`.
Again, there's git-cl in `depot_tools`, and you can open that and see what it
does. And it'll redirect to `git_cl.py`, I think, which is a fairly long and
hairy Python script. But yeah. It's basically Gerrit integration, as you say.
So you can use that to send try jobs, `git cl try`. To upload, as you say, you
can use `git cl issue` to associate your current branch with a remote Gerrit
review, `git cl patch` to get a patch off Gerrit and patch it into your local
thing, `git cl web` to open the current thing in a web browser. Yeah, git-cl is
basically - git-cl help to see all the git-cl commands, or - yeah. If you have
a change that touches, like, 1,000 files, you can run `git cl split`, and it'll
upload 500 reviews. But that's usually too granular, and I wouldn't recommend
doing that. But it's possible.

40:25 SHARON: Right. Do you have a - [DOORBELL DINGS]

40:25 NICO: Oops, sorry.

40:25 SHARON: commonly - yeah.

40:30 NICO: Oh, sorry. There was - the door just rang. Maybe you didn't hear
it. Sorry.

40:30 SHARON: All right. It's all good. Do you have a lesser known git or
git-cl command that you use a lot or -

40:41 NICO: Well, I -

40:41 SHARON: is your favorite? [LAUGHS]

40:46 NICO: It's not lesser known to me, so I wouldn't know. I don't know. I
use `git cl upload` a lot.

40:53 SHARON: Right. Well, you have to use `git cl upload`, right?

40:53 NICO: I use -

40:53 SHARON: Well, you don't - maybe not but -

40:53 NICO: `git cl try` to send try jobs from my terminal, `git cl web` to see
what's going on, `git cl patch` a lot to patch stuff in locally. If I'm doing a
code review and I want to play with it, I patch it in, build a local, and see
how things are working.

41:12 SHARON: Yeah. When I patch in a thing, I go from the cl page on Gerrit
and then click the down patch thing, but -

41:21 NICO: No, even `git cl patch -b` and then some branch name, and then you
just patch - paste the Gerrit review URL.

41:28 SHARON: Oh, cool.

41:28 NICO: So it's just, yeah, Control-L to focus the URL bar. Control-C
Alt-Tab `git cl patch -b blah`, Paste, Enter, and then you have a local branch
with the thing.

41:36 SHARON: All right. Yeah, a lot of these things, once you learn about
them - at first you're like, whoa, and then you use them, and then they're not
lesser known to you, but you tell other people also a common - so another one
would be `git cl archive`, which will -

41:47 NICO: Oh, yeah, yeah.

41:47 SHARON: get rid of any local branches associated with a closed Gerrit
branch, so that's very handy, too.

41:53 NICO: Yes.

41:53 SHARON: So it's always fun to learn about things like that.

41:59 NICO: Are you fairly tidy with your branches? How many open branches do
you usually have?

41:59 SHARON: [LAUGHS] I used to be more tidy. When I tried to do a cleanup
thing, I had more branches. I think right now I've got around 20-something
branches. I like having not very many. I think to some people, that's a lot. To
some people, that's not very many. I mean, ideally, I have under five, right?
[LAUGHS] But -

42:18 NICO: I don't know. I usually have a couple 10, sometimes. Have a bunch
of machines. I think on some of them it's over 100, but yeah. Every now and
then, I run `git cl archive` and it removes half of them, but -

42:29 SHARON: Yes. All right, cool. Is there anything that we didn't cover so
far that you would like to share? So things that maybe you get asked all the
time, things that people could do better when it comes to build-related things?
Things that you can do that make the build better or don't make it worse, that
kind of thing? Or just anything else you would like to get out there?

42:58 NICO: I guess one thing that's maybe implicitly stated, but currently not
explicitly documented, as far as I know, but I'm hoping to change that, is - so
Chrome tries to have a quiet build. Like, if you build this zero build output,
except that one Ninja file, Ninja line that's changing, right? There's, well,
another code basis - I think it's fairly common - that there's many screenfulls
of warning that scroll by. And we very explicitly try not to do that because if
the build emits lots of warnings, then people just learn to ignore warnings. So
we think something should either be a serious problem that people need to know
about, then it should be an error, or it should be not interesting. Then it
should be just quiet. So if you add a build step that adds a random script, the
script shouldn't print anything, just about progress. Shouldn't say, doing
this, doing this, doing this. Should either print something and say something's
wrong and fail those build step or not say anything. So that's one thing.

43:51 SHARON: That's - yeah, that's true.

43:51 NICO: And the other thing -

43:51 SHARON: Like, you only really get a bunch of terminal output if you have
a compile or a linker error, whatever.

43:57 NICO: Right.

43:57 SHARON: I hadn't ever considered that. If you build something and it
works, you get very few lines of output. And I hadn't ever thought that was
intentional before, but you're right in that if it was a ton, you would just
not look at any of it. So yeah, that's very cool.

44:09 NICO: Yeah. And on that same note, we don't do deprecation warnings
because we don't do any warnings. So if people - like, people like deprecating
things, but people don't like tidying up calls to deprecated functions. So if
you want to deprecate something in Chrome, the idea is basically, you remove
all callers, and then you remove the deprecated thing. And we don't allow you
to say - to add a warning that tells everyone, hey, please, everyone, remove
your calls. The onus is on the person who wants to deprecate something instead
of punting that to everyone else.

44:46 SHARON: Yeah, I mean, the thing that I was working on has a deprecating
effect, so removing callers, which is why I have so many branches. But I've
also seen presubmit warnings for if you include something deprecated. So - oh,
yeah, and there's presubmit, too. OK, we'll get to that also. [LAUGHS] Tell us
more about all of this.

45:05 NICO: About presubmits? Yeah, presubmits - presubmits are terrible.
That's the short summary. So if you run a `git cl presubmit`, it'll look at a
file called presubmit.py, I think, in the current directory, and maybe in all
the directories of files - of directories that contain files you touched or
something like that. But you can just open the top-level presubmit.py file, and
there's a couple thousand lines of Python where basically everyone can add
anything they want without much oversight, so it's a fairly long - at least
historically, that used to be the case. I don't know if that's still the case
nowadays. But yeah, it's basically like a long list of things that random
people thought are good if they - like, presubmits are something that are run
before you upload, also, implicitly. And so you're supposed to clean them up.
And [INAUDIBLE] many useful things. For example, nowadays we require most code
to be autoformatted so that people don't argue about where semicolons should go
or something silly like that. So one of the things it checks is, did you run
`git cl format`, which runs, I guess, Clang format for C++ code and a bunch of
custom Python scripts for other files. But it's also - presubmits have grown
organically, and there isn't - they're kind of unowned and they're very, very
slow. And I think some people have tried to improve them recently, and they're
better than they used to be, but I don't love presubmits, I guess is the
summary. But yeah, it's another thing to check invariants that we would like to
be true about our code base.

46:48 SHARON: Yeah. I mean, I think - yes, spelling is something I think it
also checks.

46:54 NICO: It checks spelling? OK.

46:54 SHARON: Or maybe that's a separate bot in Gerrit.

46:59 NICO: Oh, yeah, yeah, yeah, yeah. Like, there's this thing called -
what's its name?

47:06 SHARON: Trucium? Tricium?

47:06 NICO: Tricium, yeah. Tricium, right. Tricium is something that adds
comments to your - automatically adds comments to your change list when you
upload it. And Tricuium can do spelling correction, but it can also - it runs
something called Clang Tidy, which is basically a static analysis engine which
has quite a few false positives, so sometimes it complains about something
that - but it's actually incorrect, and so we don't put that into the compiler
itself. So we've added a whole bunch of warnings to the compiler for things
that we think are fairly buggy. But Clang Tidy is - but these warnings have to
be - they have to have a very low false positive rate. Like, if they complain,
they should almost always be right. But sometimes, for static analysis, it's
hard to be right. Like, you can say this might be wrong. Please be sure. But
this is not something the compiler can say, so we have this other system called
Clang Tidy which also adds a comment to your C++ code which says, well, maybe
this should be a reference instead of a copy, and things like that.

48:04 SHARON: Yeah. And I think it - I've seen it - it checks for unused
variables and other - there's been useful stuff that's come from comments from
there, so definitely. All right. Very cool. So if people are interested in all
this build "infra-y" kind of stuff and they want to get more into it, what can
they do?

48:32 NICO: We have a public build@chromium.org mailing list. It's very low
volume, but if you want to reach out, you can send an email there and a few of
us will see your email and interact with you. And there's also I think the tech
build on crbug. So you can just look for build bugs and fix all our bugs for
us. That'd be cool.

48:51 SHARON: [LAUGHS]

48:51 NICO: And if there's anything specific, just talk to local OWNERS. Or if
you feel this is just something you're generally interested in and you're
looking for a project, you can talk to me, and I probably have a long list of -
I do have a long list of somewhat beginner-friendly projects that people could
help out with, I guess.

49:15 SHARON: Yeah. I mean, I think being able to - if you're looking for a
20%y kind of project or something else. But knowing how things actually get put
together is always a good skill and definitely applicable to other things. It's
the kind of thing where the more low level-knowledge you have, the more - it
works - it applies to things higher up, but not necessarily the other way
around, right?

49:34 NICO: Mm-hmm.

49:34 SHARON: So having that kind of understanding is definitely a good thing.
All right. Any last things you'd like to mention or shout out or cool things
that you want people to know about? [LAUGHS]

49:48 NICO: I guess -

49:48 SHARON: Or what - yeah, quickly, what is the future of the whole build
thing? Like, what's the ideal situation if -

49:55 NICO: Ideally, it'll all be way faster, I guess is the main thing. But
yeah, yeah, I think build speed is a big problem. And I'm not sure we have the
best handle on that. We're working on many things, but - not many. A bunch of
things. But it's - like, people keep adding all that much code, so if y'all
could delete some code, too, that would help us a lot. I mean, having -
supporting more languages is something we have to - this is something that's
happening. Like, Rust is happening. We are also on iOS also using Swift.
Currently, we can't LTO Swift with the rest because that's on a different OEM
version. There's this - in C++ - we keep upgrading C++ versions. So Peter
Kasting is working on moving us to C++20. And then 23, we'll have them, and so
on. There's maybe C++ modules at some point, which may or may not help with
build speed. And there's a bunch of tech debt that we need to clean up, but
that's not super interesting.

51:24 SHARON: I don't know. I think people in Chrome in general are more
interested and care about reducing tech debt in general, right? A lot of people
I know would be happy to just do tech debt clean-up things only, right?
Unfortunately, it doesn't really work out for job reasons. But a lot of people,
I think, are interested in, I think, in higher proportions than maybe other
places.

51:47 NICO: It depends on the tech debt. Some of it might work out for job
reasons. But, yeah.

51:54 SHARON: Yeah. I mean, some of it is easier than others, too, right? Some
of it is like, yeah, so, OK, well, go delete some code. Go clean up some
deprecated calls. [LAUGHS] All that.

52:08 NICO: Yeah, and again, I think finishing migrations is way harder than
starting them, so finish more migrations, start fewer migrations. That'd be
also cool.

52:16 SHARON: All right. I am sure everyone listening will go and do that right
away.

52:21 NICO: Yep.

52:21 SHARON: And things will immediately be better.

52:27 NICO: They've just been waiting to hear that from me, and now they're
like, ah, yeah, right. That makes sense.

52:27 SHARON: Yeah, yeah. All right. Well, you all heard it here first. Go do
that. Things will be better, et cetera. So all right. Well, thank you very
much, Nico, for being here answering all these questions. I learned a lot. A
lot of this is stuff that - everyone who works on Chrome builds Chrome, right?
But you can get by with a very minimal understanding of how these things are.
Like, you see your - you follow the Intro to Building Chrome doc. You copy the
things. You're like, OK, this works. And then you just keep doing that until
you have a problem. And depending on where you work, you might not have
problems. So it's very easy to know very little about this. But obviously, it's
so important because if we didn't have any of this infrastructure, nothing
would work. So one, I guess, thank you for doing all the stuff behind the
scenes, determinism, OSes, all that, making it a lot easier for everyone else,
but also thank you for sharing about it so people understand what's actually
going on when they run the commands they do every day.

53:31 NICO: Sure. Anytime. Thanks for having me. And it's good to hear that
it's possible to work on Chrome without knowing much about the build because
that's the goal, right? It should just work.

53:44 SHARON: Yeah.

53:44 NICO: Sometimes it does.

53:44 SHARON: [LAUGHS] Yeah. Well, thank you for all of it, and see you next
time.

53:51 NICO: Yeah. See you on the internet. Bye.

54:03 SHARON: OK. So we will stop recording -

54:03 NICO: Wee. Time for the second take.

54:03 SHARON: [LAUGHS] Let's do that, yeah, all over again.

54:11 NICO: Let's do it.

54:11 SHARON: I will stop recording.