[Build Chromium] Improve git checkout (#83225) (#87009)

* [Build Chromium] Document steps for local build, improve checkout

commit e817796d107d554504870cdb8e6dd10db1079a1e
Merge: 19e6b300260 61b4e052fdd
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Thu Nov 12 13:25:09 2020 -0700

    Merge branch 'chore/build_chromium/improvements' of github.com:tsullivan/kibana into chore/build_chromium/improvements

commit 19e6b300260822418cea4dc594bdfdf621a331e0
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Thu Nov 12 13:23:01 2020 -0700

    fixes

commit 9de24ec298792a46e88207f41c9f6aff1997dbf1
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Thu Nov 12 13:22:52 2020 -0700

    add instructions to build local

commit 872c0f68e0077ad50c541b605a87253befd1a891
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Thu Nov 12 11:58:25 2020 -0700

    simplify

commit 8dae9484efcdc483ad20ceab32cae4e97735ab1c
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Thu Nov 12 10:28:18 2020 -0700

    fixes

commit 492f5cfe25e2c7ccffce55a8733383a30aae1cb6
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Wed Nov 11 20:15:43 2020 -0700

    --wip-- [skip ci]

commit acba359b121f7be8e6c353f90e938bc5d88658a1
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Wed Nov 11 15:47:50 2020 -0700

    [Build Chromium] Improve git checkout

commit 61b4e052fdd04456fe23956ad8ce99c8e771c01e
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Wed Nov 11 20:15:43 2020 -0700

    --wip-- [skip ci]

commit 58300c9deeca15cce838b3956c55c2994f99f530
Author: Timothy Sullivan <tsullivan@elastic.co>
Date:   Wed Nov 11 15:47:50 2020 -0700

    [Build Chromium] Improve git checkout

* Apply suggestions from code review

Co-authored-by: Joel Griffith <joel@joelgriffith.net>

Co-authored-by: Joel Griffith <joel@joelgriffith.net>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Joel Griffith <joel@joelgriffith.net>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This commit is contained in:
Tim Sullivan 2020-12-29 15:39:56 -07:00 committed by GitHub
parent 8a08bc2a1f
commit 9af1d93aba
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 197 additions and 89 deletions

View file

@ -1,13 +1,15 @@
# Chromium build
We ship our own headless build of Chromium which is significantly smaller than the standard binaries shipped by Google. The scripts in this folder can be used to initialize the build environments and run the build on Mac, Windows, and Linux.
We ship our own headless build of Chromium which is significantly smaller than
the standard binaries shipped by Google. The scripts in this folder can be used
to accept a commit hash from the Chromium repository, and initialize the build
environments and run the build on Mac, Windows, and Linux.
The official Chromium build process is poorly documented, and seems to have breaking changes fairly regularly. The build pre-requisites, and the build flags change over time, so it is likely that the scripts in this directory will be out of date by the time we have to do another Chromium build.
This document is an attempt to note all of the gotchas we've come across while building, so that the next time we have to tinker here, we'll have a good starting point.
# Before you begin
You'll need access to our GCP account, which is where we have two machines provisioned for the Linux and Windows builds. Mac builds can be achieved locally, and are a great place to start to gain familiarity.
## Before you begin
If you wish to use a remote VM to build, you'll need access to our GCP account,
which is where we have two machines provisioned for the Linux and Windows
builds. Mac builds can be achieved locally, and are a great place to start to
gain familiarity.
1. Login to our GCP instance [here using your okta credentials](https://console.cloud.google.com/).
2. Click the "Compute Engine" tab.
@ -15,21 +17,89 @@ You'll need access to our GCP account, which is where we have two machines provi
4. If #3 fails, you'll have to spin up new instances. Generally, these need `n1-standard-8` types or 8 vCPUs/30 GB memory.
5. Ensure that there's enough room left on the disk. `ncdu` is a good linux util to verify what's claming space.
## Usage
```
# Create a dedicated working directory for this directory of Python scripts.
mkdir ~/chromium && cd ~/chromium
# Copy the scripts from the Kibana repo to use them conveniently in the working directory
cp -r ~/path/to/kibana/x-pack/build_chromium .
# Install the OS packages, configure the environment, download the chromium source
python ./build_chromium/init.sh [arch_name]
# Run the build script with the path to the chromium src directory, the git commit id
python ./build_chromium/build.py <commit_id>
# You can add an architecture flag for ARM
python ./build_chromium/build.py <commit_id> arm64
```
## Getting the Commit ID
Getting `<commit_id>` can be tricky. The best technique seems to be:
1. Create a temporary working directory and intialize yarn
2. `yarn add puppeteer # install latest puppeter`
3. Look through puppeteer's node module files to find the "chromium revision" (a custom versioning convention for Chromium).
4. Use `https://crrev.com` and look up the revision and find the git commit info.
The official Chromium build process is poorly documented, and seems to have
breaking changes fairly regularly. The build pre-requisites, and the build
flags change over time, so it is likely that the scripts in this directory will
be out of date by the time we have to do another Chromium build.
This document is an attempt to note all of the gotchas we've come across while
building, so that the next time we have to tinker here, we'll have a good
starting point.
## Build args
Chromium is built via a build tool called "ninja". The build can be configured by specifying build flags either in an "args.gn" file or via commandline args. We have an "args.gn" file per platform:
A good how-to on building Chromium from source is
[here](https://chromium.googlesource.com/chromium/src/+/master/docs/get_the_code.md).
- mac: darwin/args.gn
- linux 64bit: linux-x64/args.gn
There are documents for each OS that will explain how to customize arguments
for the build using the `gn` tool. Those instructions do not apply for the
Kibana Chromium build. Our `build.py` script ensure the correct `args.gn`
file gets used for build arguments.
We have an `args.gn` file per platform:
- mac: `darwin/args.gn`
- linux 64bit: `linux-x64/args.gn`
- windows: `windows/args.gn`
- ARM 64bit: linux-aarch64/args.gn
- windows: windows/args.gn
The various build flags are not well documented. Some are documented [here](https://www.chromium.org/developers/gn-build-configuration). Some, such as `enable_basic_printing = false`, I only found by poking through 3rd party build scripts.
To get a list of the build arguments that are enabled, install `depot_tools` and run
`gn args out/headless --list`. It prints out all of the flags and their
settings, including the defaults.
As of this writing, there is an officially supported headless Chromium build args file for Linux: `build/args/headless.gn`. This does not work on Windows or Mac, so we have taken that as our starting point, and modified it until the Windows / Mac builds succeeded.
The various build flags are not well documented. Some are documented
[here](https://www.chromium.org/developers/gn-build-configuration).
As of this writing, there is an officially supported headless Chromium build
args file for Linux: `build/args/headless.gn`. This does not work on Windows or
Mac, so we have taken that as our starting point, and modified it until the
Windows / Mac builds succeeded.
**NOTE:** Please, make sure you consult @elastic/kibana-security before you change, remove or add any of the build flags.
## Building locally
You can skip the step of running `<os_name>/init.sh` for your OS if you already
have your environment set up, and the chromium source cloned.
To get the Chromium code, refer to the [documentation](https://chromium.googlesource.com/chromium/src/+/master/docs/get_the_code.md).
Install `depot_tools` as suggested, since it comes with useful scripts. Use the
`fetch` command to clone the chromium repository. To set up and run the build,
use the Kibana `build.py` script (in this directory).
It's recommended that you create a working directory for the chromium source
code and all the build tools, and run the commands from there:
```
mkdir ~/chromium && cd ~/chromium
cp -r ~/path/to/kibana/x-pack/build_chromium .
python ./build_chromium/init.sh [arch_name]
python ./build_chromium/build.py <commit_id>
```
## VMs
I ran Linux and Windows VMs in GCP with the following specs:
@ -57,7 +127,8 @@ The more cores the better, as the build makes effective use of each. For Linux,
## Initializing each VM / environment
You only need to initialize each environment once. NOTE: on Mac OS you'll need to install XCode and accept the license agreement.
In a VM, you'll want to use the init scripts to to initialize each environment.
On Mac OS you'll need to install XCode and accept the license agreement.
Create the build folder:
@ -86,16 +157,6 @@ In windows, at least, you will need to do a number of extra steps:
## Building
Find the sha of the Chromium commit you wish to build. Most likely, you want to build the Chromium revision that is tied to the version of puppeteer that we're using.
Find the Chromium revision (run in kibana's working directory):
- `cat node_modules/puppeteer-core/package.json | grep chromium_revision`
- Take the revision number from that, and tack it to the end of this URL: https://crrev.com
- (For example, puppeteer@1.19.0 has rev (674921): https://crrev.com/674921)
- Grab the SHA from there
- (For example, rev 674921 has sha 312d84c8ce62810976feda0d3457108a6dfff9e6)
Note: In Linux, you should run the build command in tmux so that if your ssh session disconnects, the build can keep going. To do this, just type `tmux` into your terminal to hop into a tmux session. If you get disconnected, you can hop back in like so:
- SSH into the server

View file

@ -1,55 +1,80 @@
import subprocess, os, sys, platform, zipfile, hashlib, shutil
from build_util import runcmd, mkdir, md5_file, script_dir, root_dir, configure_environment
import os, subprocess, sys, platform, zipfile, hashlib, shutil
from os import path
from build_util import (
runcmd,
runcmdsilent,
mkdir,
md5_file,
configure_environment,
)
# This file builds Chromium headless on Windows, Mac, and Linux.
# Verify that we have an argument, and if not print instructions
if (len(sys.argv) < 2):
print('Usage:')
print('python build.py {chromium_version}')
print('python build.py {chromium_version} [arch_name]')
print('Example:')
print('python build.py 68.0.3440.106')
print('python build.py 4747cc23ae334a57a35ed3c8e6adcdbc8a50d479')
print('python build.py 4747cc23ae334a57a35ed3c8e6adcdbc8a50d479 arm64 # build for ARM architecture')
print
sys.exit(1)
src_path = path.abspath(path.join(os.curdir, 'chromium', 'src'))
build_path = path.abspath(path.join(src_path, '..', '..'))
build_chromium_path = path.abspath(path.dirname(__file__))
argsgn_file = path.join(build_chromium_path, platform.system().lower(), 'args.gn')
# The version of Chromium we wish to build. This can be any valid git
# commit, tag, or branch, so: 68.0.3440.106 or
# 4747cc23ae334a57a35ed3c8e6adcdbc8a50d479
source_version = sys.argv[1]
base_version = source_version[:7].strip('.')
# Set to "arm" to build for ARM on Linux
arch_name = sys.argv[2] if len(sys.argv) >= 3 else 'x64'
print('Building Chromium ' + source_version + ' for ' + arch_name)
if arch_name != 'x64' and arch_name != 'arm64':
raise Exception('Unexpected architecture: ' + arch_name)
# Set the environment variables required by the build tools
print('Configuring the build environment')
configure_environment()
print('Building Chromium ' + source_version + ' for ' + arch_name + ' from ' + src_path)
print('src path: ' + src_path)
print('depot_tools path: ' + path.join(build_path, 'depot_tools'))
print('build_chromium_path: ' + build_chromium_path)
print('args.gn file: ' + argsgn_file)
print
# Sync the codebase to the correct version, syncing master first
# to ensure that we actually have all the versions we may refer to
print('Syncing source code')
# Sync the codebase to the correct version
print('Setting local tracking branch')
print(' > cd ' + src_path)
os.chdir(src_path)
os.chdir(os.path.join(root_dir, 'chromium/src'))
checked_out = runcmdsilent('git checkout build-' + base_version)
if checked_out != 0:
print('Syncing remote version')
runcmd('git fetch origin ' + source_version)
print('Creating a new branch for tracking the source version')
runcmd('git checkout -b build-' + base_version + ' ' + source_version)
runcmd('git checkout master')
runcmd('git fetch origin')
runcmd('gclient sync --with_branch_heads --with_tags --jobs 16')
runcmd('git checkout ' + source_version)
runcmd('gclient sync --with_branch_heads --with_tags --jobs 16')
runcmd('gclient runhooks')
depot_tools_path = os.path.join(build_path, 'depot_tools')
path_value = depot_tools_path + os.pathsep + os.environ['PATH']
print('Updating PATH for depot_tools: ' + path_value)
os.environ['PATH'] = path_value
print('Updating all modules')
runcmd('gclient sync')
# Copy build args/{Linux | Darwin | Windows}.gn from the root of our directory to out/headless/args.gn,
platform_build_args = os.path.join(script_dir, platform.system().lower(), 'args.gn')
argsgn_destination = path.abspath('out/headless/args.gn')
print('Generating platform-specific args')
print('Copying build args: ' + platform_build_args + ' to out/headless/args.gn')
mkdir('out/headless')
shutil.copyfile(platform_build_args, 'out/headless/args.gn')
print(' > cp ' + argsgn_file + ' ' + argsgn_destination)
shutil.copyfile(argsgn_file, argsgn_destination)
print('Adding target_cpu to args')
f = open('out/headless/args.gn', 'a')
f.write('\rtarget_cpu = "' + arch_name + '"')
f.write('\rtarget_cpu = "' + arch_name + '"\r')
f.close()
runcmd('gn gen out/headless')
@ -67,37 +92,38 @@ if platform.system() != 'Windows' and arch_name != 'arm64':
# Create the zip and generate the md5 hash using filenames like:
# chromium-4747cc2-linux_x64.zip
base_filename = 'out/headless/chromium-' + source_version[:7].strip('.') + '-' + platform.system().lower() + '_' + arch_name
base_filename = 'out/headless/chromium-' + base_version + '-' + platform.system().lower() + '_' + arch_name
zip_filename = base_filename + '.zip'
md5_filename = base_filename + '.md5'
print('Creating ' + zip_filename)
print('Creating ' + path.join(src_path, zip_filename))
archive = zipfile.ZipFile(zip_filename, mode='w', compression=zipfile.ZIP_DEFLATED)
def archive_file(name):
"""A little helper function to write individual files to the zip file"""
from_path = os.path.join('out/headless', name)
to_path = os.path.join('headless_shell-' + platform.system().lower() + '_' + arch_name, name)
from_path = path.join('out/headless', name)
to_path = path.join('headless_shell-' + platform.system().lower() + '_' + arch_name, name)
archive.write(from_path, to_path)
return to_path
# Each platform has slightly different requirements for what dependencies
# must be bundled with the Chromium executable.
if platform.system() == 'Linux':
archive_file('headless_shell')
archive_file(os.path.join('swiftshader', 'libEGL.so'))
archive_file(os.path.join('swiftshader', 'libGLESv2.so'))
archive_file(path.join('swiftshader', 'libEGL.so'))
archive_file(path.join('swiftshader', 'libGLESv2.so'))
if arch_name == 'arm64':
archive_file(os.path.join('swiftshader', 'libEGL.so'))
archive_file(path.join('swiftshader', 'libEGL.so'))
elif platform.system() == 'Windows':
archive_file('headless_shell.exe')
archive_file('dbghelp.dll')
archive_file('icudtl.dat')
archive_file(os.path.join('swiftshader', 'libEGL.dll'))
archive_file(os.path.join('swiftshader', 'libEGL.dll.lib'))
archive_file(os.path.join('swiftshader', 'libGLESv2.dll'))
archive_file(os.path.join('swiftshader', 'libGLESv2.dll.lib'))
archive_file(path.join('swiftshader', 'libEGL.dll'))
archive_file(path.join('swiftshader', 'libEGL.dll.lib'))
archive_file(path.join('swiftshader', 'libGLESv2.dll'))
archive_file(path.join('swiftshader', 'libGLESv2.dll.lib'))
elif platform.system() == 'Darwin':
archive_file('headless_shell')
@ -107,6 +133,6 @@ elif platform.system() == 'Darwin':
archive.close()
print('Creating ' + md5_filename)
print('Creating ' + path.join(src_path, md5_filename))
with open (md5_filename, 'w') as f:
f.write(md5_file(zip_filename))

View file

@ -1,33 +1,45 @@
import os, hashlib
import os, hashlib, platform, sys
# This file contains various utility functions used by the init and build scripts
# Compute the root build and script directory as relative to this file
script_dir = os.path.realpath(os.path.join(__file__, '..'))
root_dir = os.path.realpath(os.path.join(script_dir, '..'))
def runcmdsilent(cmd):
"""Executes a string command in the shell"""
print(' > ' + cmd)
return os.system(cmd)
def runcmd(cmd):
"""Executes a string command in the shell"""
print(cmd)
print(' > ' + cmd)
result = os.system(cmd)
if result != 0:
raise Exception(cmd + ' returned ' + str(result))
def mkdir(dir):
print(' > mkdir -p ' + dir)
"""Makes a directory if it doesn't exist"""
if not os.path.exists(dir):
print('mkdir -p ' + dir)
return os.makedirs(dir)
def md5_file(filename):
"""Builds a hex md5 hash of the given file"""
md5 = hashlib.md5()
with open(filename, 'rb') as f:
for chunk in iter(lambda: f.read(128 * md5.block_size), b''):
with open(filename, 'rb') as f:
for chunk in iter(lambda: f.read(128 * md5.block_size), b''):
md5.update(chunk)
return md5.hexdigest()
def configure_environment():
"""Configures temporary environment variables required by Chromium's build"""
depot_tools_path = os.path.join(root_dir, 'depot_tools')
os.environ['PATH'] = depot_tools_path + os.pathsep + os.environ['PATH']
def configure_environment(arch_name, build_path, src_path):
"""Runs install scripts for deps, and configures temporary environment variables required by Chromium's build"""
if platform.system() == 'Linux':
if arch_name:
print('Running sysroot install script...')
sysroot_cmd = src_path + '/build/linux/sysroot_scripts/install-sysroot.py --arch=' + arch_name
runcmd(sysroot_cmd)
print('Running install-build-deps...')
runcmd(src_path + '/build/install-build-deps.sh')
depot_tools_path = os.path.join(build_path, 'depot_tools')
full_path = depot_tools_path + os.pathsep + os.environ['PATH']
print('Updating PATH for depot_tools: ' + full_path)
os.environ['PATH'] = full_path

View file

@ -1,38 +1,47 @@
import os, platform, sys
from build_util import runcmd, mkdir, md5_file, root_dir, configure_environment
from os import path
from build_util import runcmd, mkdir, md5_file, configure_environment
# This is a cross-platform initialization script which should only be run
# once per environment, and isn't intended to be run directly. You should
# run the appropriate platform init script (e.g. Linux/init.sh) which will
# call this once the platform-specific initialization has completed.
os.chdir(root_dir)
# Set to "arm" to build for ARM on Linux
arch_name = sys.argv[1] if len(sys.argv) >= 2 else 'x64'
build_path = path.abspath(os.curdir)
src_path = path.abspath(path.join(build_path, 'chromium', 'src'))
if arch_name != 'x64' and arch_name != 'arm64':
raise Exception('Unexpected architecture: ' + arch_name)
# Configure git
print('Configuring git globals...')
runcmd('git config --global core.autocrlf false')
runcmd('git config --global core.filemode false')
runcmd('git config --global branch.autosetuprebase always')
# Grab Chromium's custom build tools, if they aren't already installed
# (On Windows, they are installed before this Python script is run)
if not os.path.isdir('depot_tools'):
runcmd('git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git')
# Put depot_tools on the path so we can properly run the fetch command
configure_environment()
if not path.isdir('depot_tools'):
print('Installing depot_tools...')
runcmd('git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git')
else:
print('Updating depot_tools...')
original_dir = os.curdir
os.chdir(path.join(build_path, 'depot_tools'))
runcmd('git checkout master')
runcmd('git pull origin master')
os.chdir(original_dir)
configure_environment(arch_name, build_path, src_path)
# Fetch the Chromium source code
mkdir('chromium')
os.chdir('chromium')
runcmd('fetch chromium')
# Build Linux deps
if platform.system() == 'Linux':
os.chdir('src')
if len(sys.argv) >= 2:
sysroot_cmd = 'build/linux/sysroot_scripts/install-sysroot.py --arch=' + sys.argv[1]
print('Running `' + sysroot_cmd + '`')
runcmd(sysroot_cmd)
runcmd('build/install-build-deps.sh')
chromium_dir = path.join(build_path, 'chromium')
if not path.isdir(chromium_dir):
mkdir(chromium_dir)
os.chdir(chromium_dir)
runcmd('fetch chromium')
else:
print('Directory exists: ' + chromium_dir + '. Skipping chromium fetch.')