rustPlatform.importCargoLock: add allowBuiltinFetchGit option
25 KiB
Rust
To install the rust compiler and cargo put
environment.systemPackages = [
rustc
cargo
];
into your configuration.nix
or bring them into scope with nix-shell -p rustc cargo
.
For other versions such as daily builds (beta and nightly),
use either rustup
from nixpkgs (which will manage the rust installation in your home directory),
or use a community maintained Rust overlay.
buildRustPackage
: Compiling Rust applications with Cargo
Rust applications are packaged by using the buildRustPackage
helper from rustPlatform
:
{ lib, fetchFromGitHub, rustPlatform }:
rustPlatform.buildRustPackage rec {
pname = "ripgrep";
version = "12.1.1";
src = fetchFromGitHub {
owner = "BurntSushi";
repo = pname;
rev = version;
hash = "sha256-+s5RBC3XSgb8omTbUNLywZnP6jSxZBKSS1BmXOjRF8M=";
};
cargoHash = "sha256-jtBw4ahSl88L0iuCXxQgZVm1EcboWRJMNtjxLVTtzts=";
meta = with lib; {
description = "A fast line-oriented regex search tool, similar to ag and ack";
homepage = "https://github.com/BurntSushi/ripgrep";
license = licenses.unlicense;
maintainers = [ maintainers.tailhook ];
};
}
buildRustPackage
requires either the cargoSha256
or the
cargoHash
attribute which is computed over all crate sources of this
package. cargoHash256
is used for traditional Nix SHA-256 hashes,
such as the one in the example above. cargoHash
should instead be
used for SRI hashes. For example:
cargoHash = "sha256-l1vL2ZdtDRxSGvP0X/l3nMw8+6WF67KPutJEzUROjg8=";
Both types of hashes are permitted when contributing to nixpkgs. The
Cargo hash is obtained by inserting a fake checksum into the
expression and building the package once. The correct checksum can
then be taken from the failed build. A fake hash can be used for
cargoSha256
as follows:
cargoSha256 = lib.fakeSha256;
For cargoHash
you can use:
cargoHash = lib.fakeHash;
Per the instructions in the Cargo Book
best practices guide, Rust applications should always commit the Cargo.lock
file in git to ensure a reproducible build. However, a few packages do not, and
Nix depends on this file, so if it is missing you can use cargoPatches
to
apply it in the patchPhase
. Consider sending a PR upstream with a note to the
maintainer describing why it's important to include in the application.
The fetcher will verify that the Cargo.lock
file is in sync with the src
attribute, and fail the build if not. It will also will compress the vendor
directory into a tar.gz archive.
The tarball with vendored dependencies contains a directory with the
package's name
, which is normally composed of pname
and
version
. This means that the vendored dependencies hash
(cargoSha256
/cargoHash
) is dependent on the package name and
version. The cargoDepsName
attribute can be used to use another name
for the directory of vendored dependencies. For example, the hash can
be made invariant to the version by setting cargoDepsName
to
pname
:
rustPlatform.buildRustPackage rec {
pname = "broot";
version = "1.2.0";
src = fetchCrate {
inherit pname version;
sha256 = "sha256-aDQA4A5mScX9or3Lyiv/5GyAehidnpKKE0grhbP1Ctc=";
};
cargoHash = "sha256-tbrTbutUs5aPSV+yE0IBUZAAytgmZV7Eqxia7g+9zRs=";
cargoDepsName = pname;
# ...
}
Importing a Cargo.lock
file
Using cargoSha256
or cargoHash
is tedious when using
buildRustPackage
within a project, since it requires that the hash
is updated after every change to Cargo.lock
. Therefore,
buildRustPackage
also supports vendoring dependencies directly from
a Cargo.lock
file using the cargoLock
argument. For example:
rustPlatform.buildRustPackage {
pname = "myproject";
version = "1.0.0";
cargoLock = {
lockFile = ./Cargo.lock;
};
# ...
}
This will retrieve the dependencies using fixed-output derivations from the specified lockfile.
One caveat is that Cargo.lock
cannot be patched in the patchPhase
because it runs after the dependencies have already been fetched. If
you need to patch or generate the lockfile you can alternatively set
cargoLock.lockFileContents
to a string of its contents:
rustPlatform.buildRustPackage {
pname = "myproject";
version = "1.0.0";
cargoLock = let
fixupLockFile = path: f (builtins.readFile path);
in {
lockFileContents = fixupLockFile ./Cargo.lock;
};
# ...
}
Note that setting cargoLock.lockFile
or cargoLock.lockFileContents
doesn't add a Cargo.lock
to your src
, and a Cargo.lock
is still
required to build a rust package. A simple fix is to use:
postPatch = ''
cp ${./Cargo.lock} Cargo.lock
'';
The output hash of each dependency that uses a git source must be
specified in the outputHashes
attribute. For example:
rustPlatform.buildRustPackage rec {
pname = "myproject";
version = "1.0.0";
cargoLock = {
lockFile = ./Cargo.lock;
outputHashes = {
"finalfusion-0.14.0" = "17f4bsdzpcshwh74w5z119xjy2if6l2wgyjy56v621skr2r8y904";
};
};
# ...
}
If you do not specify an output hash for a git dependency, building
the package will fail and inform you of which crate needs to be
added. To find the correct hash, you can first use lib.fakeSha256
or
lib.fakeHash
as a stub hash. Building the package (and thus the
vendored dependencies) will then inform you of the correct hash.
For usage outside nixpkgs, allowBuiltinFetchGit
could be used to
avoid having to specify outputHashes
. For example:
rustPlatform.buildRustPackage rec {
pname = "myproject";
version = "1.0.0";
cargoLock = {
lockFile = ./Cargo.lock;
allowBuiltinFetchGit = true;
};
# ...
}
Cargo features
You can disable default features using buildNoDefaultFeatures
, and
extra features can be added with buildFeatures
.
If you want to use different features for check phase, you can use
checkNoDefaultFeatures
and checkFeatures
. They are only passed to
cargo test
and not cargo build
. If left unset, they default to
buildNoDefaultFeatures
and buildFeatures
.
For example:
rustPlatform.buildRustPackage rec {
pname = "myproject";
version = "1.0.0";
buildNoDefaultFeatures = true;
buildFeatures = [ "color" "net" ];
# disable network features in tests
checkFeatures = [ "color" ];
# ...
}
Cross compilation
By default, Rust packages are compiled for the host platform, just like any
other package is. The --target
passed to rust tools is computed from this.
By default, it takes the stdenv.hostPlatform.config
and replaces components
where they are known to differ. But there are ways to customize the argument:
-
To choose a different target by name, define
stdenv.hostPlatform.rustc.config
as that name (a string), and that name will be used instead.For example:
import <nixpkgs> { crossSystem = (import <nixpkgs/lib>).systems.examples.armhf-embedded // { rustc.config = "thumbv7em-none-eabi"; }; }
will result in:
--target thumbv7em-none-eabi
-
To pass a completely custom target, define
stdenv.hostPlatform.rustc.config
with its name, andstdenv.hostPlatform.rustc.platform
with the value. The value will be serialized to JSON in a file called${stdenv.hostPlatform.rustc.config}.json
, and the path of that file will be used instead.For example:
import <nixpkgs> { crossSystem = (import <nixpkgs/lib>).systems.examples.armhf-embedded // { rustc.config = "thumb-crazy"; rustc.platform = { foo = ""; bar = ""; }; }; }
will result in:
--target /nix/store/asdfasdfsadf-thumb-crazy.json # contains {"foo":"","bar":""}
Note that currently custom targets aren't compiled with std
, so cargo test
will fail. This can be ignored by adding doCheck = false;
to your derivation.
Running package tests
When using buildRustPackage
, the checkPhase
is enabled by default and runs
cargo test
on the package to build. To make sure that we don't compile the
sources twice and to actually test the artifacts that will be used at runtime,
the tests will be ran in the release
mode by default.
However, in some cases the test-suite of a package doesn't work properly in the
release
mode. For these situations, the mode for checkPhase
can be changed like
so:
rustPlatform.buildRustPackage {
/* ... */
checkType = "debug";
}
Please note that the code will be compiled twice here: once in release
mode
for the buildPhase
, and again in debug
mode for the checkPhase
.
Test flags, e.g., --package foo
, can be passed to cargo test
via the
cargoTestFlags
attribute.
Another attribute, called checkFlags
, is used to pass arguments to the test
binary itself, as stated
here.
Tests relying on the structure of the target/
directory
Some tests may rely on the structure of the target/
directory. Those tests
are likely to fail because we use cargo --target
during the build. This means that
the artifacts
are stored in target/<architecture>/release/
,
rather than in target/release/
.
This can only be worked around by patching the affected tests accordingly.
Disabling package-tests
In some instances, it may be necessary to disable testing altogether (with doCheck = false;
):
- If no tests exist -- the
checkPhase
should be explicitly disabled to skip unnecessary build steps to speed up the build. - If tests are highly impure (e.g. due to network usage).
There will obviously be some corner-cases not listed above where it's sensible to disable tests. The above are just guidelines, and exceptions may be granted on a case-by-case basis.
However, please check if it's possible to disable a problematic subset of the test suite and leave a comment explaining your reasoning.
This can be achieved with --skip
in checkFlags
:
rustPlatform.buildRustPackage {
/* ... */
checkFlags = [
# reason for disabling test
"--skip=example::tests:example_test"
];
}
Using cargo-nextest
Tests can be run with cargo-nextest
by setting useNextest = true
. The same options still apply, but nextest
accepts a different set of arguments and the settings might need to be
adapted to be compatible with cargo-nextest.
rustPlatform.buildRustPackage {
/* ... */
useNextest = true;
}
Setting test-threads
buildRustPackage
will use parallel test threads by default,
sometimes it may be necessary to disable this so the tests run consecutively.
rustPlatform.buildRustPackage {
/* ... */
dontUseCargoParallelTests = true;
}
Building a package in debug
mode
By default, buildRustPackage
will use release
mode for builds. If a package
should be built in debug
mode, it can be configured like so:
rustPlatform.buildRustPackage {
/* ... */
buildType = "debug";
}
In this scenario, the checkPhase
will be ran in debug
mode as well.
Custom build
/install
-procedures
Some packages may use custom scripts for building/installing, e.g. with a Makefile
.
In these cases, it's recommended to override the buildPhase
/installPhase
/checkPhase
.
Otherwise, some steps may fail because of the modified directory structure of target/
.
Building a crate with an absent or out-of-date Cargo.lock file
buildRustPackage
needs a Cargo.lock
file to get all dependencies in the
source code in a reproducible way. If it is missing or out-of-date one can use
the cargoPatches
attribute to update or add it.
rustPlatform.buildRustPackage rec {
(...)
cargoPatches = [
# a patch file to add/update Cargo.lock in the source code
./add-Cargo.lock.patch
];
}
Compiling non-Rust packages that include Rust code
Several non-Rust packages incorporate Rust code for performance- or
security-sensitive parts. rustPlatform
exposes several functions and
hooks that can be used to integrate Cargo in non-Rust packages.
Vendoring of dependencies
Since network access is not allowed in sandboxed builds, Rust crate
dependencies need to be retrieved using a fetcher. rustPlatform
provides the fetchCargoTarball
fetcher, which vendors all
dependencies of a crate. For example, given a source path src
containing Cargo.toml
and Cargo.lock
, fetchCargoTarball
can be used as follows:
cargoDeps = rustPlatform.fetchCargoTarball {
inherit src;
hash = "sha256-BoHIN/519Top1NUBjpB/oEMqi86Omt3zTQcXFWqrek0=";
};
The src
attribute is required, as well as a hash specified through
one of the hash
attribute. The following optional attributes can
also be used:
name
: the name that is used for the dependencies tarball. Ifname
is not specified, then the namecargo-deps
will be used.sourceRoot
: when theCargo.lock
/Cargo.toml
are in a subdirectory,sourceRoot
specifies the relative path to these files.patches
: patches to apply before vendoring. This is useful when theCargo.lock
/Cargo.toml
files need to be patched before vendoring.
If a Cargo.lock
file is available, you can alternatively use the
importCargoLock
function. In contrast to fetchCargoTarball
, this
function does not require a hash (unless git dependencies are used)
and fetches every dependency as a separate fixed-output derivation.
importCargoLock
can be used as follows:
cargoDeps = rustPlatform.importCargoLock {
lockFile = ./Cargo.lock;
};
If the Cargo.lock
file includes git dependencies, then their output
hashes need to be specified since they are not available through the
lock file. For example:
cargoDeps = rustPlatform.importCargoLock {
lockFile = ./Cargo.lock;
outputHashes = {
"rand-0.8.3" = "0ya2hia3cn31qa8894s3av2s8j5bjwb6yq92k0jsnlx7jid0jwqa";
};
};
If you do not specify an output hash for a git dependency, building
cargoDeps
will fail and inform you of which crate needs to be
added. To find the correct hash, you can first use lib.fakeSha256
or
lib.fakeHash
as a stub hash. Building cargoDeps
will then inform
you of the correct hash.
Hooks
rustPlatform
provides the following hooks to automate Cargo builds:
cargoSetupHook
: configure Cargo to use dependencies vendored throughfetchCargoTarball
. This hook uses thecargoDeps
environment variable to find the vendored dependencies. If a project already vendors its dependencies, the variablecargoVendorDir
can be used instead. When theCargo.toml
/Cargo.lock
files are not insourceRoot
, then the optionalcargoRoot
is used to specify the Cargo root directory relative tosourceRoot
.cargoBuildHook
: use Cargo to build a crate. If the crate to be built is a crate in e.g. a Cargo workspace, the relative path to the crate to build can be set through the optionalbuildAndTestSubdir
environment variable. Features can be specified withcargoBuildNoDefaultFeatures
andcargoBuildFeatures
. Additional Cargo build flags can be passed throughcargoBuildFlags
.maturinBuildHook
: use Maturin to build a Python wheel. Similar tocargoBuildHook
, the optional variablebuildAndTestSubdir
can be used to build a crate in a Cargo workspace. Additional Maturin flags can be passed throughmaturinBuildFlags
.cargoCheckHook
: run tests using Cargo. The build type for checks can be set usingcargoCheckType
. Features can be specified withcargoCheckNoDefaultFeatures
andcargoCheckFeatures
. Additional flags can be passed to the tests usingcheckFlags
andcheckFlagsArray
. By default, tests are run in parallel. This can be disabled by settingdontUseCargoParallelTests
.cargoNextestHook
: run tests using cargo-nextest. The same options forcargoCheckHook
also applies tocargoNextestHook
.cargoInstallHook
: install binaries and static/shared libraries that were built usingcargoBuildHook
.bindgenHook
: for crates which usebindgen
as a build dependency, letsbindgen
findlibclang
andlibclang
find the libraries inbuildInputs
.
Examples
Python package using setuptools-rust
For Python packages using setuptools-rust
, you can use
fetchCargoTarball
and cargoSetupHook
to retrieve and set up Cargo
dependencies. The build itself is then performed by
buildPythonPackage
.
The following example outlines how the tokenizers
Python package is
built. Since the Python package is in the source/bindings/python
directory of the tokenizers
project's source archive, we use
sourceRoot
to point the tooling to this directory:
{ fetchFromGitHub
, buildPythonPackage
, rustPlatform
, setuptools-rust
}:
buildPythonPackage rec {
pname = "tokenizers";
version = "0.10.0";
src = fetchFromGitHub {
owner = "huggingface";
repo = pname;
rev = "python-v${version}";
hash = "sha256-rQ2hRV52naEf6PvRsWVCTN7B1oXAQGmnpJw4iIdhamw=";
};
cargoDeps = rustPlatform.fetchCargoTarball {
inherit src sourceRoot;
name = "${pname}-${version}";
hash = "sha256-miW//pnOmww2i6SOGbkrAIdc/JMDT4FJLqdMFojZeoY=";
};
sourceRoot = "source/bindings/python";
nativeBuildInputs = [ setuptools-rust ] ++ (with rustPlatform; [
cargoSetupHook
rust.cargo
rust.rustc
]);
# ...
}
In some projects, the Rust crate is not in the main Python source
directory. In such cases, the cargoRoot
attribute can be used to
specify the crate's directory relative to sourceRoot
. In the
following example, the crate is in src/rust
, as specified in the
cargoRoot
attribute. Note that we also need to specify the correct
path for fetchCargoTarball
.
{ buildPythonPackage
, fetchPypi
, rustPlatform
, setuptools-rust
, openssl
}:
buildPythonPackage rec {
pname = "cryptography";
version = "3.4.2"; # Also update the hash in vectors.nix
src = fetchPypi {
inherit pname version;
hash = "sha256-xGDilsjLOnls3MfVbGKnj80KCUCczZxlis5PmHzpNcQ=";
};
cargoDeps = rustPlatform.fetchCargoTarball {
inherit src;
sourceRoot = "${pname}-${version}/${cargoRoot}";
name = "${pname}-${version}";
hash = "sha256-PS562W4L1NimqDV2H0jl5vYhL08H9est/pbIxSdYVfo=";
};
cargoRoot = "src/rust";
# ...
}
Python package using maturin
Python packages that use Maturin
can be built with fetchCargoTarball
, cargoSetupHook
, and
maturinBuildHook
. For example, the following (partial) derivation
builds the retworkx
Python package. fetchCargoTarball
and
cargoSetupHook
are used to fetch and set up the crate dependencies.
maturinBuildHook
is used to perform the build.
{ lib
, buildPythonPackage
, rustPlatform
, fetchFromGitHub
}:
buildPythonPackage rec {
pname = "retworkx";
version = "0.6.0";
src = fetchFromGitHub {
owner = "Qiskit";
repo = "retworkx";
rev = version;
hash = "sha256-11n30ldg3y3y6qxg3hbj837pnbwjkqw3nxq6frds647mmmprrd20=";
};
cargoDeps = rustPlatform.fetchCargoTarball {
inherit src;
name = "${pname}-${version}";
hash = "sha256-heOBK8qi2nuc/Ib+I/vLzZ1fUUD/G/KTw9d7M4Hz5O0=";
};
format = "pyproject";
nativeBuildInputs = with rustPlatform; [ cargoSetupHook maturinBuildHook ];
# ...
}
Setting Up nix-shell
Oftentimes you want to develop code from within nix-shell
. Unfortunately
buildRustCrate
does not support common nix-shell
operations directly
(see this issue)
so we will use stdenv.mkDerivation
instead.
Using the example hello
project above, we want to do the following:
- Have access to
cargo
andrustc
- Have the
openssl
library available to a crate through it's normal compilation mechanism (pkg-config
).
A typical shell.nix
might look like:
with import <nixpkgs> {};
stdenv.mkDerivation {
name = "rust-env";
nativeBuildInputs = [
rustc cargo
# Example Build-time Additional Dependencies
pkg-config
];
buildInputs = [
# Example Run-time Additional Dependencies
openssl
];
# Set Environment Variables
RUST_BACKTRACE = 1;
}
You should now be able to run the following:
$ nix-shell --pure
$ cargo build
$ cargo test
Controlling Rust Version Inside nix-shell
To control your rust version (i.e. use nightly) from within shell.nix
(or
other nix expressions) you can use the following shell.nix
# Latest Nightly
with import <nixpkgs> {};
let src = fetchFromGitHub {
owner = "mozilla";
repo = "nixpkgs-mozilla";
# commit from: 2019-05-15
rev = "9f35c4b09fd44a77227e79ff0c1b4b6a69dff533";
hash = "sha256-18h0nvh55b5an4gmlgfbvwbyqj91bklf1zymis6lbdh75571qaz0=";
};
in
with import "${src.out}/rust-overlay.nix" pkgs pkgs;
stdenv.mkDerivation {
name = "rust-env";
buildInputs = [
# Note: to use stable, just replace `nightly` with `stable`
latest.rustChannels.nightly.rust
# Add some extra dependencies from `pkgs`
pkg-config openssl
];
# Set Environment Variables
RUST_BACKTRACE = 1;
}
Now run:
$ rustc --version
rustc 1.26.0-nightly (188e693b3 2018-03-26)
To see that you are using nightly.
Using community Rust overlays
There are two community maintained approaches to Rust toolchain management:
Oxalica's overlay allows you to select a particular Rust version and components. See their documentation for more detailed usage.
Fenix is an alternative to rustup
and can also be used as an overlay.
Both oxalica's overlay and fenix better integrate with nix and cache optimizations.
Because of this and ergonomics, either of those community projects
should be preferred to the Mozilla's Rust overlay (nixpkgs-mozilla
).
How to select a specific rustc
and toolchain version
You can consume the oxalica overlay and use it to grab a specific Rust toolchain version.
Here is an example shell.nix
showing how to grab the current stable toolchain:
{ pkgs ? import <nixpkgs> {
overlays = [
(import (fetchTarball "https://github.com/oxalica/rust-overlay/archive/master.tar.gz"))
];
}
}:
pkgs.mkShell {
nativeBuildInputs = with pkgs; [
pkg-config
rust-bin.stable.latest.minimal
];
}
You can try this out by:
- Saving that to
shell.nix
- Executing
nix-shell --pure --command 'rustc --version'
As of writing, this prints out rustc 1.56.0 (09c42c458 2021-10-18)
.
How to use an overlay toolchain in a derivation
You can also use an overlay's Rust toolchain with buildRustPackage
.
The below snippet demonstrates invoking buildRustPackage
with an oxalica overlay selected Rust toolchain:
with import <nixpkgs> {
overlays = [
(import (fetchTarball "https://github.com/oxalica/rust-overlay/archive/master.tar.gz"))
];
};
rustPlatform.buildRustPackage rec {
pname = "ripgrep";
version = "12.1.1";
nativeBuildInputs = [
rust-bin.stable.latest.minimal
];
src = fetchFromGitHub {
owner = "BurntSushi";
repo = "ripgrep";
rev = version;
hash = "sha256-1hqps7l5qrjh9f914r5i6kmcz6f1yb951nv4lby0cjnp5l253kps=";
};
cargoSha256 = "03wf9r2csi6jpa7v5sw5lpxkrk4wfzwmzx7k3991q3bdjzcwnnwp";
meta = with lib; {
description = "A fast line-oriented regex search tool, similar to ag and ack";
homepage = "https://github.com/BurntSushi/ripgrep";
license = licenses.unlicense;
maintainers = [ maintainers.tailhook ];
};
}
Follow the below steps to try that snippet.
- create a new directory
- save the above snippet as
default.nix
in that directory - cd into that directory and run
nix-build
Rust overlay installation
You can use this overlay by either changing your local nixpkgs configuration,
or by adding the overlay declaratively in a nix expression, e.g. in configuration.nix
.
For more information see the manual on installing overlays.
Declarative Rust overlay installation
This snippet shows how to use oxalica's Rust overlay.
Add the following to your configuration.nix
, home-configuration.nix
, shell.nix
, or similar:
{ pkgs ? import <nixpkgs> {
overlays = [
(import (builtins.fetchTarball "https://github.com/oxalica/rust-overlay/archive/master.tar.gz"))
# Further overlays go here
];
};
};
Note that this will fetch the latest overlay version when rebuilding your system.