Skopeo used by our docker tools was patched to work in the build
sandbox (it used /var/tmp which is not available in the sandbox).
Since this temporary directory can now be set at build time, we remove
the patch from our docker tools.
The extraCommands was, previously, simply put in the body of the script
using nix expansion `${extraCommands}` (which looks exactly like bash
expansion!).
This causes issues like in #34779 where scripts will eventually create
invalid bash.
The solution is to use a script like `run-as-root`.
* * *
Fixes#34779
toPath has confusing semantics and is never necessary; it can always
either just be omitted or replaced by pre-concatenating `/.`. It has
been marked as "!!! obsolete?" for more than 10 years in a C++
comment, hopefully removing it will let us properly deprecate and,
eventually, remove it.
The hack of using `crossConfig` to enforce stricter handling of
dependencies is replaced with a dedicated `strictDeps` for that purpose.
(Experience has shown that my punning was a terrible idea that made more
difficult and embarrising to teach teach.)
Now that is is clear, a few packages now use `strictDeps`, to fix
various bugs:
- bintools-wrapper and cc-wrapper
Regression introduced in 736848723e.
This commit most certainly hasn't been tested with sandboxing enabled
and breaks not only pullImage but also the docker-tools NixOS VM test
because it doesn't find it's certificate path and also relies on
/var/tmp being there.
Fixing the certificate path is the easiest one because it can be done
via environment variable.
I've used overrideAttrs for changing the hardcoded path to /tmp (which
is available in sandboxed builds and even hardcoded in Nix), so that
whenever someone uses Skopeo from all-packages.nix the path is still
/var/tmp.
The reason why this is hardcoded to /var/tmp can be seen in a comment in
vendor/github.com/containers/image/storage/storage_image.go:
Do not use the system default of os.TempDir(), usually /tmp, because
with systemd it could be a tmpfs.
With sandboxed builds this isn't the case, however for using Nix without
NixOS this could turn into a problem if this indeed is the case.
So in the long term this needs to have a proper solution.
In addition to that, I cleaned up the expression a bit.
Tested by building dockerTools.examples.nixFromDockerHub and the
docker-tools NixOS VM test.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @nlewo, @Mic92, @Profpatsch, @globin, @LnL7
The script would parse the output of `git submodule status` but
didn't handle paths with spaces in them. This would result in the
following error when trying to determine the URL of the submodule:
error: key does not contain a section: .url
* fetchs3: add configurable name
Change the default from "foo" to the basename of the s3 URL and make it
configurable.
* fetchs3: fix error on missing credentials.session_token
The session token should default to null instead of failing
* fetchs3: make use of the region argument
Set it to null if you don't want to use it
* fetchs3: prefer local build
Fetcher-types spend more time on network than CPU
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).
An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).
Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
... binutils and gcc add it already anyway.
Without this it's easy to get cross-toolchain paths longer than 256
chars and nix-daemon will then fail to commit them to /nix/store on XFS.
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
(cherry picked from commit ba52ae5048)
Following legacy packing conventions, `isArm` was defined just for
32-bit ARM instruction set. This is confusing to non packagers though,
because Aarch64 is an ARM instruction set.
The official ARM overview for ARMv8[1] is surprisingly not confusing,
given the overall state of affairs for ARM naming conventions, and
offers us a solution. It divides the nomenclature into three levels:
```
ISA: ARMv8 {-A, -R, -M}
/ \
Mode: Aarch32 Aarch64
| / \
Encoding: A64 A32 T32
```
At the top is the overall v8 instruction set archicture. Second are the
two modes, defined by bitwidth but differing in other semantics too, and
buttom are the encodings, (hopefully?) isomorphic if they encode the
same mode.
The 32 bit encodings are mostly backwards compatible with previous
non-Thumb and Thumb encodings, and if so we can pun the mode names to
instead mean "sets of compatable or isomorphic encodings", and then
voilà we have nice names for 32-bit and 64-bit arm instruction sets
which do not use the word ARM so as to not confused either laymen or
experienced ARM packages.
[1]: https://developer.arm.com/products/architecture/a-profile
Adds a couple of useful NetBSD and OpenBSD derivations. Some of these
will be integrated into Nixpkgs later.
Noncomprehensive list:
- netbsd.getent
- netbsd.getconf
- netbsd.fts
- openbsd.mg
- netbsd.compat (can replace libbsd)
Since the script running is a failure condition, we should fail the
build properly, not leaving it up to the missing output to determine
that the build went wrong. This should partly address #38952 — nix
build will print out the build log on non-zero exits.
Also fix numberous bugs, such as:
- Not getting confused on more flags taking file arguments.
- Ensuring children reexport their children, but the original
binary/library doesn't.
- Not spawning children when it turns out we just dynamically link
under the threshold but our total number of inputs exceeeds it.
- Children were always named `libunnamed-*`, when that name was
supposed to be the last resort only.
ld-wrapper's own RPATH check hardcodes `.so`, but darwin uses `.dylib`
*and* (in practice due to lousy build systems) `.so`. We don't care
however because we never inject `--rpath` like that in practice on
Darwin. Hopefully someday we won't on linux either.
Pull request #38470 added support for running/building kernels without
modules. This got merged in 38e04bbf29 but
unfortunately while this works perfectly on kernels without modules it
also makes sure that *every* kernel gets no modules.
So all of our VM tests fail since that merge with something like this:
machine# loading module loop...
machine# modprobe: FATAL: Module loop not found in directory /lib/modules/4.14.33
machine# loading module vfat...
machine# modprobe: FATAL: Module vfat not found in directory /lib/modules/4.14.33
machine# loading module nls_cp437...
machine# modprobe: FATAL: Module nls_cp437 not found in directory /lib/modules/4.14.33
machine# loading module nls_iso8859-1...
machine# modprobe: FATAL: Module nls_iso8859-1 not found in directory /lib/modules/4.14.33
machine# loading module fuse...
machine# modprobe: FATAL: Module fuse not found in directory /lib/modules/4.14.33
machine# loading module dm_mod...
machine# modprobe: FATAL: Module dm_mod not found in directory /lib/modules/4.14.33
I shortly tested this against the "misc" VM test and the test is working
again.
In the long term (and I currently don't have time for this) it would be
better to also have a VM test which tests a kernel without modules.
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @roberth, @7c6f434c
Setting the hash to null is a convenient way to bypass the hash check
while developing. It looks like the ability to do this was inadvertently
removed while adding vendor directory support.
This still checks that the user is explicitly setting the value but
allows null as a valid option.
This is to go to a reproducible image build.
Note without this options image are identical from the Docker point of
view but generated docker archives could have different hashes.
This is to improve image creation reproducibility. Since the nar
format doesn't support hard link, the tar stream of a layer can be
different if a dependency of a layer has been built locally or if it
has been fetched from a binary cache.
If the dependency has been build locally, it can contain hard links
which are encoded in the tar stream. If the dependency has been
fetched from a binary cache, the tar stream doesn't contain any hard
link. So even if the content is the same, tar streams are different.
Resolved the following conflicts (by carefully applying patches from the both
branches since the fork point):
pkgs/development/libraries/epoxy/default.nix
pkgs/development/libraries/gtk+/3.x.nix
pkgs/development/python-modules/asgiref/default.nix
pkgs/development/python-modules/daphne/default.nix
pkgs/os-specific/linux/systemd/default.nix
Per @Ericson2314's suggestion [1], make it more clear that the active
hardenings are decided via whitelist; the blacklist is merely for the
debug messages.
1: 36d5ce41d4 (r133279731)
Existing "mips64el" should be "mipsel".
This is just the barest minimum so that nixpkgs can recognize them as
systems - although required for building individual derivations onto
MIPS boards, it is not sufficient if you want to actually build nixos on
those targets
Previously, cargoUpdateHook was meaningful as it was used
in
[`cargo-fetch-deps`](19d3cf81d3/pkgs/build-support/rust/fetch-cargo-deps (L71)).
However, this entire file was removed in
5f8cf0048e. As far as I can
tell, nothing in the code is using it, but it is still
being passed around:
https://github.com/NixOS/nixpkgs/search?q=cargoUpdateHook&type=Code&utf8=%E2%9C%93
There are, however, legitimate use cases for it. For example,
in some software, some dependencies are not locked in Cargo.toml
and this causes Cargo to try fetching another version of them.
This doesn't work well with vendoring crates.
This hook allows to inject patching or whatever necessary workarounds
in the crate vendoring process. I suppose that's what it was for
in there in the first place.
This patch restores this hook and makes it usable again.
Before the code would fail silently for zero values and with some output for
empties. We now currently handle both via defaulting value to zero and making
`let` return success error code when there's no syntax error.
A separate function for building Bazel-bazed packages. Internally it splits the
build into two phases, fetching and building.
Users are expected to provide `fetchArgs.sha256` -- checksum of fetched
dependencies. Local dependencies should be removed in `fetchArgs.preInstall`.
Overall `fetchArgs` and `buildArgs` can be used to add specific steps to fetch
and build.
`find -executable` finds everything with the executable bit set,
including directories. Thats not harmful in this scenario as `cp` won't
copy those directories, but it does result in a few warning messages.
I originally wrote this for packaging proprietary games in Vuizvui[1]
but I thought it would be generally useful as we have a fair amount of
proprietary software lurking around in nixpkgs, which are a bit tedious
to maintain, especially when the library dependencies change after an
update.
So this setup hook searches for all ELF executables and libraries in the
resulting output paths after install phase and uses patchelf to set the
RPATH and interpreter according to what dependencies are available
inside the builder.
For example consider something like this:
stdenv.mkDerivation {
...
nativeBuildInputs = [ autoPatchelfHook ];
buildInputs = [ mesa zlib ];
...
}
Whenever for example an executable requires mesa or zlib, the RPATH will
automatically be set to the lib dir of the corresponding dependency.
If the library dependency is required at runtime, an attribute called
runtimeDependencies can be used to list dependencies that are added to
all executables that are discovered unconditionally.
Beside this, it also makes initial packaging of proprietary software
easier, because one no longer has to manually figure out the
dependencies in the first place.
[1]: https://github.com/openlab-aux/vuizvui
Signed-off-by: aszlig <aszlig@nix.build>
Closes: #34506
Among other things, this will allow *2nix tools to output plain data
while still being composable with the traditional
callPackage/.override interfaces.
Summary:
According to git-submodule manpage,
"git submodule status" prefixes the hash with a '-' if it is not
initialized, and other chars in other circumstances.
(this is consistent on the various git versions tested)
nix-prefetch-git runs "git submodule init" which does you'd think,
but apparently despite this earlier versions of git before 2.16
would still give the hash the '-' suffix.
In particular this is the behavior when using 2.15 and 2.14.1
from the nixos-17.09 and nixos-17.03 channels respectively.
The script then used awk to drop the first char of the first field
which does the wrong thing when there is no prefix emitted:
while there is a space character before the hash, this is not
part of the field and so we ended up eating the first character
of the hash.
To fix this in a way that also works with the previous behavior,
this commit instead uses awk to grab the hash field
and uses tr to delete any '-' chars should they be present.
This seems to work in my testing, and for example can now
successfully fetch the source for "nginxModules.brotli"
where previously it would generate an error:
fatal: '22564a95d9ab58865a096b8d9f7324ea5f2e03e' is not a commit and a branch 'fetchgit' cannot be created from it
(we dropped a '2' from the beginning of the hash)
We still ensure the old and new ones start, respectfully, with `a/` and
`b/`. Use with `stripLen` to ensure tha the old `a/` and `/b` are gone
if a new prefix is added.
cargo-vendor generates almost the right cargo config. Store it with the
vendored files and patch it on use.
This allows to re-use the generated config when using git dependencies.
- All deps go on the PATH
- CC and Bintools wrappers with their host != depender's host still get their
setup hooks run.
- Environment hooks get applied to all packages
This isn't so elegent, but eases the transition on a very significant
PR.
We now have the information to properly determine the role the
cc-wrapper dependency has, by taking advantage of `offset`. No longer
use the soon-to-be-deprecated crossConfig environment variable, the
temp hack used before this change.
Changes:
* doesn't handle root user separately
* doesn't chdir("/") which makes using it seamless
* only bind mounts, doesn't symlink (i.e. files)
Incidentally, fixes#33106.
It's about two times shorter than the previous version, and much
easier to read/follow through. It uses GLib quite heavily, along with
RAII (available in GCC/Clang).
It would be nice to be able to track Nix requests. It's not trustworthy,
but can be helpful for stats and routing in HTTP logs.
Since `fetchurl` is used so widely, we should "magically" get a UA on
`fetchzip`, `fetchFromGitHub`, and other related fetchers.
Since `fetchurl` is only used for fixed-output derivations, this should
cause no mass rebuild.
User-Agent example: curl/7.57.0 Nixpkgs/18.03
* Wrap LEN macro in parantheses
* Drop env_filter in favor of stateful environ_blacklist_filter,
use execvp instead of execvpe, don't explicitly use environ
* Add argument error logging wherever it makes sense
* Drop strjoin in favor of asprintf
* char* -> const char* where appropriate
* Handle stat errors
* Print user messages with fputs, not errorf
* Abstract away is_str_in (previously bind_blacklisted)
* Cleanup temporary directory on error
* Some minor syntactic and naming changes
Thanks to Jörg Thalheim and Tuomas Tynkkynen for the code review!
Factor a bintools (i.e. binutils / cctools) wrapper out of cc-wrapper. While
only LD is wrapped, the setup hook defines environment variables on behalf of
other utilites.
They aren't meant to be critical (uncatchable) errors.
Tested with nix-env + checkMeta:
[ "x86_64-linux" "i686-linux" "x86_64-darwin" "aarch64-linux" ]
