32da92bdf6 gitian: Improve error handling (Wladimir J. van der Laan)
Pull request description:
Improve error handling in gitian builds:
- Set fail-on-error and pipefail flag, this causes a command to fail when either of the pipe stages fails, not only when the last of the stages fails, so this improves error detection.
- Also use `xargs` instead of `find -exec`, because `find` will not propagate errors in the executed command, but `xargs` will.
This will avoid some issues like #15541 where non-determinism is silently introduced due to errors caused by environment conditions (such as lack of disk space in that case).
Tree-SHA512: d5d3f22ce2d04a75e5c25e935744327c3adc704c2d303133f2918113573a564dff3d3243d5569a2b93ee7eb0e97f8e1b1ba81767e966af9015ea711a14091035
284f424d5a Fix osslsigncode compile issue in gitian-build (Chun Kuan Lee)
Pull request description:
Install libssl1.0-dev that is compatible with osslsigncode.
Fixes#13762
Verifed that this gitian descriptor file can sign 0.16.2rc2.
Tree-SHA512: 3029b86e77567a4e033b5ad95826e60df12a0486ac3c4afcac48218f5c76ba49e7f1c1307ce93ffc465ca2f24e12c401e4542929263688e4bd6521aeca3ff73b
6b5506a286 Fix Qt's rcc determinism for depends/gitian (Fuzzbawls)
Pull request description:
With the update to Qt 5.9 having been merged, Qt's `rcc` tool now embeds a file's last modified time in it's output. Since the build system generates temporary files for all locale translations (`*.qm` files) at build time, the resulting `qrc_bitcoin_locale.cpp` file was always being generated in a non-deterministic way.
This is a backport of https://bugreports.qt.io/browse/QTBUG-62511, which is included in Qt versions 5.11+, that allows for an environment variable (`QT_RCC_SOURCE_DATE_OVERRIDE`) to override the behavior described above. This environment variable is in turn set in the gitian descriptors, as that is where determinism is vital for release purposes.
Prior to this, the `qt_libbitcoinqt_a-qrc_bitcoin_locale.o` object file (included into `libbitcoinqt.a`) was returning a different `sha256sum` for each and every build, regardless of file contents change, thus breaking determinism in the resulting binaries.
This should fix#13731
Tree-SHA512: 174017e41f9afc3950ef54a9419de81577ec900db9aec3c78ccd3d879c6aecaaeb944fde0615b933f43e6ca9d7898a27ec071cdd0b91cb772755a3012de96725
Release version strings were broken in Gitian by 7522. This is a minimal fix
suitable for 0.15.
After this, we should fix up version handling for good so that gitian packages
the correct string in the release tarball, so that git is not required to get
the tag name.
1d8df0141 Fix MD formatting in REST-interface.md and spelling mistake in test_runner.py (MeshCollider)
41f3e84aa Fix inconsistencies and grammar in various files (MeshCollider)
Pull request description:
Just a simple fix of some inconsistent capitalization, formatting and grammar in a few files (no code changes)
Tree-SHA512: 60b12a5a5c69a1af4a25b7db0b32ed806ed62ad2966cee08b3792a7cfa7f51848fd485349b4c09e60a7eedfdf55ee730c51daa066d6e226ae404c93342bf3e13
ed1fcdc Bugfix: Detect genbuild.sh in repo correctly (Luke Dashjr)
e98e3dd Bugfix: Only use git for build info if the repository is actually the right one (Luke Dashjr)
Tree-SHA512: 510d7ec8cfeff4e8e0c7ac53631eb32c7acaada7017e7f8cc2e6f60d86afe1cd131870582e01022f961c85a783a130bcb8fef971f8b110070c9c02afda020726
09fe2d9 release: update docs to show basic codesigning procedure (Cory Fields)
f642753 release: create a bundle for the new signing script (Cory Fields)
0068361 release: add win detached sig creator and our cert chain (Cory Fields)
Tree-SHA512: 032ad84697c70faaf857b9187f548282722cffca95d658e36413dc048ff02d9183253373254ffcc1158afb71140753f35abfc9fc8781ea5329c04d13c98759c0
Also change the mac filename to match
The procedure remains the same, but now there's a nifty script to automate
the signing process.
Future steps:
- Build osslsigncode in the gitian-win descriptor so that the signer itself is
deterministic.
- Verify in the gitian-win-signer descriptor that the expected cert chain was
used.
The consistency is helpful for gauging Gitian build progress. Right now it's necessary to remember which platform builds in which order, which can be confusing if you're attempting to get a quick idea of how far along your builds are.
C_INCLUDE_PATH and CPLUS_INCLUDE_PATH work globally as though -isystem was used
for each invocation.
Since that changes the build results, force a rebuild of x86 depends by adding
the value to $HOST_ID_SALT.
- create a script to handle split debug. This will also eventually need to check
targets, and use dsymutil for osx.
- update config.guess/config.sub for bdb for aarch64.
- temporarily disable symbol checks for arm/aarch64
- quit renaming to linux32/linux64 and use the host directly
This also adds a hack to work around an Ubuntu bug in the gcc-multilib package:
https://bugs.launchpad.net/ubuntu/+source/gcc-defaults-armhf-cross/+bug/1347820
The problem is that gcc-multilib conflicts with the aarch toolchain.
gcc-multilib installs a symlink that points
/usr/include/asm -> /usr/include/x86_64-linux-gnu/asm.
Without this link, gcc -m32 can't find asm/errno.h (and others), since
/usr/include/x86_64-linux-gnu isn't in its default include path. But
/usr/include/i386-linux-gnu is (though it doesn't exist on disk).
So work around the problem by linking
/usr/include/i386-linux-gnu/asm -> /usr/include/x86_64-linux-gnu/asm.
The symlink fix is actually quite reasonable, but echoing the password into
sudo is nasty, and should probably be addressed in gitian itself. It makes more
sense to enable passwordless sudo for the build user by default.
The -debug tarballs/zips contain detached debugging symbols. To use them, place
in the same dir as the target binary, and invoke gdb as usual.
Also, because the debug symbols add a substantial space requirement, the build
dirs are now deleted when they're no longer needed.
This removes the following executables from the binary gitian release:
- test_bitcoin-qt[.exe]
- bench_bitcoin[.exe]
@jonasschnelli and me discussed this on IRC a few days ago - unlike the
normal `bitcoin_tests` which is useful to see if it is safe to run
bitcoin on a certain OS/environment combination, there is no good reason
to include these. Better to leave them out to reduce the download
size.
Sizes from the 0.12 release:
```
2.4M bitcoin-0.12.0/bin/bench_bitcoin.exe
22M bitcoin-0.12.0/bin/test_bitcoin-qt.exe
```
Common sentiment is that the miniupnpc codebase likely contains further
vulnerabilities.
I'd prefer to get rid of the dependency completely, but a compromise for
now is to at least disable it by default.
Rather than fetching a signature.tar.gz from somewhere on the net, instruct
Gitian to use a signature from a tag in the bitcoin-detached-sigs repository
which corresponds to the tag of the release being built.
This changes detached-sig-apply.sh to take a dirname rather than a tarball as
an argument, though detached-sig-create.sh still outputs a tarball for
convenience.
For Gitian releases:
- Windows builds remain unchanged. libstdc++ was already linked statically.
- OSX builds remain unchanged. libstdc++ is tied to the SDK and not worth
messing with.
- Linux builds now statically link libstdc++.
For Travis:
- Match the previous behavior by adding --enable-reduce-exports as
necessary.
- Use static libstdc++ for the full Linux build.
Since permissions and timestamps are changed for the sake of determinism,
. must not be added to the archive. Otherwise, tar may try to modify pwd when
extracting.
Descriptors now make use of the dependencies builder, so results are cached.
A very new version (>= e9741525c) of Gitian should be used in order to take
advantage of caching.
Upgrade for https://www.openssl.org/news/secadv_20140605.txt
Just in case - there is no vulnerability that affects ecdsa signing or
verification.
The MITM attack vulnerability (CVE-2014-0224) may have some effect on
our usage of SSL/TLS.
As long as payment requests are signed (which is the common case), usage
of the payment protocol should also not be affected.
The TLS usage in RPC may be at risk for MITM attacks. If you have
`-rpcssl` enabled, be sure to update OpenSSL as soon as possible.
A qt installation date snuck into the host utils (lrelease etc)
This doesn't affect the end product, so no dependency version bump.
It also doesn't explain why gavin's and mine build is different
If the `libqt4-dev` package is installed it picks the moc executable
from the system instead of our custom-built one. This results in
compatibility errors.
This commit convinces configure to pick the right one.
This avoids conflicts between the libraries statically linked into bitcoin and any
libraries we may link dynamically (such as Qt and OpenSSL, see issue #4094).
It also avoids start-up overhead to not export any unnecessary symbols.
To do this, build a linker script that marks all symbols as local.
Should make it possible to run the resulting GUI executable on
Linux distributions that use Qt 4.6, such as Debian Wheezy and Tails.
Builds a mini-SDK for building against Qt 4.6. This includes the headers
as well as host utilities such as `lrelease`, `qrc` and `moc`.
This speeds up the gitian build a bit - libqt4-dev pulled in a lot of packages,
and is no longer needed as this provides a replacement of our own.
Note: This does not replace the Qt build with at static library. After this
commit we still build dynamically against the system Qt library. The only
difference is that compatibility with an older version is maintained. This
loses minor GUI functionality (such as setPlaceholderText) but still
allows integration into the window management of the host OS, unlike
when statically linking.
Bumps deps-linux, deps-win dependency versions as well.
qt-win does not need to be bumped, as although it depends on deps-win,
Qt doesn't use miniupnp. I verified this by rebuilding the dependency
and checking the the output is the same. Not having to rebuild Qt is a
good thing as it is huge.
IIRC this was the case with 0.8.6, so let's keep this to avoid the risk
of losing connectable nodes with 0.9 release.
Also our miniupnpc library was recently updated and I've heard
reports that it works better than before now.
