Commit graph

1284 commits

Author SHA1 Message Date
Alex Willmer 47b49a05ea
contrib: Fix SyntaxWarning in Python base58 implementation
In Python integers should be compared for equality (`i == j`), not identity (`i is j`). Recent versions of CPython 3.x emit a SyntaxWarning when they encounter this incorrect usage, e.g.

```
$ python3 base58.py 
base58.py:110: SyntaxWarning: "is" with a literal. Did you mean "=="?
  assert get_bcaddress_version('15VjRaDX9zpbA8LVnbrCAFzrVzN7ixHNsC') is 0
Tests passed
```
2020-06-15 21:36:54 +01:00
MarcoFalke fa4cd1fdae
ci: Switch to bitcoincore.org download 2020-06-07 10:30:38 -04:00
MarcoFalke fab7d954b2
test: Make valgrind.supp work on aarch64
The equivalent suppression on aarch64 looks like:

{
   <insert_a_suppression_name_here>
   Memcheck:Param
   pwrite64(buf)
   fun:__libc_pwrite64
   fun:pwrite
   fun:__os_io
   obj:/usr/lib/aarch64-linux-gnu/libdb_cxx-5.3.so
   fun:__log_flush_int
   fun:__log_flush
   fun:__memp_sync_int
   fun:__db_sync
   fun:__db_refresh
   fun:__db_close
   fun:__fop_subdb_setup
   fun:__db_open
   fun:__db_open_pp
}
2020-06-03 10:58:27 -04:00
fanquake 17df15efb5
Merge #18958: guix: Make V=1 more powerful for debugging
f852761aec guix: Add clarifying documentation for V env var (Carl Dong)
85f4a4b082 guix: Make V=1 more powerful for debugging (Carl Dong)

Pull request description:

  ```
  - Print commands in both unexpanded and expanded forms
  - Set VERBOSE=1 for CMake
  ```

  Ping MarcoFalke hopefully you use `V=1` already for the Guix builds on DrahtBot?

ACKs for top commit:
  fanquake:
    ACK f852761aec. Ran a Windows Guix build and compared the output from master and this PR when using `V=1`. i.e `HOSTS=x86_64-w64-mingw32 PATH="/root/.config/guix/current/bin${PATH:+:}$PATH" V=1 ./contrib/guix/guix-build.sh`.

Tree-SHA512: 8bc466fa7b869618bbd5a0a91c6b23d4785009289f8dfb93b0349317463a9ab9ece128c72436e02a0819722a63e703100aed15807867a716fda891292fcb9d9d
2020-05-21 15:09:48 +08:00
Carl Dong f852761aec
guix: Add clarifying documentation for V env var 2020-05-20 13:11:16 -04:00
fanquake eacedfb023
scripts: add additional type annotations to security-check.py 2020-05-14 15:30:52 +08:00
fanquake 83d063e954
scripts: add run_command to security-check.py
Deduplicate all the subprocess code as mentioned in 18713.
2020-05-14 15:29:58 +08:00
fanquake 13f606b4f9
scripts: remove NONFATAL from security-check.py 2020-05-14 14:36:27 +08:00
fanquake 061acf62a1
scripts: no-longer check for 32 bit windows in security-check.py 2020-05-14 14:36:27 +08:00
Carl Dong 85f4a4b082
guix: Make V=1 more powerful for debugging
- Print commands in both unexpanded and expanded forms
- Set VERBOSE=1 for CMake
2020-05-12 09:37:31 -04:00
fanquake d96fdc2a39
Merge #18741: guix: Make source tarball using git-archive
bfe1ba2f5b rel-builds: Specify core.abbrev for git-rev-parse (Carl Dong)
27e63e01cc build: Accomodate makensis v2.x (Carl Dong)
1f2c39a30e guix: Remove logical cores requirement (Carl Dong)
a4f6ffa71e lint: Also enable source statements for non-gitian (Carl Dong)
d256f91cb1 rel-builds: Directly deploy win installer to OUTDIR (Carl Dong)
fa791da02f nsis: Specify OutFile path only once (Carl Dong)
14701604d0 guix: Expose GIT_COMMON_DIR in container as readonly (Carl Dong)
f5a6ac4f48 guix: Make source tarball using git-archive (Carl Dong)
395c1137f6 gitian: Limit sourced script to just assignments (Carl Dong)

Pull request description:

  Based on: #18556
  Related: https://github.com/bitcoin/bitcoin/pull/17595#discussion_r399728721

ACKs for top commit:
  fanquake:
    ACK bfe1ba2f5b - I agree with Carl, and am going to merge this. I'd like for Linux Guix builds to be working again, and we can rebase #18818.

Tree-SHA512: c87ada7e3de17ca0b692a91029b86573442ded5780fc081c214773f6b374a0cdbeaf6f6898c36669c2e247ee32aa7f82defb1180f8decac52c65f0c140f18674
2020-05-06 13:13:36 +08:00
MarcoFalke fbd522721c
Merge #18885: contrib: Move optimize-pngs.py script to the maintainer repo
fa13090d20 contrib: Remove optimize-pngs.py script, which lives in the maintainer repo (MarcoFalke)

Pull request description:

  Moved to https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py

  Bitcoin Core should focus on the full node implementation, not on scripts to compress png images.

  This script is only used when new PNG files are added to the repo. This happens about once every two years. So fetching the script from the other repo should not be a burden, but removing it from this repo is going to cut down on the meta files we need to maintain in the main repo.

ACKs for top commit:
  practicalswift:
    ACK fa13090d20 -- `+0 lines, -82 lines` :)
  promag:
    ACK fa13090d20.
  hebasto:
    ACK fa13090d20, verified that script is already [moved](https://github.com/bitcoin-core/bitcoin-maintainer-tools/pull/56).

Tree-SHA512: 37d111adae769bcddc6ae88041032d5a2b8b228fec67f555c8333c38de3992f5138b30bea868d7d6d6b7f966a47133e5853134373b149ab23cba3b8b560ecb31
2020-05-05 10:40:39 -04:00
MarcoFalke fa13090d20
contrib: Remove optimize-pngs.py script, which lives in the maintainer repo
https://github.com/bitcoin-core/bitcoin-maintainer-tools/blob/master/optimize-pngs.py
2020-05-05 07:25:58 -04:00
fanquake d7120f7f78
valgrind : remove duplicate BCLog::Logger suppression
This was added in #17770, but is identical to the supression above.
2020-05-05 11:10:36 +08:00
fanquake 708e3c7e85
valgrind: remove rest_blockhash_by_height suppression
This should no-longer be necessary after #18785.
2020-05-05 11:10:35 +08:00
Carl Dong bfe1ba2f5b
rel-builds: Specify core.abbrev for git-rev-parse
Chose 12 because the kernel uses it:

  https://public-inbox.org/git/CA+55aFy0_pwtFOYS1Tmnxipw9ZkRNCQHmoYyegO00pjMiZQfbg@mail.gmail.com/raw

And also because it's a nice number.
2020-05-04 13:00:08 -04:00
Carl Dong 1f2c39a30e
guix: Remove logical cores requirement
Thanks MarcoFalke for pushing this to its limits and testing :-)
2020-05-01 12:33:47 -04:00
Carl Dong d256f91cb1
rel-builds: Directly deploy win installer to OUTDIR 2020-05-01 12:30:20 -04:00
Carl Dong 14701604d0
guix: Expose GIT_COMMON_DIR in container as readonly
When using worktrees or submodules, you'll see a `.git' plain text file
at the root of your working tree instead of the usual `.git' directory.

This plain text file will point to the real GIT_DIR, under the
GIT_COMMON_DIR. From experimentation, the full GIT_COMMON_DIR is
required to exist for operations such as git-archive(1), so we expose it
as readonly inside the container.
2020-04-28 10:36:37 -04:00
Carl Dong f5a6ac4f48
guix: Make source tarball using git-archive 2020-04-28 10:36:36 -04:00
Carl Dong 395c1137f6
gitian: Limit sourced script to just assignments
Previously, the sourced script would create the source tarball. Now, it
only assigns variables and the source-ing script has more flexibility in
determining what to do with these variables.

See later commit showing how this flexibility is useful in our Guix
builds.
2020-04-28 10:36:35 -04:00
fanquake 65fb3dfc8d
Merge #18556: build: Drop make dist in gitian builds
2aa48edec0 refactor: Drop unused ${WRAP_DIR}/${HOST} directory (Hennadii Stepanov)
1362be0447 build: Drop make dist in gitian builds (Hennadii Stepanov)

Pull request description:

  After the merge of #18331, the packaged source tarball is created by `git archive`, but the binaries are built from another one which is made by `make dist`.

  With this PR the only source tarball, created by `git archive`, is used both for binaries building and for packaging to users.

  Close #16588.
  Close #18547.

  As a good side-effect, #18349 becomes redundant.

  **Change in behavior**

  The following variables 1b151e3ffc/configure.ac (L2-L6)

  are no longer used for naming of directories and tarballs.

  Instead of them the gitian descriptors use a git tag (if available) or a commit hash.

  ---

  Also a small refactor commit picked from #18404.

ACKs for top commit:
  dongcarl:
    ACK 2aa48edec0
  MarcoFalke:
    ACK 2aa48edec0
  fanquake:
    ACK 2aa48edec0 - I've had a quick look over this, and don't want to block merging if this actually gets as closer to finally having this all sorted out. Obviously we've still got #18741, and after speaking to Carl this morning, there will likely be even more changes after that (not Guix specific).

Tree-SHA512: d3b16f87e48d1790a3264940c28acd5d881bfd10f3ce94fb0c8a6af76d8039289d01e0cd4972adac49ae24362857251f6c1e5e09e3e9fbf636c10708b4015a7c
2020-04-28 16:44:17 +08:00
fanquake ac21090f20
Merge #18629: scripts: add PE .reloc section check to security-check.py
3e38023af7 scripts: add PE .reloc section check to security-check.py (fanquake)

Pull request description:

  The `ld` in binutils has historically had a few issues with PE binaries, there's a good summary in this [thread](https://sourceware.org/bugzilla/show_bug.cgi?id=19011).

  One issue in particular was `ld` stripping the `.reloc` section out of PE binaries, even though it's required for functioning ASLR. This was [reported by a Tor developer in 2014](https://sourceware.org/bugzilla/show_bug.cgi?id=17321) and they have been patching their [own binutils](https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/binutils) ever since. However their patch only made it into binutils at the [start of this year](https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=dc9bd8c92af67947db44b3cb428c050259b15cd0). It adds an `--enable-reloc-section` flag, which is turned on by default if you are using `--dynamic-base`. In the mean time this issue has also been worked around by other projects, such as FFmpeg, see [this commit](91b668acd6).

  I have checked our recent supported Windows release binaries, and they do contain a `.reloc` section. From what I understand, we are using all the right compile/linker flags, including `-pie` & `-fPIE`, and have never run into the crashing/entrypoint issues that other projects might have seen.

  One other thing worth noting here, it how Debian/Ubuntu patch the binutils that they distribute, because that's what we end up using in our gitian builds.

  In the binutils-mingw-w64 in Bionic (18.04), which we currently use in gitian, PE hardening options/security flags are enabled by default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8ubuntu1/changelog) and the [relevant commit](452b3013b8).

  However in Focal (20.04), this has now been reversed. PE hardening options are no-longer the default. See the [changelog](https://changelogs.ubuntu.com/changelogs/pool/universe/b/binutils-mingw-w64/binutils-mingw-w64_8.8/changelog) and [relevant commit](7bd8b2fbc2), which cites same .reloc issue mentioned here.

  Given that we explicitly specify/opt-in to everything that we want to use, the defaults aren't necessarily an issue for us. However I think it highlights the importance of continuing to be explicit about what we want, and not falling-back or relying on upstream.

  This was also prompted by the possibility of us doing link time garbage collection, see #18579 & #18605. It seemed some sanity checks would be worthwhile in-case the linker goes haywire while garbage collecting.

  I think Guix is going to bring great benefits when dealing with these kinds of issues. Carl you might have something to say in that regard.

ACKs for top commit:
  dongcarl:
    ACK 3e38023af7

Tree-SHA512: af14d63bdb334bde548dd7de3e0946556b7e2598d817b56eb4e75b3f56c705c26aa85dd9783134c4b6a7aeb7cb4de567eed996e94d533d31511f57ed332287da
2020-04-28 13:32:45 +08:00
fanquake d8ca51db5d
Merge #18589: Fix naming of macOS SDK and clarify version
eb37275a6f Fix naming of macOS SDK and clarify version (Andrew Chow)

Pull request description:

  Fixes the `MacOSX10.14.sdk.tar.gz` creation command to have `MacOSX.sdk` be correctly named as `MacOSX10.14.sdk` and for the resulting file to be placed in the current directory. Gitian requires that `tar.gz` contains a folder named `MacOSX10.14.sdk` and the command did not do this originally. Having the file be placed in the current directory is a convenience so builders don't have to go find it.

  Also clarifies which version of Xcode to download and where it can be downloaded.

ACKs for top commit:
  fanquake:
    ACK eb37275a6f - tested the macOS and Linux SDK extraction. Also noticed something seemingly broken with Apple `tar`, but will open an issue to follow up.
  Sjors:
    ACK eb37275 for the macOS instruction

Tree-SHA512: d691e14711cf195999291dd6fb7ffe552c86f8b30d2b1a77e88b4db6050dd817ba128b047cf36d29b0bb0d4183e709b7c03aa27f31b64e562ea8cd948434ca55
2020-04-24 17:22:54 +08:00
Andrew Chow eb37275a6f Fix naming of macOS SDK and clarify version 2020-04-23 13:22:16 -04:00
fanquake 3e38023af7
scripts: add PE .reloc section check to security-check.py 2020-04-23 08:40:24 +08:00
fanquake 8334ee31f8
scripts: add MACHO LAZY_BINDINGS test to test-security-check.py
I didn't add the relevant test in #18295.
2020-04-21 11:32:06 +08:00
fanquake 7b99c7454c
scripts: add MACHO Canary check to security-check.py 2020-04-21 11:32:01 +08:00
MarcoFalke 54f812d9d2
Merge #18673: scripted-diff: Sort test includes
fa4632c417 test: Move boost/stdlib includes last (MarcoFalke)
fa488f131f scripted-diff: Bump copyright headers (MarcoFalke)
fac5c37300 scripted-diff: Sort test includes (MarcoFalke)

Pull request description:

  When writing tests, often includes need to be added or removed. Currently the list of includes is not sorted, so developers that write tests and have `clang-format` installed will either have an unrelated change (sorting) included in their commit or they will have to manually undo the sort.

  This pull preempts both issues by just sorting all includes in one commit.

  Please be aware that this is **NOT** a change to policy to enforce clang-format or any other developer guideline or process. Developers are free to use whatever tool they want, see also #18651.

  Edit: Also includes a commit to bump the copyright headers, so that the touched files don't need to be touched again for that.

ACKs for top commit:
  practicalswift:
    ACK fa4632c417
  jonatack:
    ACK fa4632c417, light review and sanity checks with gcc build and clang fuzz build

Tree-SHA512: 130a8d073a379ba556b1e64104d37c46b671425c0aef0ed725fd60156a95e8dc83fb6f0b5330b2f8152cf5daaf3983b4aca5e75812598f2626c39fd12b88b180
2020-04-17 10:12:13 -04:00
Hennadii Stepanov 2aa48edec0
refactor: Drop unused ${WRAP_DIR}/${HOST} directory
This commit removes the directory that is no longer used since #16667.
2020-04-17 16:09:11 +03:00
Hennadii Stepanov 1362be0447
build: Drop make dist in gitian builds 2020-04-17 16:09:04 +03:00
Wladimir J. van der Laan 8f2497941e
Merge #18598: gitian: Add missing automake package to gitian-win-signer.yml
e44aeefaae gitian: Add missing automake package to gitian-win-signer.yml (Andrew Chow)

Pull request description:

  automake is needed to build osslsigncode otherwise autogen.sh fails with the docker virtualization method.

ACKs for top commit:
  hebasto:
    ACK e44aeefaae, for `osslsigncode-1.7.1` we did not run `autogen.sh` in the past.
  fanquake:
    ACK e44aeefaae
  jonatack:
    ACK e44aeef

Tree-SHA512: a0e615c1b099ee1c469ce41f886f2ece6746234a5a800743a4e8be671e4114fd30e1c35bc0ddcb75778409564129d0fde7ac4e3d70b0f7691f97f729f34c8e0c
2020-04-16 22:27:46 +02:00
MarcoFalke fa488f131f
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-
2020-04-16 13:33:09 -04:00
fanquake e831f18b1e
Merge #18619: gitian: add jonatack gpg key fingerprint
905e2e85ba gitian: add jonatack gpg key fingerprint (Jon Atack)

Pull request description:

  per request https://github.com/bitcoin-core/gitian.sigs/pull/1221#issuecomment-612778063

ACKs for top commit:
  laanwj:
    ACK 905e2e85ba
  fanquake:
    ACK 905e2e85ba

Tree-SHA512: bddd734f13c53859280db2fa94b47cbb2a8b3f17ed6a6fdda2bf04f7e201e310ae24930e8f4be2f8b65a949659a9d3369704ed70031da9653a66a513fe597b67
2020-04-15 15:38:02 +08:00
fanquake 5447d57bff
Merge #18624: Added my fingerprint Stephan Oeste (Emzy)
c47adf8df4 Added my fingerprint Stephan Oeste (Emzy) (Stephan Oeste)

Pull request description:

  By request from laanwj added my PGP fingerprint.
  See: https://github.com/bitcoin-core/gitian.sigs/pull/1220#issuecomment-612778442

ACKs for top commit:
  Sjors:
    ACK c47adf8. Fingerprint matches Twitter profile: https://twitter.com/emzy (haven't verified it in any other way)
  fanquake:
    ACK c47adf8df4

Tree-SHA512: 3e39ae88f507a12f11fb2d5c779eba79ee2daeddecd0dc3f1fddfa29ce963d0e9af3fa5a10357157812597c10205a6beae31cc70af9471a782da23d8753b7cbd
2020-04-15 15:31:40 +08:00
Stephan Oeste c47adf8df4
Added my fingerprint Stephan Oeste (Emzy)
By request from added my PGP fingerprint.
See: https://github.com/bitcoin-core/gitian.sigs/pull/1220#issuecomment-612778442
2020-04-13 16:39:42 +02:00
Jon Atack 905e2e85ba
gitian: add jonatack gpg key fingerprint 2020-04-13 14:23:06 +02:00
fanquake f2b5b0a3b4
build: add linker optimization flags to guix
Any -O argument will enable optimizations in GNU ld. We can use -O2
here, as this matches our compile flags. Note that this would also
enable additional optimizations if using the lld or gold linkers,
when compared to -O0.
2020-04-12 18:38:00 +08:00
fanquake b8b050a8d6
build: add linker optimization flags to gitian descriptors
Any -O argument will enable optimizations in GNU ld. We can use -O2
here, as this matches our compile flags. Note that this would also
enable additional optimizations if using the lld or gold linkers,
when compared to -O0.
2020-04-12 18:36:56 +08:00
fanquake ed3b8eada8
Merge #17595: guix: Enable building for x86_64-w64-mingw32 target
a35e323589 guix: Appease travis. (Carl Dong)
0b66d22da5 guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong)
ba0b99bdd6 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong)
93439a71ed guix: Bump to upstream commit with mingw-w64 changes (Carl Dong)
35a96792dd guix: Check mingw symbols, improve SSP fix docs (Carl Dong)
449d8fe25b guix: Expand on INT trap message (Carl Dong)
3f1f03c67a guix: Spelling fixes (Carl Dong)
ff821dd2a1 guix: Reinstate make-ssp-fixed-gcc (Carl Dong)
360a9e0ad5 guix: Bump time-machine for mingw-w64 patches (Carl Dong)
93e41b7e3b guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong)
ef4f7e4c45 guix: Set the well-known timezone env var (Carl Dong)
acf4b3b3b5 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong)
c4cce00eac guix: Remove dead links from README. (Carl Dong)
df953a4c9a guix: Appease shellcheck. (Carl Dong)
91897c95e1 guix: Improve guix-build.sh documentation (Carl Dong)
570d769c6c guix: Build support for Windows (Carl Dong)

Pull request description:

  ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~
  Based on: #17933 (Time Machines are... shall we say... superior 😁)

  This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon)

ACKs for top commit:
  fanquake:
    ACK a35e323589 2/3

Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 18:34:18 +08:00
Andrew Chow e44aeefaae gitian: Add missing automake package to gitian-win-signer.yml
automake is needed to build osslsigncode otherwise autogen.sh fails.
2020-04-11 14:15:05 -04:00
Wladimir J. van der Laan dabe2bb11a build: Bump gitian descriptors to 0.21
Per the release process.
2020-04-10 19:46:39 +02:00
fanquake d486991aa5
Merge #18295: scripts: add MACHO lazy bindings check to security-check.py
5ca90f8b59 scripts: add MACHO lazy bindings check to security-check.py (fanquake)

Pull request description:

  This is a slightly belated follow up to #17686 and some discussion with Cory. It's not entirely clear if we should make this change due to the way the macOS dynamic loader appears to work. However I'm opening this for some discussion. Also related to #17768.

  #### Issue:
  [`LD64`](https://opensource.apple.com/source/ld64/) doesn't set the [MH_BINDATLOAD](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) bit in the header of MACHO executables, when building with `-bind_at_load`. This is in contradiction to the [documentation](https://opensource.apple.com/source/ld64/ld64-450.3/doc/man/man1/ld.1.auto.html):
  ```bash
  -bind_at_load
       Sets a bit in the mach header of the resulting binary which tells dyld to
       bind all symbols when the binary is loaded, rather than lazily.
  ```

  The [`ld` in Apples cctools](https://opensource.apple.com/source/cctools/cctools-927.0.2/ld/layout.c.auto.html) does set the bit, however the [cctools-port](https://github.com/tpoechtrager/cctools-port/) that we use for release builds, bundles `LD64`.

  However; even if the linker hasn't set that bit, the dynamic loader ([`dyld`](https://opensource.apple.com/source/dyld/)) doesn't seem to ever check for it, and from what I understand, it looks at a different part of the header when determining whether to lazily load symbols.

  Note that our release binaries are currently working as expected, and no lazy loading occurs.

  #### Example:

  Using a small program, we can observe the behaviour of the dynamic loader.

  Conducted using:
  ```bash
  clang++ --version
  Apple clang version 11.0.0 (clang-1100.0.33.17)
  Target: x86_64-apple-darwin18.7.0

  ld -v
  @(#)PROGRAM:ld  PROJECT:ld64-530
  BUILD 18:57:17 Dec 13 2019
  LTO support using: LLVM version 11.0.0, (clang-1100.0.33.17) (static support for 23, runtime is 23)
  TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)
  ```

  ```cpp
  #include <iostream>
  int main() {
  	std::cout << "Hello World!\n";
  	return 0;
  }
  ```

  Compile and check the MACHO header:
  ```bash
  clang++ test.cpp -o test
  otool -vh test
  ...
  Mach header
        magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
  MH_MAGIC_64  X86_64        ALL LIB64     EXECUTE    16       1424   NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE

  # Run and dump dynamic loader bindings:
  DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=no_bind.txt ./test
  Hello World!
  ```

  Recompile with `-bind_at_load`. Note still no `BINDATLOAD` flag:
  ```bash
  clang++ test.cpp -o test -Wl,-bind_at_load
  otool -vh test
  Mach header
        magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
  MH_MAGIC_64  X86_64        ALL LIB64     EXECUTE    16       1424   NOUNDEFS DYLDLINK TWOLEVEL WEAK_DEFINES BINDS_TO_WEAK PIE
  ...
  DYLD_PRINT_BINDINGS=1 DYLD_PRINT_TO_FILE=bind.txt ./test
  Hello World!
  ```

  If we diff the outputs, you can see that `dyld` doesn't perform any lazy bindings when the binary is compiled with `-bind_at_load`, even if the `BINDATLOAD` flag is not set:
  ```diff
  @@ -1,11 +1,27 @@
  +dyld: bind: test:0x103EDF030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x103EDF030 = 0x7FFF70C9FA58
  +dyld: bind: test:0x103EDF038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x103EDF038 = 0x7FFF70CA12C2
  +dyld: bind: test:0x103EDF068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x103EDF068 = 0x7FFF70CA12B6
  +dyld: bind: test:0x103EDF070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x103EDF070 = 0x7FFF70CA1528
  +dyld: bind: test:0x103EDF080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x103EDF080 = 0x7FFF70C9FAE6
  <trim>
  -dyld: lazy bind: test:0x10D4AC0C8 = libsystem_platform.dylib:_strlen, *0x10D4AC0C8 = 0x7FFF73C5C6E0
  -dyld: lazy bind: test:0x10D4AC068 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryC1ERS3_, *0x10D4AC068 = 0x7FFF70CA12B6
  -dyld: lazy bind: test:0x10D4AC038 = libc++.1.dylib:__ZNKSt3__18ios_base6getlocEv, *0x10D4AC038 = 0x7FFF70CA12C2
  -dyld: lazy bind: test:0x10D4AC030 = libc++.1.dylib:__ZNKSt3__16locale9use_facetERNS0_2idE, *0x10D4AC030 = 0x7FFF70C9FA58
  -dyld: lazy bind: test:0x10D4AC080 = libc++.1.dylib:__ZNSt3__16localeD1Ev, *0x10D4AC080 = 0x7FFF70C9FAE6
  -dyld: lazy bind: test:0x10D4AC070 = libc++.1.dylib:__ZNSt3__113basic_ostreamIcNS_11char_traitsIcEEE6sentryD1Ev, *0x10D4AC070 = 0x7FFF70CA1528
  ```

  Note: `dyld` also has a `DYLD_BIND_AT_LAUNCH=1` environment variable, that when set, will force any lazy bindings to be non-lazy:
  ```bash
  dyld: forced lazy bind: test:0x10BEC8068 = libc++.1.dylib:__ZNSt3__113basic_ostream
  ```

  #### Thoughts:
  After looking at the dyld source, I can't find any checks for `MH_BINDATLOAD`. You can see the flags it does check for, such as MH_PIE or MH_BIND_TO_WEAK [here](https://opensource.apple.com/source/dyld/dyld-732.8/src/ImageLoaderMachO.cpp.auto.html).

  It seems that the lazy binding of any symbols depends on whether or not [lazy_bind_size](https://opensource.apple.com/source/xnu/xnu-6153.11.26/EXTERNAL_HEADERS/mach-o/loader.h.auto.html) from the `LC_DYLD_INFO_ONLY` load command is > 0. Which was mentioned in [#17686](https://github.com/bitcoin/bitcoin/pull/17686#issue-350216254).

  #### Changes:
  This PR is one of [Corys commits](7b6ba26178), that I've rebased and modified to make build. I've also included an addition to the `security-check.py` script to check for the flag.

  However, given the above, I'm not entirely sure this patch is the correct approach. If the linker no-longer inserts it, and the dynamic loader doesn't look for it, there might be little benefit to setting it. Or, maybe this is an oversight from Apple and needs some upstream discussion. Looking for some thoughts / Concept ACK/NACK.

  One alternate approach we could take is to drop the patch and modify security-check.py to look for `lazy_bind_size` == 0 in the `LC_DYLD_INFO_ONLY` load command, using `otool -l`.

ACKs for top commit:
  theuni:
    ACK 5ca90f8b59

Tree-SHA512: 444022ea9d19ed74dd06dc2ab3857a9c23fbc2f6475364e8552d761b712d684b3a7114d144f20de42328d1a99403b48667ba96885121392affb2e05b834b6e1c
2020-04-10 07:42:20 +08:00
Carl Dong a35e323589
guix: Appease travis. 2020-04-07 19:27:31 -04:00
Carl Dong 0b66d22da5
guix: Use gcc-9 for mingw-w64 instead of 8
The libtool unsorted 'find' determinism issue seemed to have been solved
in gcc-9's git: d41cd173e23ebea7c758644d6ad6e0fde1c2e3a6 or SVN: r262451

Furthermore, it seems that Ubuntu Focal 20.04 LTS is going to ship with
gcc 9 and mingw-w64 7, which will match what we have now.

-----

A note on this:

Careful observers will see that previously I stated that all released
versions of gcc were bootstrapped with a libtool 2.2.7a, meaning that
they all had the unsorted 'find' determinism issue first resolved in
libtool 2.2.7b.

However, I was mistaken, gcc's ltmain.sh CLAIMS it was generated by
libtool 2.2.7a, but it was in fact edited manually. It seems that gcc
maintains their own versions of ltmain.sh and libtool.m4, and only
sometimes backports patches from upstream.

Quite confusing.
2020-04-07 19:01:26 -04:00
Carl Dong ba0b99bdd6
guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends
This is no longer needed after 3bef7c22 in the mingw-w64 git repository,
which is first included in mingw-w64 v7.0.0.

As of the previous bump to our Guix time machine, we now use mingw-w64
v7.0.0.
2020-04-07 19:00:50 -04:00
Carl Dong 93439a71ed
guix: Bump to upstream commit with mingw-w64 changes
Most of the mingw-w64 toolchain changes have now been upstreamed, we can
point to a commit that exists upstream.

NOTE: I'm not changing the URL yet until we see that Guix upstream will
accept all my patches for macOS.

-----

The Guix tree that's referred to by this commit contains the following
changes relevant to our mingw-w64 build:

b066c25026

  Adds a PACKAGES-WITH-*PATCHES procedure which we can use in the future
  to apply patches to packages if those patches are not considered
  appropriate to upstream Guix

4719b71572

  Adds mingw-w64 (the libc itself) reproducibility patches, taken from
  debian.

79825bee07 + 401d28e433 + c1c50cb5b0

  Add mingw-w64 specific binutils patches, taken from debian.
  Specifically, the "Make DLL import libraries reproducible" patch made
  libbitcoinconsensus.dll.a build reproducibly. The followup commits
  were hotfixes for my mistakes.

0f864175dc

  Bumps mingw-w64 to v7.0.0. This is the first release that enables
  secure APIs by default (which we need), and gains _FORTIFY_SOURCE
  support. This will also be what Ubuntu Focal 20.04 LTS releases with.

cdf00cf75d

  Bumps NSIS to v3.05. This is the first release that includes a fix for
  a reproducibility bug found by some of the electrum developers. See
  details here: https://sourceforge.net/p/nsis/bugs/1230/
2020-04-07 19:00:49 -04:00
Wladimir J. van der Laan adac12ae73
Merge #18506: net: Hardcoded seeds update for 0.20
0eeb0468e7 net: Hardcoded seeds update for 0.20 (Wladimir J. van der Laan)

Pull request description:

  Update hardcoded seeds from http://bitcoin.sipa.be/seeds.txt.gz,
  according to release process.

  Output from makeseeds.py:
  ```
    IPv4   IPv6  Onion Pass
  1364173 244127   2454 Initial
  1364173 244127   2454 Skip entries with invalid address
  1129552 213117   2345 After removing duplicates
  1129548 213117   2345 Skip entries from suspicious hosts
  338216 191944   2249 Enforce minimal number of blocks
  336851 188993   2189 Require service bit 1
    6998   1520    150 Require minimum uptime
    5682   1290     89 Require a known and recent user agent
    5622   1279     89 Filter out hosts with multiple bitcoin ports
     512    146     89 Look up ASNs and limit results per ASN and per net
  ```

Top commit has no ACKs.

Tree-SHA512: ce1c2cda18dd5bd22586a5283a0877f3bd890437cc29dc1d85452ba4a4d28032f591c8b37f3329e8e649556cf83750b6949a068fad76d1773853d93014609da0
2020-04-06 13:38:32 +02:00
fanquake 5ca90f8b59
scripts: add MACHO lazy bindings check to security-check.py 2020-04-04 09:54:25 +08:00
Wladimir J. van der Laan 0eeb0468e7 net: Hardcoded seeds update for 0.20
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt
seeds_sipa.txt seeds_sjors.txt, according to release process.

Output from makeseeds.py:
```
  IPv4   IPv6  Onion Pass
1364173 244127   2454 Initial
1364173 244127   2454 Skip entries with invalid address
1129552 213117   2345 After removing duplicates
1129548 213117   2345 Skip entries from suspicious hosts
338216 191944   2249 Enforce minimal number of blocks
336851 188993   2189 Require service bit 1
  6998   1520    150 Require minimum uptime
  5682   1290     89 Require a known and recent user agent
  5622   1279     89 Filter out hosts with multiple bitcoin ports
   512    146     89 Look up ASNs and limit results per ASN and per net
```
2020-04-03 16:29:26 +02:00