40b17f5f9 [scripts] lint-whitespace: use perl instead of grep -P (Sjors Provoost)
Pull request description:
MacOS does not support `grep -P` out of the box. This change makes
it easier for developers to check for whitespace problems locally.
Based on [this](https://stackoverflow.com/a/16658690) and [this](https://serverfault.com/a/504387) Stack Exchange answer.
Tested with:
```sh
export TRAVIS_COMMIT_RANGE='fe78c9a...62e0453'
contrib/devtools/lint-whitespace.sh
This diff appears to have added new lines with tab characters instead of spaces.
The following changes were suspected:
diff --git a/src/test/bignum_tests.cpp b/src/test/bignum_tests.cpp
@@ -0,0 +1,110 @@
+ num.setint64(n);
```
Tree-SHA512: 37c342a0ca2580289cf326a278a051a7c21ba918d6b2143fd9987f159fab85f1de3d770fcf532a642cd5d1957afc8595678128196e102dc473924758f133db7f
Before this PR, the linenumber infomaition is output if trailing-space
or tab code was found, but the output occurence is only per a file.
This PR separates the output timing of file name and line number.
As a result, users will find where they need to fix more easily.
526e28220a contrib: Add support for out-of-tree builds in gen-manpages.sh (Wladimir J. van der Laan)
Pull request description:
This adds support for setting the environment variable `BUILDDIR` to point to executables that are outside the source directory.
E.g. to invoke the tool when the build is in $PWD/build:
```bash
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
```
This avoids having to manually copy the generated manpages after they end up in the build instead of source path, when setting TOPDIR instead.
Tree-SHA512: 8dc6dd7a47a0c014ae7d27f0ac9d86f69238ec6bac8a3007b975bb88c9f37014755c716c5e62604dd91baad2f8a41fd1544cdca3ba4b59bc76602e6593f4a4a7
This adds support for setting the environment variable `BUILDDIR`
to point to executables that are outside the source directory.
E.g. to invoke the tool when the build is in $PWD/build:
```bash
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
```
415f86c6ae [scripts] Add missing univalue file to copyright_header.py (fanquake)
Pull request description:
This needs to be added so that PRs like #12062 don't modify the subtree.
Tree-SHA512: 3642bdb0c8271ae700857a79fa5800b0c26c4b3f126d4406f224293817fb74d498fa1fc581d576ae747fbbb6952d4369fc4ab823ab48fd0946c1e8ccbe93cee6
13a81b19d Add quotes to variable assignment (as requested by @TheBlueMatt) (practicalswift)
683b9d280 Fix valid path output (practicalswift)
193c2fb4c Use bash instead of POSIX sh. POSIX sh does not support arrays. (practicalswift)
80f5f28d3 Fix incorrect quoting of quotes (the previous quotes had no effect beyond unquoting) (practicalswift)
564a172df Add required space to [[ -n "$1" ]] (previously [[ -n"$1" ]]) (practicalswift)
1e44ae0e1 Add error handling: exit if cd fails (practicalswift)
b9e79ab41 Remove "\n" from echo argument. echo does not support escape sequences. (practicalswift)
f6b3382fa Remove unused variables (practicalswift)
Pull request description:
Shell script cleanups:
* Add required space to `[ -n ]`.
* Avoid quote within quote.
* Exit if `cd` fails.
* Remove `\n` which is not handled by `echo`.
* ~~Remove redundant `$` in arithmetic variable expression.~~
* ~~Use `$(command)` instead of legacy form `` `command` ``.~~
* Arrays are not supported in POSIX `sh`. Use `bash` when arrays are used.
* ~~`[ foo -a bar ]` is not well defined, use `[ foo ] && [ bar ]` instead.~~
* ~~`[ foo -o bar ]` is not well defined, use `[ foo ] || [ bar ]` instead.~~
Tree-SHA512: 80f6ded58bce625b15b4da30d69d2714c633e184e62b21ed67d2c58e2ebaa08b4147593324012694d02bf4f1f252844cdff2fd1cf5e817ddb07e2777db7a6390
487aff421 Check subtree consistency in Travis (Pieter Wuille)
e1d0cc23a Improve git-subtree-check.sh (Pieter Wuille)
Pull request description:
Apparently many of our subtrees get modified by PRs in this repository, without getting noticed.
To improve upon this:
* Make git-subtree-check.sh capable of doing a weaker consistency check (that doesn't need access to external repositories), but which should be sufficient to detect unintended changes. It can be fooled by a fake subtree merge commit, but that would hopefully be obvious to reviewers.
* Make Travis invoke this subtree check for each of our subtrees.
Note that Travis is currently expected to fail on this PR, as 2 out of 4 subtrees (`src/secp156k1` and `src/univalue` have been modified directly in master).
Tree-SHA512: 465b680392d3daf38a8c1dda77d6f74b1d1c23324c378774777fb95aa673e119a8f7e3ccc124e41d97b5ac8975f3d79f3015797d2d309666582394364917ec4e
ab8e8b9 Remove unused variables in shell scripts. (practicalswift)
Pull request description:
Remove unused variables in shell scripts. Use `_` where we don't care about the result.
Tree-SHA512: 35049e79ee432c805f061456c32902a92811b5214d50ce6770b22d1442cc5999ed53cfe05bb2347f6995ca33c707a0f3fe92d5829c0385c4a3e254953924cbc4
We have several pieces of information about subtrees:
1) What their current directory contents is
2) What their directory contents was at the time of the last subtree merge
3) What the directory contents of the upstream project is in the commit referred to by the subtree merge.
Normally, all 3 should be identical. git-subtree-check.sh so far only compared (1) with (3) however.
Fix this by comparing all three, and give some more useful diff output in the case of mismatch.
The added benefit is that (1) and (2) can be compared without needing to see the upstream repository.
Lint checks should not test these, they are historical documents,
and we don't want to encourage silly changes to them to satisfy
a checker.
Hopefully makes travis pass again on master.
Tree-SHA512: 37e6716c4fd5e8a4e579f9b84042e6b0ac224836b6c851cd1ca3f7d46611ffd3003bed0ae08dd0457f69d6eaa485a0d21c631e7ef16b14bdb0f2f78ea700332d
This adds a new CHECK_DOC check that looks for newly introduced trailing
whitespace. Existing trailing whitespace (of which there is plenty!)
will not trigger an error.
This is written in a generic way so that new lint-*.sh scripts can be
added to contrib/devtools/, as I'd like to contribute additional lint
checks in the future.
77aa9e59e test: Check RPC argument mapping (Wladimir J. van der Laan)
Pull request description:
Parse the dispatch tables from the server implementation files, and the conversion table from the client (see #10751).
Perform the following consistency checks:
- Arguments defined in conversion table, must be present in dispatch table. If not, it was probably forgotten to add them to the dispatch table, and they will not work.
- Arguments defined in conversion table must have the same names as in the dispatch table. If not, they will not work.
- All aliases for an argument must either be present in the conversion table, or not. Anything in between means an oversight and some aliases won't work.
Any of these results in an error.
It also performs a consistency check to see if the same named argument is sometimes converted, and sometimes not. E.g. one RPC call might have a 'verbose' argument that is converted,
another RPC call might have one that is not converted. This is not necessarily wrong, but points at a possible error (as well as makes the API harder to memorize) - so it is emitted as a warning (could upgrade this to error).
This test is added to travis and run when `CHECK_DOC`. Currently fails with the following output:
```
* Checking consistency between dispatch tables and vRPCConvertParams
ERROR: createrawtransaction argument 3 (named optintorbf in vRPCConvertParams) is not defined in dispatch table
ERROR: getblock argument ['verbosity', 'verbose'] has conflicts in vRPCConvertParams conversion specifier [True, False]
WARNING: conversion mismatch for argument named verbose ([('getblock', False), ('getblockheader', True), ('getmempoolancestors', True), ('getmempooldescendants', True), ('getrawmempool', True), ('getrawtransaction', True)])
```
- ~#10698 fixes the first ERROR~
- #10747 fixes the second ERROR, as well as the WARNING
Update: #10698 was merged, leaving:
```
* Checking consistency between dispatch tables and vRPCConvertParams
ERROR: getblock argument ['verbosity', 'verbose'] has conflicts in vRPCConvertParams conversion specifier [True, False]
WARNING: conversion mismatch for argument named verbose ([('getblock', False), ('getblockheader', True), ('getmempoolancestors', True), ('getmempooldescendants', True), ('getrawmempool', True), ('getrawtransaction', True)])
```
Tree-SHA512: feabebfbeda5d4613b2b9d5265aa6bde4e1a0235297ffd48fa415ad7edc531d9ed7913fe76d191ac60d481a915a326f216bc93de3c671e45e1d14e97d07dea7a
Parse the dispatch tables from the server implementation files,
and the conversion table from the client.
Perform the following consistency checks:
- Arguments defined in conversion table, must be present in dispatch
table. If not, it was probably forgotten to add them to the
dispatch table, and they will not work.
- Arguments defined in conversion table must have the same names as
in the dispatch table. If not, they will not work.
- All aliases for an argument must either be present in the
conversion table, or not. Anything in between means an oversight
and some aliases won't work.
Any of these results in an error.
It also performs a consistency check to see if the same
named argument is sometimes converted, and sometimes not. E.g.
one RPC call might have a 'verbose' argument that is converted,
another RPC call might have one that is not converted. This is not
necessarily wrong, but points at a possible error (as well as
makes the API harder to memorize) - so it is emitted as a warning
(could upgrade this to error).
Fetch the destination branch as well as PR in one go. Saves a few
seconds (as well as one ssh authentication, when using a yubikey) when
using github-merge.py.
5432fc3 Fail on commit with VERIFY SCRIPT but no scripted-diff (Pieter Wuille)
Tree-SHA512: 687b6c825f6f882f9c8e9d301bc893383710bad70216fa40b95f7e24d83a7148f9c759c3e4bd905a6cd26960829f8f6bd03106dc6c83ac312bf34ad239917018
Some minor github-merge improvements I've made over time:
User interface:
- Print merge details again before signing off, to refresh your memory -
usually I'll have done lots of different things in the shell so this
will have scrolled out a long time ago.
- Require a valid answer on the prompts. One of the requested answers
must be typed, if not, the prompt will re-ask. This prevents
accidentally rejecting.
Efficiency:
- Condense "accept merge" and "sign off" prompts. There's no reason to
have this as two separate prompts, both are just opportunities to skip
out on the merge, no action is performed in between.
Merging:
- Strip spaces from github title. This avoids redundant spaces
surrounding it from getting into the commit message.
This changes tree_sha512sum() to requests the objects for hashing from
git instead of from the working tree.
The change should make the process more deterministic (it hashes what
will be pushed) and hopefully avoids the frequent miscomputed SHA512's
that happen now.
bbd7579 Fix regsig checking for subkey sigs in verify-commits (Matt Corallo)
d025bc7 Allow any subkey in verify-commits (Matt Corallo)
eddc77a Add comment re: why SHA1 is disabled (Peter Todd)
d9c450f Verify Tree-SHA512s in merge commits, enforce sigs are not SHA1 (Matt Corallo)
be908a6 Fail merge if there are any symlinks (Matt Corallo)
Tree-SHA512: bb66c59cc1c6b1c86d7d8be7adb0769c6598c0e28ad927409941f30af87d390521e82fc13700ee22e92db1bd571db3e19a152ec7b2c0349c6e06f5de62c0b65f
This removes the option from the wallet to not pay a fee on "small"
transactions which spend "old" inputs.
This code is no longer worth keeping around, as almost all miners
prefer not to include transactions which pay no fee at all.
If both numeric format specifiers and "others" are used, assume we're
dealing with a Qt-formatted message. In the case of Qt formatting (see
https://doc.qt.io/qt-5/qstring.html#arg) only numeric formats are
replaced at all. This means "(percentage: %1%)" is valid (which was
introduced in #9461), without needing any kind of escaping that would be
necessary for strprintf. Without this, this function would wrongly
detect '%)' as a printf format specifier.
Three categories of modifications:
1)
1 instance of 'The Bitcoin Core developers \n',
1 instance of 'the Bitcoin Core developers\n',
3 instances of 'Bitcoin Core Developers\n', and
12 instances of 'The Bitcoin developers\n'
are made uniform with the 443 instances of 'The Bitcoin Core developers\n'
2)
3 instances of 'BitPay, Inc\.\n' are made uniform with the other 6
instances of 'BitPay Inc\.\n'
3)
4 instances where there was no '(c)' between the 'Copyright' and the year
where it deviates from the style of the local directory.
62c2915 build: supply `-Wl,--high-entropy-va` (Wladimir J. van der Laan)
9a75d29 devtools: Check for high-entropy ASLR in 64-bit PE executables (Wladimir J. van der Laan)
d19583f improved gen-manpages.sh, includes bitcoin-tx and strips commit tag, now also runs binaries from build dir by default, added variables for more control (nomnombtc)
09546ca regenerated all manpages with commit tag stripped, also add bitcoin-tx (nomnombtc)
ae6e754 change help string --enable-man to --disable-man (nomnombtc)
a32c102 add conditional for --enable-man, default is yes (nomnombtc)
dc84b6f add doc/man to subdir if configure flag --enable-man is set (nomnombtc)
00dba72 add doc/man/Makefile.am to include manpages (nomnombtc)
eb5643b add autogenerated manpages by help2man (nomnombtc)
6edf2fd add gen-manpages.sh description to README.md (nomnombtc)
d2cd9c0 add script to generate manpages with help2man (nomnombtc)
Three subcommands to this script:
1) ./copyright_header.py report
Examines git-tracked files with extensions that match:
INCLUDE = ['*.h', '*.cpp', '*.cc', '*.c', '*.py']
Helps to:
-> Identify source files without copyright
-> Identify source files added with something other than "The Bitcoin Core
developers" holder so we can be sure it is appropriate
-> Identify unintentional typos in the copyright line
2) ./copyright_header.py update
Replaces fix-copyright-headers.py. It does file editing in native python
rather than subprocessing out to perl as was the case with
fix-copyright-headers.py. It also shares code with the 'report' functions.
3) ./copyright_header.py insert
Inserts a copyright header into a source file with the proper format and
dates.
fafe7b3 contrib: Make fix-copyright-headers.py more portable (MarcoFalke)
fa27c0a [doc] Fix typos in comments, doxygen: Fix comment syntax (MarcoFalke)
fabfd5d [qa] pull-tester: Don't mute zmq ImportError (MarcoFalke)
67a5502 init: Fix typo in help message for -whitelistforcerelay (Wladimir J. van der Laan)
- create a script to handle split debug. This will also eventually need to check
targets, and use dsymutil for osx.
- update config.guess/config.sub for bdb for aarch64.
- temporarily disable symbol checks for arm/aarch64
- quit renaming to linux32/linux64 and use the host directly
This also adds a hack to work around an Ubuntu bug in the gcc-multilib package:
https://bugs.launchpad.net/ubuntu/+source/gcc-defaults-armhf-cross/+bug/1347820
The problem is that gcc-multilib conflicts with the aarch toolchain.
gcc-multilib installs a symlink that points
/usr/include/asm -> /usr/include/x86_64-linux-gnu/asm.
Without this link, gcc -m32 can't find asm/errno.h (and others), since
/usr/include/x86_64-linux-gnu isn't in its default include path. But
/usr/include/i386-linux-gnu is (though it doesn't exist on disk).
So work around the problem by linking
/usr/include/i386-linux-gnu/asm -> /usr/include/x86_64-linux-gnu/asm.
The symlink fix is actually quite reasonable, but echoing the password into
sudo is nasty, and should probably be addressed in gitian itself. It makes more
sense to enable passwordless sudo for the build user by default.
As we are already using the API to retrieve the pull request
title, also retrieve the base branch.
This makes sure that pull requests for 0.12 automatically end up in
0.12, and pull requests for master automatically end up in master,
and so on.
It is still possible to override the branch from the command line
or using the `githubmerge.branch` git option.
Ubuntu 16.04 "xenial xerus" does not come with Python 2.x by default.
It is possible to install a python-2.7 package, but this has its own
problem: no `python` or `python2` symlink (see #7717).
This fixes the following scripts to work with python 3:
- `make check` (bctest,py, bitcoin-util-test.py)
- `make translate` (extract_strings_qt.py)
- `make symbols-check` (symbol-check.py)
- `make security-check` (security-check.py)
Explicitly call the python commands using $(PYTHON) instead
of relying on the interpreter line at the top of the scripts.
This makes github-merge.py the first developer tool to go
all Python 3 (for context see #7717).
The changes are straightforward as the script already was
`from __future__ import division,print_function,unicode_literals`.
However urllib2 changed name, and json will only accept unicode data not
bytes.
This retains py2 compatibility for now: not strictly necessary
as it's not used by the build system - but it was easy.
Use 'utf-8' instead of the Python 2 default of 'ascii' to encode/decode
commit messages.
This can be removed when switching to Python 3, as 'utf-8' is the
default there.
Necessary for merging #7422 due to the ฿ in ฿tcDrak.
This is meant to be a direct translation of the bash script,
with the difference that it retrieves the PR title from github,
thus creating pull messages like:
Merge #12345: Expose transaction temperature over RPC
