fc7f84a9ca tests: Add fuzzing harness for Keccak and SHA3_256 (practicalswift)
Pull request description:
Add fuzzing harness for Keccak and SHA3_256.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
laanwj:
uACK fc7f84a9ca
elichai:
utACK :) fc7f84a9ca
Tree-SHA512: 01e1610e1c178d5f42578e2dd5644a4165596db34cf5037d574a5285e0ace4b06dc33ab81a308595246117537fe175294efd4bfc174ffc2e8eac98f0ec9dd3e9
a8a64acaf3 [BroadcastTransaction] Remove unsafe move operator (Amiti Uttarwar)
125c038126 [p2p] Remove dead code (Amiti Uttarwar)
fc66d0a65c [p2p] Check for nullptr before dereferencing pointer (Adam Jonas)
cb79b9dbf4 [mempool] Revert unbroadcast set to tracking just txid (Amiti Uttarwar)
Pull request description:
Addresses some outstanding review comments from #18044
- reverts unbroadcast txids to a set instead of a map (simpler, communicates intent better, takes less space, no efficiency advantages of map)
- adds safety around two touchpoints (check for nullptr before dereferencing pointer, remove an inaccurate std::move operator)
- removes some dead code
Links to comments on wtxid PR: [1](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460495254) [2](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460496023) [3](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r463532611)
thanks to jnewbery & adamjonas for flagging these ! !
ACKs for top commit:
sdaftuar:
utACK a8a64acaf3
naumenkogs:
utACK a8a64acaf3
jnewbery:
utACK a8a64acaf3
Tree-SHA512: 7be669cb30cc17fb9e06b50e636ef7887c6a27354697987e4e4d38dba4b8f50e175647587430cd9bc3295bec01ce8b1e6639a50a4249d8fff9b1ca1b9ead3277
[META] This is a pure refactor commit.
Move PruneBlockFile to BlockManager because:
1. PruneOneBlockFile only acts on BlockManager
2. Eliminates the need for callers (FindFilesToPrune{,Manual}) to have a
reference to the larger ChainstateManager, just a reference to
BlockManager is enough. See following commits.
916d3596c4 help: Generate checkpoint height from chainparams (Luke Dashjr)
Pull request description:
Not sure if this is worth putting in Core, but might as well until checkpoints are removed entirely.
ACKs for top commit:
laanwj:
re-ACK 916d3596c4
Tree-SHA512: d8eb26b570ee730fdd75ca916507134db5f2f68987a911e33544b7f1c9ccfd1c76b9c9db63056971956b6daf16910f17ecfc197481c2f7b0773afdfbf7d381cf
fc9278d162 build: AX_PTHREAD serial 27 (fanquake)
15c27c4441 build: split PTHREAD_* flags out of AM_LDFLAGS (fanquake)
68e3e22944 scripted-diff: add FUZZ_SUITE_LDFLAGS_COMMON (fanquake)
afecde8046 build: add PTHREAD_LIBS to LDFLAGS configure output (fanquake)
Pull request description:
TLDR: Split pthread flags out of ldflags, and stop using them when building libconsensus.
Building libconsensus on Linux using Clang currently warns. i.e:
```bash
./autogen.sh
./configure --disable-tests --disable-bench --with-utils=no --with-daemon=no --with-gui=no --disable-wallet --with-libs=yes CC=clang CXX=clang++
make V=1 -j6
... -Wl,-z -Wl,relro -Wl,-z -Wl,now -pthread -Wl,-soname -Wl,libbitcoinconsensus.so.0 -o .libs/libbitcoinconsensus.so.0.0.0
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
```
Besides wanting to quiet the warnings, after digging into this it seemed we could clean up how we are passing around the pthread flags. I also learnt a bit more about how libtools builds shared libraries, and that passing `-pthread` on the link line wouldn't be enough to link against pthreads anyways, due to libtools usage of -nostdlib (see [related discussion where we build DLLs](476436b2de/configure.ac (L603))).
This can be demonstrated with a patch to libconsensus:
```patch
diff --git a/src/script/bitcoinconsensus.cpp b/src/script/bitcoinconsensus.cpp
index 15e204062..10bf3582f 100644
--- a/src/script/bitcoinconsensus.cpp
+++ b/src/script/bitcoinconsensus.cpp
@@ -10,6 +10,8 @@
#include <script/interpreter.h>
#include <version.h>
+#include <pthread.h>
+
namespace {
/** A class that deserializes a single CTransaction one time. */
@@ -127,3 +129,10 @@ unsigned int bitcoinconsensus_version()
// Just use the API version for now
return BITCOINCONSENSUS_API_VER;
}
+
+void *func_pthread(void *x) { return x; }
+
+void f() {
+ pthread_t t;
+ pthread_create(&t,0,func_pthread,0);
+}
```
After building, you'll find you have a `libbitcoinconsensus.so` using pthread symbols, but which isn't linked against libpthread:
```bash
ldd -r src/.libs/libbitcoinconsensus.so
linux-vdso.so.1 (0x00007ffe49378000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f553cee7000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f553cda2000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f553cd88000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f553cbc5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f553d15d000)
undefined symbol: pthread_create (src/.libs/libbitcoinconsensus.so)
```
This libtool behaviour has been known about for some time, i.e this [thread from 2005](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460), describes the same issue. The suggestion from libtool maintainers at the time is to add `-lpthread` to LDFLAGS.
Also worth noting is that some of the users in those threads were also using the `AX_PTHREADS` macro, same as us, to determine how to compile with/link against pthreads. This macro has [recently been updated](https://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=commitdiff;h=2fb904589643eb6ca6122f834891b58d1d51b347), with reference to this issue. You can compare the output from the version we currently use, to the new version:
```bash
# our ax_pthread macro:
PTHREAD_CFLAGS = -pthread
PTHREAD_LIBS =
PTHREAD_CC = gcc / clang
# the new ax_pthread macro
PTHREAD_CFLAGS = -pthread
PTHREAD_LIBS = -lpthread
PTHREAD_CC = gcc / clang
```
Note that as part of this PR I've also added `PTHREAD_LIBS` to the split out flags. Although we weren't using it anywhere previously (and wouldn't have seemed to matter for the most part, given it was likely empty for most builders), the macro assumes it's use. i.e:
> NOTE: You are assumed to not only compile your program with these flags,
> but also to link with them as well. For example, you might link with
> $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS
ACKs for top commit:
laanwj:
Code review ACK fc9278d162
hebasto:
re-ACK fc9278d162, only rebased and renamed s/`AM_PTHREAD_FLAGS`/`PTHREAD_FLAGS`/ since my [previous](https://github.com/bitcoin/bitcoin/pull/19558#pullrequestreview-473487730) review..
kallewoof:
ACK fc9278d162
Tree-SHA512: 7c0a5b0f0de4f54b1d7dce0e69020b341c37a383bb7c715867cc96c648774a557b1ddb42eb1b676f7bb2b822b69795bec14dc6272362d80662a21f10cb80331c
Windows headers define SendMessage as a macro, which leads to problems
with the method name "SendMessage". To circumvent this, we rename the
method to "SendZmqMessage".
-BEGIN VERIFY SCRIPT-
sed -i 's/SendMessage/SendZmqMessage/g' src/zmq/zmqpublishnotifier.*
-END VERIFY SCRIPT-
bf1f913c44 cli -netinfo: display multiple levels of details (Jon Atack)
077b3ac928 cli: change -netinfo optional arg from bool to int (Jon Atack)
4e2f2ddd64 cli: add getpeerinfo last_{block,transaction} to -netinfo (Jon Atack)
644be659ab cli: add -netinfo server version check and error message (Jon Atack)
ce57bf6cc0 cli: create peer connections report sorted by dir, minping (Jon Atack)
f5edd66e5d cli: create vector of Peer structs for peers data (Jon Atack)
3a0ab93e1c cli: add NetType enum struct and NetTypeEnumToString() (Jon Atack)
c227100919 cli: create local addresses, ports, and scores report (Jon Atack)
d3f77b736e cli: create inbound/outbound peer connections report (Jon Atack)
19377b2fd2 cli: start dashboard report with chain and version header (Jon Atack)
a3653c159e cli: tally peer connections by type (Jon Atack)
54799b66b4 cli: add ipv6 and onion address type detection helpers (Jon Atack)
12242b17a5 cli: create initial -netinfo option, NetinfoRequestHandler class (Jon Atack)
Pull request description:
This PR is inspired by laanwj's python script mentioned in #19405, which it turns out I ended up using every day and extending because I got hooked on using it to monitor Bitcoin peer connections.
For the full experience, run `./src/bitcoin-cli -netinfo 4`
On Linux, try it with watch `watch ./src/bitcoin-cli -netinfo 4`
Help doc
```
$ ./src/bitcoin-cli -help | grep -A3 netinfo
-netinfo
Get network peer connection information from the remote server. An
optional integer argument from 0 to 4 can be passed for different
peers listings (default: 0).
```
ACKs for top commit:
vasild:
ACK bf1f913
0xB10C:
ACK bf1f913c44
practicalswift:
ACK bf1f913c44 -- patch looks correct and is limited to `src/bitcoin-cli.cpp`
Tree-SHA512: b9d18e5cc2ffd2bb9f0295b5ac7609da8a9bbecaf823a26dfa706b5f07d5d1a8343081dad98b16aa9dc8efd8f41bc1a4acdc40259727de622dc7195ccf59c572
92326d8976 [rpc] add send method (Sjors Provoost)
2c2a1445dc [rpc] add snake case aliases for transaction methods (Sjors Provoost)
1bc8d0fd59 [rpc] walletcreatefundedpsbt: allow inputs to be null (Sjors Provoost)
Pull request description:
`walletcreatefundedpsbt` has some interesting features that `sendtoaddress` and `sendmany` don't have:
* manual coin selection
* outputting a PSBT (it was controversial to add this, see #18201)
* create a transaction without adding to wallet (which leads to broadcasting, unless `-walletbroadcast=0`)
At the same time `walletcreatefundedpsbt` can't broadcast a transaction, which is inconvenient for simple use cases.
This PR introduces a new `send` RPC method which creates a PSBT, signs it if possible and adds it to the wallet by default. If it can't sign all inputs, it outputs a PSBT. If `add_to_wallet` is set to `false` it will return the transaction in both PSBT and hex format.
Because it uses a PSBT internally, it will much easier to add hardware wallet support to this method (see #16546).
For `bitcoin-cli` users, it tries to keep the simplest use case easy to use:
```sh
bitcoin-cli -regtest send '{"ADDRESS": 0.1}' 1 sat/b
```
This paves the way for deprecating `sendtoaddress` and `sendmany` though there's no rush. The only missing feature compared to these older methods is adding labels to a destination address.
Depends on:
- [x] #16377 (`[rpc] don't automatically append inputs in walletcreatefundedpsbt`)
- [x] #11413 (`[wallet] [rpc] sendtoaddress/sendmany: Add explicit feerate option`)
- [x] #18244 (`[rpc] have lockUnspents also lock manually selected coins`)
ACKs for top commit:
meshcollider:
Light re-utACK 92326d8976
achow101:
ACK 92326d8976 Reviewed code and test, ran tests.
kallewoof:
utACK 92326d8976
Tree-SHA512: 7552ef1b193d4c06e381c44932fdb0d54f64383e4c7d6b988f49d059c7d4bba45ce6aa7813e03df86360ad9dad6f3010eb76ee7da480551742d5fd98c2251c0f
~CMainCleanup:
1. Is vestigial
2. References the g_chainman global (we should minimize g_chainman refs)
3. Only acts on g_chainman.m_blockman
4. Does the same thing as BlockManager::Unload
This was confusing: which one is the good one ? After testing the value
is right but not the comment, so fix it.
Signed-off-by: Antoine Poinsot <darosior@protonmail.com>
Note that with this change we are no-longer including PTHREAD_* flags
when building libbitcoinconsensus.
Also note that we are including PTHREAD_LIBS in AM_PTHREAD_FLAGS
812037cb80 Change CSipHasher's count variable to uint8_t (Pieter Wuille)
Pull request description:
SipHash technically supports arbitrarily long inputs (at least, I couldn't find a limit in the [paper](https://eprint.iacr.org/2012/351.pdf)), but only the low 8 bits of the length matter. Because of that we should use an unsigned type to track the length (as any signed type could overflow, which is UB). `uint8_t` is sufficient, however.
Fixes#19930.
ACKs for top commit:
laanwj:
anyhow re-ACK 812037cb80
elichai:
utACK 812037cb80
practicalswift:
ACK 812037cb80
theStack:
ACK 812037cb80
Tree-SHA512: 5b1440c9e4591460da198991fb421ad47d2d96def2014e761726ce361aa9575752f2c4085656e7e9badee3660ff005cc76fbd1afe4848faefe4502f3412bd896
b9c1a76481 Squashed 'src/secp256k1/' changes from 2ed54da18a..8ab24e8dad (Pieter Wuille)
Pull request description:
This updates our src/secp256k1 subtree to the latest libsecp256k1 upstream version.
As it adds BIP340 support (see https://github.com/bitcoin-core/secp256k1/pull/558), this is a prerequisite for #17977. In particular, it contains:
* A few generic library improvements
* Support for x-only public keys as used by BIP340.
* Support for "key pair" objects, making signing more efficient by using a precomputed public key.
* Signing support for BIP340 Schnorr (single-party) signatures.
* Verification support for BIP340 Schnorr signatures.
* Support for verifying tweaked x-only keys, as used by BIP341's Taproot construction.
Things that are not included:
* MuSig, nor any kind of multisignatures, threshold signatures, ... on top.
* Batch verification.
* Support for variable-length messages in BIP340 (which are still being discussed, but won't affect BIP341, or Bitcoin Core).
* A few more generic improvements that are still in the pipeline, including faster modular inversions.
ACKs for top commit:
instagibbs:
ACK 894fb33f4c
fanquake:
ACK 894fb33f4c. Any Valgrind concerns will be addressed upstream, see discussion in https://github.com/bitcoin-core/secp256k1/pull/813, and if necessary, can be pulled into our tree prior to the 0.21.0 branch off. They are not a blocker for merging this PR in it's current state.
benthecarman:
ACK `894fb33`
Tree-SHA512: 6dc992f4477069b7fbd223316f1be955750923be1479c38adad2312649fdca1f316edb375c42ef9d97cea2407caaef49fb8c93abd6c037fe1a522910cbbc2479
8b39a87558 bugfix: make LoadWallet assigns status always (Akio Nakamura)
Pull request description:
In my enviroment, ```test/functional/wallet_multiwallet.py``` failed in line 237 for master( 147d50d63 ).
It got an expected rpc-error-message, but error code was not (-4) but (-18).
This is because that although loadwallet() in rpcwallet.cpp assumes LoadWallet() always assign some value to the 'status', but LoadWallet() does not do so in some situation.
This PR intends to fix above and prevends loadwallet() returns ambiguous error code.
ACKs for top commit:
hebasto:
re-ACK 8b39a87558, that is the same as 1728059730abef04f3fa84de0b6e20044be7a9d6.
ryanofsky:
Code review ACK 8b39a87558 (same as previous)
meshcollider:
utACK 8b39a87558
Tree-SHA512: a75d8240f60325bfdb69a07d392269fec97de743f38fe108371eb63a0aba5d8ce3cc484ecc69e81febf8040f5ab64f3a9450b98f8e07a0c17803784bb6f342bf
8ab24e8dad Merge #558: Add schnorrsig module which implements BIP-340 compliant signatures
f3733c5433 Merge #797: Fix Jacobi benchmarks and other benchmark improvements
cb5524adc5 Add benchmark for secp256k1_ge_set_gej_var
5c6af60ec5 Make jacobi benchmarks vary inputs
d0fdd5f009 Randomize the Z coordinates in bench_internal
c7a3424c5f Rename bench_internal variables
875d68b95f Merge #699: Initialize field elements when resulting in infinity
54caf2e74f Merge #799: Add fallback LE/BE for architectures with known endianness + SHA256 selftest
f431b3f28a valgrind_ctime_test: Add schnorrsig_sign
16ffa9d97c schnorrsig: Add taproot test case
8dfd53ee3f schnorrsig: Add benchmark for sign and verify
4e43520026 schnorrsig: Add BIP-340 compatible signing and verification
7332d2db6b schnorrsig: Add BIP-340 nonce function
7a703fd97d schnorrsig: Init empty experimental module
eabd9bc46a Allow initializing tagged sha256
6fcb5b845d extrakeys: Add keypair_xonly_tweak_add
58254463f9 extrakeys: Add keypair struct with create, pub and pub_xonly
f0010349b8 Separate helper functions for pubkey_create and seckey_tweak_add
910d9c284c extrakeys: Add xonly_pubkey_tweak_add & xonly_pubkey_tweak_add_test
176bfb1110 Separate helper function for ec_pubkey_tweak_add
4cd2ee474d extrakeys: Add xonly_pubkey with serialize, parse and from_pubkey
f49c9896b0 Merge #806: Trivial: Add test logs to gitignore
aabf00c155 Merge #648: Prevent ints from wrapping around in scratch space functions
f5adab16a9 Merge #805: Remove the extremely outdated TODO file.
bceefd6547 Add test logs to gitignore
1c325199d5 Remove the extremely outdated TODO file.
47e6618e11 extrakeys: Init empty experimental module
3e08b02e2a Make the secp256k1_declassify argument constant
8bc6aeffa9 Add SHA256 selftest
670cdd3f8b Merge #798: Check assumptions on integer implementation at compile time
5e5fb28b4a Use additional system macros to figure out endianness
7c068998ba Compile-time check assumptions on integer types
02b6c87b52 Add support for (signed) __int128
979961c506 Merge #787: Use preprocessor macros instead of autoconf to detect endianness
887bd1f8b6 Merge #793: Make scalar/field choice depend on C-detected __int128 availability
0dccf98a21 Use preprocessor macros instead of autoconf to detect endianness
b2c8c42cf1 Merge #795: Avoid linking libcrypto in the valgrind ct test.
57d3a3c64c Avoid linking libcrypto in the valgrind ct test.
79f1f7a4f1 Autodetect __int128 availability on the C side
0d7727f95e Add SECP256K1_FE_STORAGE_CONST_GET to 5x52 field
805082de11 Merge #696: Run a Travis test on s390x (big endian)
39295362cf Test travis s390x (big endian)
6034a04fb1 Merge #778: secp256k1_gej_double_nonzero supports infinity
f60915906d Merge #779: travis: Fix argument quoting for ./configure
9e49a9b255 travis: Fix argument quoting for ./configure
18d36327fd secp256k1_gej_double_nonzero supports infinity
214cb3c321 Merge #772: Improve constant-timeness on PowerPC
40412b1930 Merge #774: tests: Abort if malloc() fails during context cloning tests
2e1b9e0458 tests: Abort if malloc() fails during context cloning tests
67a429f31f Suppress a harmless variable-time optimization by clang in _int_cmov
5b196338f0 Remove redundant "? 1 : 0" after comparisons in scalar code
3e5cfc5c73 Merge #741: Remove unnecessary sign variable from wnaf_const
66bb9320c0 Merge #773: Fix some compile problems on weird/old compilers.
1309c03c45 Fix some compile problems on weird/old compilers.
2309c7dd4a Merge #769: Undef HAVE___INT128 in basic-config.h to fix gen_context compilation
22e578bb11 Undef HAVE___INT128 in basic-config.h to fix gen_context compilation
3f4a5a10e4 Merge #765: remove dead store in ecdsa_signature_parse_der_lax
f00d6575ca remove dead store in ecdsa_signature_parse_der_lax
dbd41db16a Merge #759: Fix uninitialized variables in ecmult_multi test
2e7fc5b537 Fix uninitialized variables in ecmult_multi test
37dba329c6 Remove unnecessary sign variable from wnaf_const
6bb0b77e15 Fix test_constant_wnaf for -1 and add a test for it.
47a7b8382f Clear field elements when writing infinity
61d1ecb028 Added test with additions resulting in infinity
60f7f2de5d Don't assume that ALIGNMENT > 1 in tests
ada6361dec Use ROUND_TO_ALIGN in scratch_create
8ecc6ce50e Add check preventing rounding to alignment from wrapping around in scratch_alloc
4edaf06fb0 Add check preventing integer multiplication wrapping around in scratch_max_allocation
git-subtree-dir: src/secp256k1
git-subtree-split: 8ab24e8dad9d43fc6661842149899e3cc9213b24
ab654c7d58 Unroll Keccak-f implementation (Pieter Wuille)
3f01ddb01b Add SHA3 benchmark (Pieter Wuille)
2ac8bf9583 Implement keccak-f[1600] and SHA3-256 (Pieter Wuille)
Pull request description:
Add a simple (and initially unoptimized) Keccak/SHA3 implementation based on https://github.com/mjosaarinen/tiny_sha3/blob/master/sha3.c, as one will be needed for TORv3 support (the conversion from BIP155 encoding to .onion notation uses a SHA3-based checksum). In follow-up commits, a benchmark is added, and the Keccakf function is unrolled for a (for me) 4.9x speedup.
Test vectors are taken from https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/secure-hashing#sha3vsha3vss.
ACKs for top commit:
practicalswift:
ACK ab654c7d58 -- patch looks correct and no sanitizer complaints when doing some basic fuzz testing of the added code (remember: **don't trust: fuzz!**) :)
laanwj:
re-ACK ab654c7d58
vasild:
ACK ab654c7
Tree-SHA512: 8a91b18c46e8fb178b7ff82046cff626180362337e515b92fbbd771876e795da2ed4e3995eb4849773040287f6e687237f469a90474ac53f521fc12e0f5031d9
The field m_protect is used to protect from eviction both by bad/lagging
chain and extra outbound peers logics. Outbound block-relay peers are
always excluded from this protection.
Instead of printing "<n> of the last 100 blocks have unexpected version"
as a warning appended to UpdateTip, it is now printed in the validation
log category.
Although loadwallet() in rpcwallet.cpp assumes LoadWallet() always
assign some value to the 'status', but LoadWallet() does not do so
in some situation.
This fixes above and prevends loadwallet() returns ambiguous error code.
No longer create a default wallet. The default wallet will still be
loaded if it exists and not other wallets were specified (anywhere,
including settings.json, bitcoin.conf, and command line).
Tests are updated to be started with -wallet= if they need the default
wallet.
Added test to wallet_startup.py testing that no default wallet is
created and that it is loaded if it exists and no other wallets were
specified.
There is a keyword that allows us to break out of loops. Use it.
There's a small change in behaviour here: if we process multiple orphans
that are still orphans, then we'll only call mempool.check() once at the
end, instead of after processing each tx.
bb6a32ce99 [net processing] Move Misbehaving() to PeerManager (John Newbery)
aa114b1c9b [net_processing] Move SendBlockTransactions into PeerManager (John Newbery)
3115e00f75 [net processing] Move MaybePunishPeerForTx to PeerManager (John Newbery)
e662e2d42a [net processing] Move ProcessOrphanTx to PeerManager (John Newbery)
b70cd890e3 [net processing] Move MaybePunishNodeForBlock into PeerManager (John Newbery)
d7778351bf [net processing] Move ProcessHeadersMessage to PeerManager (John Newbery)
64f6162651 [whitespace] tidy up indentation after scripted diff (John Newbery)
58bd369b0d scripted-diff: [net processing] Rename PeerLogicValidation to PeerManager (John Newbery)
2297b26b3c [net_processing] Pass chainparams to PeerLogicValidation constructor (John Newbery)
824bbd1ffb [move only] Collect all private members of PeerLogicValidation together (John Newbery)
Pull request description:
Continues the work of moving net_processing logic into PeerLogicValidation. See https://github.com/bitcoin/bitcoin/pull/19704 and https://github.com/bitcoin/bitcoin/pull/19607#discussion_r462032894 for motivation.
This PR also renames `PeerLogicValidation` to `PeerManager` as suggested in https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618.
ACKs for top commit:
MarcoFalke:
re-ACK bb6a32ce99 only change is rebase due to conflict in struct NodeContext and variable rename 🤸
hebasto:
re-ACK bb6a32ce99, only rebased, and added renaming `s/peer_logic/peerman/` into scripted-diff since my [previous](https://github.com/bitcoin/bitcoin/pull/19791#pullrequestreview-483118079) review (verified with `git range-diff`).
Tree-SHA512: a2de4a521688fd25125b401e5575402c52b328a0fa27b3010567008d4f596b960aabbd02b2d81f42658f88f4365443fadb1008150a62fbcea123fb42d85a2c21
Block-relay-only peers were introduced by #15759. According to its
author, it was intented to make them only immune to outbound peer
rotation-based eviction and not from all eviction as modified comment
leans to think of.
Clearly indicate that outbound block-relay peers aren't protected
from eviction by the bad/lagging chain logic.
-BEGIN VERIFY SCRIPT-
sed -i 's/PeerLogicValidation/PeerManager/g' $(git grep -l PeerLogicValidation ./src ./test)
sed -i 's/peer_logic/peerman/g' $(git grep -l peer_logic ./src ./test)
-END VERIFY SCRIPT-
PeerLogicValidation was originally net_processing's implementation to
the validation interface. It has since grown to contain much of
net_processing's logic. Therefore rename it to reflect its
responsibilities.
Suggested in
https://github.com/bitcoin/bitcoin/pull/10756#pullrequestreview-53892618.
Keep a references to chainparams, rather than calling the global
Params() function every time it's needed. This is fine, since
globalChainParams does not get updated once it's been set, and it's
available at the point of constructing the PeerLogicValidation object.
We don't have a project style for ordering class members, but it always
makes sense to have no more than one of each public/protected/private
specifier.
Also move documentation for MaybeDiscourageAndDisconnect to the header.
296be8f58e Get rid of unused functions CTxMemPool::GetMemPoolChildren, CTxMemPool::GetMemPoolParents (Jeremy Rubin)
46d955d196 Remove mapLinks in favor of entry inlined structs with iterator type erasure (Jeremy Rubin)
Pull request description:
Currently we have a peculiar data structure in the mempool called maplinks. Maplinks job is to track the in-pool children and parents of each transaction. This PR can be primarily understood and reviewed as a simple refactoring to remove this extra data structure, although it comes with a nice memory and performance improvement for free.
Maplinks is particularly peculiar because removing it is not as simple as just moving it's inner structure to the owning CTxMempoolEntry. Because TxLinks (the class storing the setEntries for parents and children) store txiters to each entry in the mempool corresponding to the parent or child, it means that the TxLinks type is "aware" of the boost multiindex (mapTx) it's coming from, which is in turn, aware of the entry type stored in mapTx. Thus we used maplinks to store this entry associated data we in an entirely separate data structure just to avoid a circular type reference caused by storing a txiter inside a CTxMempoolEntry.
It turns out, we can kill this circular reference by making use of iterator_to multiindex function and std::reference_wrapper. This allows us to get rid of the maplinks data structure and move the ownership of the parents/child sets to the entries themselves.
The benefit of this good all around, for any of the reasons given below the change would be acceptable, and it doesn't make the code harder to reason about or worse in any respect (as far as I can tell, there's no tradeoff).
### Simpler ownership model
No longer having to consistency check that mapLinks did have records for our CTxMempoolEntry, impossible to have a mapLinks entry outlive or incorrectly die before a CTxMempoolEntry.
### Memory Usage
We get rid of a O(Transactions) sized map in the mempool, which is a long lived data structure.
### Performance
If you have a CTxMemPoolEntry, you immediately know the address of it's children/parents, rather than having to do a O(log(Transactions)) lookup via maplinks (which we do very often). We do it in *so many* places that a true benchmark has to look at a full running node, but it is easy enough to show an improvement in this case.
The ComplexMemPool shows a good coherence check that we see the expected result of it being 12.5% faster / 1.14x faster.
```
Before:
# Benchmark, evals, iterations, total, min, max, median
ComplexMemPool, 5, 1, 1.40462, 0.277222, 0.285339, 0.279793
After:
# Benchmark, evals, iterations, total, min, max, median
ComplexMemPool, 5, 1, 1.22586, 0.243831, 0.247076, 0.244596
```
The ComplexMemPool benchmark only checks doing addUnchecked and TrimToSize for 800 transactions. While this bench does a good job of hammering the relevant types of function, it doesn't test everything.
Subbing in 5000 transactions shows a that the advantage isn't completely wiped out by other asymptotic factors (this isn't the only bottleneck in growing the mempool), but it's only a bit proportionally slower (10.8%, 1.12x), which adds evidence that this will be a good change for performance minded users.
```
# Benchmark, evals, iterations, total, min, max, median
ComplexMemPool, 5, 1, 59.1321, 11.5919, 12.235, 11.7068
# Benchmark, evals, iterations, total, min, max, median
ComplexMemPool, 5, 1, 52.1307, 10.2641, 10.5206, 10.4306
```
I don't think it's possible to come up with an example of where a maplinks based design would have better performance, but it's something for reviewers to consider.
# Discussion
## Why maplinks in the first place?
I spoke with the author of mapLinks (sdaftuar) a while back, and my recollection from our conversation was that it was implemented because he did not know how to resolve the circular dependency at the time, and there was no other reason for making it a separate map.
## Is iterator_to weird?
iterator_to is expressly for this purpose, see https://www.boost.org/doc/libs/1_51_0/libs/multi_index/doc/tutorial/indices.html#iterator_to
> iterator_to provides a way to retrieve an iterator to an element from a pointer to the element, thus making iterators and pointers interchangeable for the purposes of element pointing (not so for traversal) in many situations. This notwithstanding, it is not the aim of iterator_to to promote the usage of pointers as substitutes for real iterators: the latter are specifically designed for handling the elements of a container, and not only benefit from the iterator orientation of container interfaces, but are also capable of exposing many more programming bugs than raw pointers, both at compile and run time. iterator_to is thus meant to be used in scenarios where access via iterators is not suitable or desireable:
>
> - Interoperability with preexisting APIs based on pointers or references.
> - Publication of pointer-based interfaces (for instance, when designing a C-compatible library).
> - The exposure of pointers in place of iterators can act as a type erasure barrier effectively decoupling the user of the code from the implementation detail of which particular container is being used. Similar techniques, like the famous Pimpl idiom, are used in large projects to reduce dependencies and build times.
> - Self-referencing contexts where an element acts upon its owner container and no iterator to itself is available.
In other words, iterator_to is the perfect tool for the job by the last reason given. Under the hood it should just be a simple pointer cast and have no major runtime overhead (depending on if the function call is inlined).
Edit by laanwj: removed at sign from the description
ACKs for top commit:
jonatack:
re-ACK 296be8f per `git range-diff ab338a19 3ba1665 296be8f`, sanity check gcc 10.2 debug build is clean.
hebasto:
re-ACK 296be8f58e, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19478#pullrequestreview-482400727) review (verified with `git range-diff`).
Tree-SHA512: f5c30a4936fcde6ae32a02823c303b3568a747c2681d11f87df88a149f984a6d3b4c81f391859afbeb68864ef7f6a3d8779f74a58e3de701b3d51f78e498682e
zmqconfig.h is currently not really needed anywhere, except that
it declares zmqError (which is then defined in
zmqnotificationinterface.cpp). Note in particular that there is
no need to conditionally include zmq.h only if ZMQ is enabled, because
the place in the core code where the ZMQ library itself is included
(init.cpp) is conditional already on that.
This commit removes zmqconfig.h and replaces it by a much simpler
zmqutil.h library for zmqError. The definition of the function is
moved to the matching (newly created) zmqutil.cpp.
Instead of returning a raw pointer from CZMQNotifierFactory and
implicitly requiring the caller to know that it has to take ownership,
return a std::unique_ptr to make this explicit.
This also changes the typedef for CZMQNotifierFactory to use the new
C++11 using syntax, which makes it (a little) less cryptic.
This factors out the common logic to run over all ZMQ notifiers, call a
function on them, and remove them from the list if the function fails is
extracted to a helper method.
Note that this also fixes a potential memory leak: When a notifier was
removed previously after its callback returned false, it would just be
removed from the list without destructing the object. This is now done
correctly by std::unique_ptr behind the scenes.
This is a pure refactoring of zmqnotificationinterface to make the
code easier to read and maintain. It replaces explicit iterators
with C++11 for-each loops where appropriate and uses std::unique_ptr
to make memory ownership more explicit.
fafb381af8 Remove mempool global (MarcoFalke)
fa0359c5b3 Remove mempool global from p2p (MarcoFalke)
eeee1104d7 Remove mempool global from init (MarcoFalke)
Pull request description:
This refactor unlocks some nice potential features, such as, but not limited to:
* Removing the fee estimates global (would avoid slightly fragile workarounds such as #18766)
* Making the mempool optional for a "blocksonly" operation mode
Even absent those features, the new code without the global should be easier to maintain, read and write tests for.
ACKs for top commit:
jnewbery:
utACK fafb381af8
hebasto:
ACK fafb381af8, I have reviewed the code and it looks OK, I agree it can be merged.
darosior:
ACK fafb381af8
Tree-SHA512: a2e696dc377e2e81eaf9c389e6d13dde4a48d81f3538df88f4da502d3012dd61078495140ab5a5854f360a06249fe0e1f6a094c4e006d8b5cc2552a946becf26
7bf6dfbb48 wallet: Remove path checking code from bitcoin-wallet tool (Russell Yanofsky)
77d5bb72b8 wallet: Remove path checking code from createwallet RPC (Russell Yanofsky)
a987438e9d wallet: Remove path checking code from loadwallet RPC (Russell Yanofsky)
8b5e7297c0 refactor: Pass wallet database into CWallet::Create (Russell Yanofsky)
3c815cfe54 wallet: Remove Verify and IsLoaded methods (Russell Yanofsky)
0d94e60625 refactor: Use DatabaseStatus and DatabaseOptions types (Russell Yanofsky)
b5b414151a wallet: Add MakeDatabase function (Russell Yanofsky)
288b4ffb6b Remove WalletLocation class (Russell Yanofsky)
Pull request description:
Get rid of file path handling in wallet application code and move it down to database layer.
There is no change in behavior except for some changed error messages.
Motivation for this change is to make code more understandable, but also to prepare for adding SQLite support in #19077 so SQLite implementation can be contained at the database layer and wallet loading code does not need to become more complicated.
ACKs for top commit:
achow101:
ACK 7bf6dfbb48
meshcollider:
Code re-review and functional test run ACK 7bf6dfbb48
Tree-SHA512: 23ad18324c9e8947f0cf88a3734c2e9fb25536b2cb4d552cf5d1a4ade320fbffb73bb2d1b3a99585c11630aa7092e0fcfc2dd4fe65b91e3a54161433a5cd13cb
fa9ee52556 doc: Add doxygen comment to IsRBFOptIn (MarcoFalke)
faef4fc9b4 Remove mempool global from interfaces (MarcoFalke)
fa831684e5 refactor: Add IsRBFOptInEmptyMempool (MarcoFalke)
Pull request description:
The chain interface has an `m_node` member, which has a pointer to the mempool global. Use the pointer instead of the global to prepare the removal of the mempool global. See #19556
ACKs for top commit:
jnewbery:
utACK fa9ee52556
darosior:
ACK fa9ee52
hebasto:
re-ACK fa9ee52556, since my [previous](https://github.com/bitcoin/bitcoin/pull/19848#pullrequestreview-482403942) review:
Tree-SHA512: 11b4c1446f0860a743fdaa67f95c52bf0262d0a4f888be0eaf07ee497448965d32be414111bf016bd568f2989cde923430e3a3889e224057b73c499f06de7199
86d4cf42d9 Increase the ip address relay branching factor for unreachable networks (Pieter Wuille)
Pull request description:
Onion addresses propagate very badly among the IPv4/IPv6 network, resulting
in difficulty for those to find each other.
The branching factor 1 is probably so low that propagations die out before
they reach another onion peer. Increase it to 1.5 on average.
ACKs for top commit:
practicalswift:
ACK 86d4cf42d9 -- patch looks correct
naumenkogs:
ACK 86d4cf4
jonatack:
ACK 86d4cf42d9. Code review, built and running with some sanity check logging. `RelayAddress()` is called by `ProcessMessage() ADDR` msg handling, from within the loop while processing each new address to relay it to a limited number of other nodes. According to git blame, the line setting `nRelayNodes` hasn't been touched since 2016 in e736772c56 *Move network-msg-processing code out of main to its own file*, which moved the line but otherwise did not change it. Running a mixed clearnet/onion node with this patch and the logging below, I'm only seeing values of `fReachable 1, nRelayNodes 2`. IIUC, I need to use the settings in `init.cpp` that call `SetReachable(*, false)`. *Edit:* with `onlynet=onion` am now seeing entries of `fReachable 0` with `nRelayNodes` values of 1 and 2.
vasild:
ACK 86d4cf42d
Tree-SHA512: 22391e16d60bcfdec9a9336728da39d68a24a183b3d1b0e8fbc038d265ca6ddf71d16db018f3678745fd9f3e9281049e42197fa0a29124833c50a9170ed6f793
Previously, `tx` was being read after having `std::move` called on it. The
std::move operator indicates to the compiler that this object may be "moved
from", so we shouldn't subsequently read from it. The current code is not
problematic since tx is passed in as a const ref. But this `std::move` is at
best misleading & at worst problematic, so remove it.
The else clause is dead code because the only way to not enter the if branch is
if TX_WITNESS_STRIPPED is true. In that case, it would not have a witness to
match the `tx.HasWitness()` else condition.
Co-authored-by: Adam Jonas <jonas@chaincode.com>
Co-authored-by: John Newbery <john@johnnewbery.com>
When I originally implemented the unbroadcast set in 18038, it just tracked
txids. After 18038 was merged, I offered a patch to 18044 to make the
unbroadcast changes compatible with wtxid relay. In this patch, I updated
`unbroadcast_txids` to a map of txid -> wtxid. Post merge review comments shed
light on the fact that this update was unnecessary, and distracting. So, this
commit updates the unbroadcast ids back to a set.
581b343d5b Add in/out connections to cli -getinfo (Jon Atack)
d9cc13e88d UNIX_EPOCH_TIME fixup in rpc getnettotals (Jon Atack)
1ab49b81cf Add in/out connections to rpc getnetworkinfo (Jon Atack)
Pull request description:
This is basic info that is present in the GUI that I've been wishing to have exposed via the RPC and CLI without needing a bash workaround or script. For human users it would also be useful to have it in `-getinfo`.
`bitcoin-cli getnetworkinfo`
```
"connections": 15,
"connections_in": 6,
"connections_out": 9,
```
`bitcoin-cli -getinfo`
```
"connections": {
"in": 6,
"out": 9,
"total": 15
},
```
Update the tests, RPC help, and release notes for the changes. Also fixup the `getnettotals` timemillis help while touching `rpc/net.cpp`.
-----
Reviewers can manually test this PR by [building from source](https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests), launching bitcoind, and then running `bitcoin-cli -getinfo`, `bitcoin-cli getnetworkinfo`, `bitcoin-cli help getnetworkinfo`, and `bitcoin-cli help getnettotals` (for the UNIX epoch time change).
ACKs for top commit:
eriknylund:
> tACK [581b343](581b343d5b) on master at [a0a422c](a0a422c34c), ran unit & functional tests and and confirmed changes on an existing datadir ✌️
benthecarman:
tACK `581b343`
willcl-ark:
tACK for 581b343d5b, this time rebased onto master at 862fde88be.
shesek:
tACK `581b343`. This provides what I needed, thanks!
n-thumann:
tACK 581b343 on master at a0a422c, ran unit & functional tests and and confirmed changes on an existing datadir ✌️
Tree-SHA512: 08dd3ac8fefae401bd8253ff3ac027603c528eeccba53cedcb127771316173a7052fce44af8fa33ac98ebc4cf2a2b11cdefd949995d55e9b9a5942b876d00dc5
020f0519ec refactor: CTxMemPool::IsUnbroadcastTx() requires CTxMemPool::cs lock (Hennadii Stepanov)
7c4bd0387a refactor: CTxMemPool::GetTotalTxSize() requires CTxMemPool::cs lock (Hennadii Stepanov)
fa5fcb032b refactor: CTxMemPool::ClearPrioritisation() requires CTxMemPool::cs lock (Hennadii Stepanov)
7140b31b90 refactor: CTxMemPool::ApplyDelta() requires CTxMemPool::cs lock (Hennadii Stepanov)
66e47e5e50 refactor: CTxMemPool::UpdateChild() requires CTxMemPool::cs lock (Hennadii Stepanov)
939807768a refactor: CTxMemPool::UpdateParent() requires CTxMemPool::cs lock (Hennadii Stepanov)
Pull request description:
This is another step to transit `CTxMemPool::cs` from `RecursiveMutex` to `Mutex`.
Split out from #19306.
Only trivial thread safety annotations and lock assertions added. No new locks. No behavior change.
Refactoring `const uint256` to `const uint256&` was [requested](https://github.com/bitcoin/bitcoin/pull/19647#discussion_r468471022) by **promag**.
Please note that now, since #19668 has been merged, it is safe to apply `AssertLockHeld()` macros as they do not swallow compile time Thread Safety Analysis warnings.
ACKs for top commit:
promag:
Core review ACK 020f0519ec.
jnewbery:
Code review ACK 020f0519ec
vasild:
ACK 020f0519e
Tree-SHA512: a31e389142d5a19b25fef0aaf1072a337278564528b5cc9209df88ae548a31440e1b8dd9bae0169fd7aa59ea06e22fe5e0413955386512b83ef1f3e7d941e890
f1ee37319a wallet: Reload previously loaded wallets on GUI startup (Andrew Chow)
Pull request description:
Enable the GUI to also use the load_on_startup feature. Wallets loaded in the GUI always have load_on_startup=true. When they are unloaded from the GUI, load_on_startup=false.
To facilitate this change, UpdateWalletSetting is moved into the wallet module and called from within LoadWallet, RemoveWallet, and Createwallet. This change does not actually touch the GUI code but rather the wallet functions that are shared between the GUI and RPC.
ACKs for top commit:
jonasschnelli:
Tested ACK f1ee37319a - works as expected. Wallets loaded via bitcoin-cli (in `-server` mode) or through the RPC console won't be loaded on startup but wallets loaded via the GUI menu will.
kristapsk:
ACK f1ee37319a, I have tested the code.
Tree-SHA512: f5b44aa763cf761d919015c5fbc0600b72434aa71e3b57007fd7530a29c3da1a9a0c98c4f22cb6cdffba61150a31170056a7d4737625e7b76f6958f3d584da8c
Checks are now consolidated in MakeBerkeleyDatabase function instead of
happening in higher level code.
This commit does not change behavior except for error messages which now
include more complete information.
New function is not currently called but will be called in upcoming commits. It
moves database path checking, and existence checking, and already-loaded
checking, and verification into a single function so this logic does not need
to be repeated all over higher level wallet code, and so higher level code does
not need to change when SQLite support is added in
https://github.com/bitcoin/bitcoin/pull/19077. This also lets higher level
wallet code make fewer assumptions about the contents of wallet directories.
This commit just adds the new function and does not change behavior in any way.
This removes a source of complexity and indirection that makes it harder to
understand path checking code. Path checks will be simplified in upcoming
commits.
There is no change in behavior in this commit other than a slightly more
descriptive error message in `loadwallet` if the default "" wallet can't be
found. (The error message is improved more in upcoming commit "wallet: Remove
path checking code from loadwallet RPC".)
752e6ad533 Protect localhost and block-relay-only peers from eviction (Suhas Daftuar)
Pull request description:
Onion peers are disadvantaged under our eviction criteria, so prevent eventual
eviction of them in the presence of contention for inbound slots by reserving
some slots for localhost peers (sorted by longest uptime).
Block-relay-only connections exist as a protection against eclipse attacks, by
creating a path for block propagation that may be unknown to adversaries.
Protect against inbound peer connection slot attacks from disconnecting such
peers by attempting to protect up to 8 peers that are not relaying transactions
but have provided us with blocks.
Thanks to gmaxwell for suggesting these strategies.
ACKs for top commit:
laanwj:
Code review ACK 752e6ad533
Tree-SHA512: dbf089c77c1f747aa1dbbbc2e9c2799c628028b0918d0c336d8d0e5338acedd573b530eb3b689c7f603a17221e557268a9f5c3f585f204bfb12e5d2e76de39a3
0bbe26a1af wallet: filter for keys only before record deser in salvage (Andrew Chow)
544e12a4e8 walletdb: Add KeyFilterFn to ReadKeyValue (Andrew Chow)
Pull request description:
When salvaging a wallet, the only things that matter are the private keys. It is not necessary to attempt to deserialize any other records, especially if those records are corrupted too.
This PR adds a `KeyFilterFn` function callback to `ReadKeyValue` that salvage uses to filter for only the records that it wants. Of course doing it this way also lets us do other filters in the future from other places should we so desire.
ACKs for top commit:
ryanofsky:
Code review ACK 0bbe26a1af. Looks great! This should make the recovery code more robust. Normally it'd be good to have a test case for the problem this fixes, but Marco already wrote one in #19078, so I think we're covered
laanwj:
Code review ACK 0bbe26a1af
Tree-SHA512: 8e3ee283a22a79273915711c4fb751f3c9b02ce94e6bf08dc468f1cfdf9fac35c693bbfd2435ce43c3a06c601b9b0a67e209621f6814bedfe3bc7a7ccc37bb01
e36f802fa4 lint: add C++ code linter (fanquake)
c4be50fea3 remove usage of boost::bind (fanquake)
Pull request description:
`boost::bind` usage was removed in #13743. However a new usage snuck in as
part of 2bc4c3eaf9 (#15225).
ACKs for top commit:
hebasto:
ACK e36f802fa4
practicalswift:
ACK e36f802fa4 -- patch looks correct
Tree-SHA512: 2b0387c5443c184bcbf7df4849db1ed1296ff82c7b4ff0aff18334a400e56a472a972d18234d3866531a088d7a8da64688e58dc9f15daaad4048697c759d55ce
Previously we deduced it was a block-relay-only based on presence of the
m_tx_relay structure. Now we have the ability to identify it directly via a
connection type accessor function.
We previously identified if we relay addresses to the connection by checking
for the existence of the m_addr_known data structure. With this commit, we
answer this question based on the connection type.
IsAddrRelayPeer() checked for the existence of the m_addr_known
Onion peers are disadvantaged under our eviction criteria, so prevent eventual
eviction of them in the presence of contention for inbound slots by reserving
some slots for localhost peers (sorted by longest uptime).
Block-relay-only connections exist as a protection against eclipse attacks, by
creating a path for block propagation that may be unknown to adversaries.
Protect against inbound peer connection slot attacks from disconnecting such
peers by attempting to protect up to 8 peers that are not relaying transactions
but appear to be full-nodes, sorted by recency of last delivered block.
Thanks to gmaxwell for suggesting these strategies.
d780293e1e net: improve nLastBlockTime and nLastTXTime documentation (Jon Atack)
Pull request description:
Follow-up to #19731 to help alleviate confusion around `nLastBlockTime` and `nLastTXTime`, now also provided by the JSON-RPC API as `last_block` and `last_transaction` in `getpeerinfo` output.
Thanks to John Newbery, credited in the commit, and to Dave Harding and Adam Jonas during discussions on how to best explain these in this week's Optech newsletter.
ACKs for top commit:
practicalswift:
ACK d780293e1e
MarcoFalke:
ACK d780293e1e
harding:
ACK d780293e1e . The added documentation matches my reading of the code and answers a question I had after seeing #19731
0xB10C:
ACK d780293e1e
Tree-SHA512: 72d47cf50a099913c7e4753cb80e11785b26fb66fa3a8b6c382fde4ea725116f3d215f93d32a567246d269768e66159f8dcf017a1bbc6d5f2489a35f81c316fa
fb56d37612 p2p: ensure inv is GenMsgTx before ToGenTxid in inv processing (John Newbery)
aa3621385e test: use CInv::MSG_WITNESS_TX flag in p2p_segwit (Jon Atack)
24ee4f01ea p2p: make gtxid(.hash) and fAlreadyHave localvars const (Jon Atack)
b1c855453b p2p: use CInv block message helpers in net_processing.cpp (Jon Atack)
acd6642167 [net processing] Change AlreadyHaveTx() to take a GenTxid (John Newbery)
5fdfb80b86 [net processing] Change AlreadyHaveBlock() to take block_hash argument (John Newbery)
430e183b89 [net processing] Remove mempool argument from AlreadyHaveBlock() (John Newbery)
42ca5618ca [net processing] Split AlreadyHave() into separate block and tx functions (John Newbery)
39f1dc9445 p2p: remove nFetchFlags from NetMsgType TX and INV processing (Jon Atack)
471714e1f0 p2p: add CInv block message helper methods (Jon Atack)
Pull request description:
Building on #19590 and the recent `wtxid` and `GenTxid` changes, this is a refactoring and cleanup PR to simplify and improve some of the net processing code.
Some of the diffs are best reviewed with `-w` to ignore spacing.
Co-authored by John Newbery.
ACKs for top commit:
laanwj:
Code review ACK fb56d37612
jnewbery:
utACK fb56d37612
vasild:
ACK fb56d3761
Tree-SHA512: ba39b58e6aaf850880a842fe5f6295e9f1870906ef690206acfc17140aae2ac854981e1066dbcd4238062478762fbd040ef772fdc2c50eea6869997c583e6a6d
c276df7759 zmq: enable tcp keepalive (mruddy)
Pull request description:
This addresses https://github.com/bitcoin/bitcoin/issues/12754.
These changes enable node operators to address the silent dropping (by network middle boxes) of long-lived low-activity ZMQ TCP connections via further operating system level TCP keepalive configuration. For example, ZMQ sockets that publish block hashes can be affected in this way due to the length of time it sometimes takes between finding blocks (e.g.- sometimes more than an hour).
Prior to this patch, operating system level TCP keepalive configurations would not take effect since the SO_KEEPALIVE option was not enabled on the underlying socket.
There are additional ZMQ socket options related to TCP keepalive that can be set. However, I decided not to implement those options in this changeset because doing so would require adding additional bitcoin node configuration options, and would not yield a better outcome. I preferred a small, easily reviewable patch that doesn't add a bunch of new config options, with the tradeoff that the fine tuning would have to be done via well-documented operating system specific configurations.
I tested this patch by running a node with:
`./src/qt/bitcoin-qt -regtest -txindex -datadir=/tmp/node -zmqpubhashblock=tcp://127.0.0.1:28332 &`
and connecting to it with:
`python3 ./contrib/zmq/zmq_sub.py`
Without these changes, `ss -panto | grep 28332 | grep ESTAB | grep bitcoin` will report no keepalive timer information. With these changes, the output from the prior command will show keepalive timer information consistent with the configuration at the time of connection establishment, e.g.-: `timer:(keepalive,119min,0)`.
I also tested with a non-TCP transport and did not witness any adverse effects:
`./src/qt/bitcoin-qt -regtest -txindex -datadir=/tmp/node -zmqpubhashblock=ipc:///tmp/bitcoin.block &`
ACKs for top commit:
adamjonas:
Just to summarize for those looking to review - as of c276df7759 there are 3 tACKs (n-thumann, Haaroon, and dlogemann), 1 "looks good to me" (laanwj) with no NACKs or any show-stopping concerns raised.
jonasschnelli:
utACK c276df7759
Tree-SHA512: b884c2c9814e97e666546a7188c48f9de9541499a11a934bd48dd16169a900c900fa519feb3b1cb7e9915fc7539aac2829c7806b5937b4e1409b4805f3ef6cd1
Enable the GUI to also use the load_on_startup feature.
Wallets loaded in the GUI always have load_on_startup=true.
When they are unloaded from the GUI, load_on_startup=false.
To facilitate this change, UpdateWalletSetting is moved into the wallet
module and called from within LoadWallet, RemoveWallet, and
Createwallet. This change does not actually touch the GUI code but
rather the wallet functions that are shared between the GUI and RPC.
ea74e10acf doc: Add best practice for annotating/asserting locks (Hennadii Stepanov)
2ee7743fe7 sync.h: Make runtime lock checks require compile-time lock checks (Anthony Towns)
23d71d171e Do not hide compile-time thread safety warnings (Hennadii Stepanov)
3ddc150857 Add missed thread safety annotations (Hennadii Stepanov)
af9ea55a72 Use LockAssertion utility class instead of AssertLockHeld() (Hennadii Stepanov)
Pull request description:
On the way of transit from `RecursiveMutex` to `Mutex` (see #19303) it is crucial to have run-time `AssertLockHeld()` assertion that does _not_ hide compile-time Clang Thread Safety Analysis warnings.
On master (65e4ecabd5) using `AssertLockHeld()` could hide Clang Thread Safety Analysis warnings, e.g., with the following patch applied:
```diff
--- a/src/txmempool.h
+++ b/src/txmempool.h
@@ -607,7 +607,7 @@ public:
void addUnchecked(const CTxMemPoolEntry& entry, setEntries& setAncestors, bool validFeeEstimate = true) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main);
void removeRecursive(const CTransaction& tx, MemPoolRemovalReason reason) EXCLUSIVE_LOCKS_REQUIRED(cs);
- void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main);
+ void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
void removeConflicts(const CTransaction& tx) EXCLUSIVE_LOCKS_REQUIRED(cs);
void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int nBlockHeight) EXCLUSIVE_LOCKS_REQUIRED(cs);
```
Clang compiles the code without any thread safety warnings.
See "Add missed thread safety annotations" commit for the actual thread safety warnings that are fixed in this PR.
ACKs for top commit:
MarcoFalke:
ACK ea74e10acf 🎙
jnewbery:
ACK ea74e10acf
ajtowns:
ACK ea74e10acf
Tree-SHA512: 8cba996e526751a1cb0e613c0cc1b10f027a3e9945fbfb4bd30f6355fd36b9f9c2e1e95ed3183fc254b42df7c30223278e18e5bdb5e1ef85db7fef067595d447
3340dbadd3 Remove -zapwallettxes (Andrew Chow)
Pull request description:
It's not clear what use there is to keeping `-zapwallettxes` given that it's intended usage has been superseded by `abandontransaction`. So this removes it outright.
Alternative to #19700
ACKs for top commit:
meshcollider:
utACK 3340dbadd3
fanquake:
ACK 3340dbadd3 - remaining manpage references will get cleaned up pre-release.
Tree-SHA512: 3e58e1ef6f4f94894d012b93e88baba3fb9c2ad75b8349403f9ce95b80b50b0b4f443cb623cf76c355930db109f491b3442be3aa02972e841450ce52cf545fc8
-zapwallettxes is made a hidden option to inform users that it is
removed and they should be using abandontransaction to do the stuck
transaction thing.
fa3d9ce325 rpc: Assert that RPCArg names are equal to CRPCCommand ones (rpcdump) (MarcoFalke)
fa32c1d5ec rpc: Assert that RPCArg names are equal to CRPCCommand ones (zmq) (MarcoFalke)
faaa46dc20 rpc: Assert that RPCArg names are equal to CRPCCommand ones (mining) (MarcoFalke)
fa93bc14c7 rpc: Remove unused return type from appendCommand (MarcoFalke)
Pull request description:
This is split out from #18531 to just touch the RPC methods in misc. Description from the main pr:
### Motivation
RPCArg names in the rpc help are currently only used for documentation. However, in the future they could be used to teach the server the named arguments. Named arguments are currently registered by the `CRPCCommand`s and duplicate the RPCArg names from the documentation. This redundancy is fragile, and has lead to errors in the past (despite having linters to catch those kind of errors). See section "bugs found" for a list of bugs that have been found as a result of the changes here.
### Changes
The changes here add an assert in the `CRPCCommand` constructor that the RPCArg names are identical to the ones in the `CRPCCommand`.
### Future work
> Here or follow up, makes sense to also assert type of returned UniValue?
Sure, but let's not get ahead of ourselves. I am going to submit any further works as follow-ups, including:
* Removing the CRPCCommand arguments, now that they are asserted to be equal and thus redundant
* Removing all python regex linters on the args, now that RPCMan can be used to generate any output, including the cli.cpp table
* Auto-formatting and sanity checking the RPCExamples with RPCMan
* Checking passed-in json in self-check. Removing redundant checks
* Checking returned json against documentation to avoid regressions or false documentation
* Compile the RPC documentation at compile-time to ensure it doesn't change at runtime and is completely static
### Bugs found
* The assert identified issue #18607
* The changes itself fixed bug #19250
ACKs for top commit:
fjahr:
tested ACK fa3d9ce325
promag:
Code review ACK fa3d9ce325.
Tree-SHA512: 068ade4b55cc195868d53b7f9a27151d45b440857bb069e261a49d102a49a38fdba5d68868516a1d66a54a73ba34681362f934ded7349e894042bde873b75719
