Commit graph

2461 commits

Author SHA1 Message Date
MarcoFalke c1e0c2ad3b
Merge #19813: util, ci: Hard code previous release tarball checksums
0374e821bd util: Hard code previous release tarball checksums (Hennadii Stepanov)
bd897ce79f scripted-diff: Move previous_release.py to test/get_previous_releases.py (Hennadii Stepanov)

Pull request description:

  #19205 introduced signature verifying for the downloaded `SHA256SUMS.asc`.
  This approach is brittle and does not work in CI environment for many reasons:
  - https://github.com/bitcoin/bitcoin/issues/19812#issuecomment-680760663
  - https://github.com/bitcoin/bitcoin/pull/19013#discussion_r459590779

  This PR:
  - implements **Sjors**' [idea](https://github.com/bitcoin/bitcoin/pull/19205#pullrequestreview-426080048):
  > Alternatively we might as well hard code the checksum for each `tar.gz` release in the source code, here.

  - is an alternative to 5a2c31e528e6bd60635096f233252f3c717f366d (#19013)

  - fixes #19812

  - updates v0.17.1 to v0.17.2

ACKs for top commit:
  MarcoFalke:
    cr ACK 0374e821bd
  Sjors:
    tACK 0374e821bd

Tree-SHA512: cacdcf9f5209eae7da357abb3445585ad2f980920fd5bf75527ce89974d3f531a4cf8b5b35edfc116b23bfdfb45c0437cb14cbc416d76ed2dc5b9e6d33cdad71
2020-08-31 16:18:29 +02:00
Samuel Dobson f98872f127
Merge #18244: rpc: fundrawtransaction and walletcreatefundedpsbt also lock manually selected coins
6d1f51343c [rpc] fundrawtransaction, walletcreatefundedpsbt lock manually selected coins (Sjors Provoost)

Pull request description:

  When using `fundrawtransaction` and `walletcreatefundedpsbt` with `lockUnspents`, it would only lock automatically selected coins, not manually selected coins. That doesn't make much sense to me if the goal is to prevent accidentally double-spending yourself before you broadcast a transaction.

  Note that when  creating a transaction, manually selected coins are automatic "unlocked" (or more accurately: the lock is ignored). Earlier versions of this PR introduced an error when a locked coin is manually selected, but this idea was abandoned after some discussion. An application that uses this RPC should either rely on automatic coin selection (with `lockUnspents`) or handle lock concurrency itself with manual coin selection. In particular it needs to make sure to avoid/pause calls with automatic coin selection between calling `lockunspent` and the subsequent spending RPC.

  See #7518 for historical background.

ACKs for top commit:
  meshcollider:
    Code review ACK 6d1f51343c
  fjahr:
    Code review ACK 6d1f51343c

Tree-SHA512: 8773c788d92f2656952e1beac147ba9956b8c5132d474e0880e4c89ff53642928b4cbfcd1cb3d17798b9284f02618a8830c93a9f7a4733e5bded96adff1d5d4d
2020-08-31 23:30:53 +12:00
Hennadii Stepanov bd897ce79f
scripted-diff: Move previous_release.py to test/get_previous_releases.py
-BEGIN VERIFY SCRIPT-
OLD=contrib/devtools/previous_release.py
NEW=test/get_previous_releases.py
sed -i "s|$OLD|$NEW|g" $(git grep -l $OLD)
git mv $OLD $NEW
-END VERIFY SCRIPT-
2020-08-29 11:26:25 +03:00
Jon Atack 7984c39be1
test framework: serialize/deserialize inv type as unsigned int 2020-08-28 20:12:02 +02:00
MarcoFalke fa1cd9e1dd
test: Remove unused lock arg from BitcoinTestFramework.wait_until 2020-08-27 18:50:08 +02:00
MarcoFalke fad2794e93
test: Rename wait until helper to wait_until_helper 2020-08-27 18:50:05 +02:00
MarcoFalke facb41bf1d
test: Remove unused p2p_lock in VersionBitsWarningTest 2020-08-27 11:41:40 +02:00
MarcoFalke 28f4e53e16
Merge #19752: test: Update wait_until usage in tests not to use the one from utils
d841301010 test: Add docstring to wait_until() in util.py to warn about its usage (Seleme Topuz)
1343c86c7c test: Update wait_until usage in tests not to use the one from utils (Seleme Topuz)

Pull request description:

  Replace global (from [test_framework/util.py](https://github.com/bitcoin/bitcoin/blob/master/test/functional/test_framework/util.py#L228)) `wait_until()` usages with the ones provided by `BitcoinTestFramework` and `P2PInterface` classes.

  The motivation behind this change is that the `util.wait_until()` expects a timeout, timeout_factor and lock and it is not aware of the context of the test framework. `BitcoinTestFramework` offers a `wait_until()` which has an understandable amount of default `timeout` and a shared `timeout_factor`. Moreover, on top of these, `mininode.wait_until()` also has a shared lock.

  closes #19080

ACKs for top commit:
  MarcoFalke:
    ACK d841301010 🦆
  kallewoof:
    utACK d841301010

Tree-SHA512: 81604f4cfa87fed98071a80e4afe940b3897fe65cf680a69619a93e97d45f25b313c12227de7040e19517fa9c003291b232f1b40b2567aba0148f22c23c47a88
2020-08-27 08:21:59 +02:00
MarcoFalke 30568d3f1e
Merge #19778: test: Fix intermittent issue in wallet_bumpfee
fafc9d5af4 test: Fix intermittent issue in wallet_bumpfee (MarcoFalke)
fa347b2f25 test: Select at least the fee in wallet_bumpfee to avoid negative amounts (MarcoFalke)

Pull request description:

  With a "dirty" mempool a transaction might fail to be accepted intermittently. For example,

  * https://travis-ci.org/github/bitcoin-core/gui/jobs/719916499#L6773 Fails acceptance
  * https://travis-ci.org/github/bitcoin-core/gui/jobs/719916499#L6954 Test fails

  Fix the issue by clearing the mempool between subtests

ACKs for top commit:
  promag:
    Code review ACK fafc9d5af4.

Tree-SHA512: 23fb6decb6343d19eafddcbdb7da0551f6be11325d1c97c30e563944000aeb02bcc4b24904d204b132c093dc1acf28445fa1fd08bfe8d8b52ddd1de51c33eeb6
2020-08-26 19:25:38 +02:00
Seleme Topuz d841301010 test: Add docstring to wait_until() in util.py to warn about its usage 2020-08-26 18:01:59 +02:00
Seleme Topuz 1343c86c7c test: Update wait_until usage in tests not to use the one from utils
Replace "wait_until()" usage from utils, with the ones from BitcoinTestFramework and P2PInterface.
closes #19080
2020-08-26 18:01:59 +02:00
Jon Atack aa3621385e
test: use CInv::MSG_WITNESS_TX flag in p2p_segwit 2020-08-26 11:57:27 +02:00
fanquake e80e5b3e4f
Merge #19760: test: Remove confusing mininode terminology
d5800da519 [test] Remove final references to mininode (John Newbery)
5e8df3312e test: resort imports (John Newbery)
85165d4332 scripted-diff: Rename mininode to p2p (John Newbery)
9e2897d020 scripted-diff: Rename mininode_lock to p2p_lock (John Newbery)

Pull request description:

  New contributors are often confused by the terminology in the test framework, and what the difference between a _node_ and a _peer_ is. To summarize:

  - a 'node' is a bitcoind instance. This is the thing whose behavior is being tested. Each bitcoind node is managed by a python `TestNode` object which is used to start/stop the node, manage the node's data directory, read state about the node (eg process status, log file), and interact with the node over different interfaces.
  - one of the interfaces that we can use to interact with the node is the p2p interface. Each connection to a node using this interface is managed by a python `P2PInterface` or derived object (which is owned by the `TestNode` object). We can open zero, one or many p2p connections to each bitcoind node. The node sees these connections as 'peers'.

  For historic reasons, the word 'mininode' has been used to refer to those p2p interface objects that we use to connect to the bitcoind node (the code was originally taken from the 'mini-node' branch of https://github.com/jgarzik/pynode/tree/mini-node). However that name has proved to be confusing for new contributors, so rename the remaining references.

ACKs for top commit:
  amitiuttarwar:
    ACK d5800da519
  MarcoFalke:
    ACK d5800da519 🚞

Tree-SHA512: 2c46c2ac3c4278b6e3c647cfd8108428a41e80788fc4f0e386e5b0c47675bc687d94779496c09a3e5ea1319617295be10c422adeeff2d2bd68378e00e0eeb5de
2020-08-26 15:43:17 +08:00
MarcoFalke fa39c62eb7
test: inline hashToHex 2020-08-25 14:27:25 +02:00
John Newbery d5800da519 [test] Remove final references to mininode 2020-08-25 10:04:25 +01:00
Jon Atack 581b343d5b
Add in/out connections to cli -getinfo 2020-08-24 18:41:24 +02:00
Jon Atack 1ab49b81cf
Add in/out connections to rpc getnetworkinfo 2020-08-24 18:41:14 +02:00
Wladimir J. van der Laan 7f609f68d8
Merge #19731: net, rpc: expose nLastBlockTime/nLastTXTime as last block/last_transaction in getpeerinfo
5da96210fc doc: release note for getpeerinfo last_block/last_transaction (Jon Atack)
cfef5a2c98 test: rpc_net.py logging and test naming improvements (Jon Atack)
21c57bacda test: getpeerinfo last_block and last_transaction tests (Jon Atack)
8a560a7d57 rpc: expose nLastBlockTime/TXTime as getpeerinfo last_block/transaction (Jon Atack)
02fbe3ae0b net: add nLastBlockTime/TXTime to CNodeStats, CNode::copyStats (Jon Atack)

Pull request description:

  This PR adds inbound peer eviction criteria `nLastBlockTime` and `nLastTXTime` to `CNodeStats` and `CNode::copyStats`, which then allows exposing them in the next commit as `last_transaction` and `last_block` Unix Epoch Time fields in RPC `getpeerinfo`.

  This may be useful for writing missing eviction tests. I'd also like to add `lasttx` and `lastblk` columns to the `-netinfo` dashboard as described in https://github.com/bitcoin/bitcoin/pull/19643#issuecomment-671093420.

  Relevant discussion at the p2p irc meeting http://www.erisian.com.au/bitcoin-core-dev/log-2020-08-11.html#l-549:
  ```text
  <jonatack> i was specifically trying to observe and figure out how to test https://github.com/bitcoin/bitcoin/issues/19500
  <jonatack> which made me realise that i didn't know what was going on with my peer conns in enough detail
  <jonatack> i'm running bitcoin locally with nLastBlockTime and nLastTXTime added to getpeerinfo for my peer connections dashboard
  <jonatack> sipa: is there a good reason why that (eviction criteria) data is not exposed through getpeerinfo currently?
  <sipa> jonatack: nope; i suspect just nobody ever added it
  <jonatack> sipa: thanks. will propose.
  ```

  The last commit is optional, but I think it would be good to have logging in `rpc_net.py`.

ACKs for top commit:
  jnewbery:
    Code review ACK 5da96210fc
  theStack:
    Code Review ACK 5da96210fc
  darosior:
    ACK 5da96210fc

Tree-SHA512: 2db164bc979c014837a676e890869a128beb7cf40114853239e7280f57e768bcb43bff6c1ea76a61556212135281863b5290b50ff9d24fce16c5b89b55d4cd70
2020-08-24 17:03:07 +02:00
MarcoFalke fafc9d5af4
test: Fix intermittent issue in wallet_bumpfee 2020-08-24 13:49:28 +02:00
MarcoFalke fa347b2f25
test: Select at least the fee in wallet_bumpfee to avoid negative amounts 2020-08-24 13:44:07 +02:00
Sebastian Falbesoner 638441928a test: add parameterized constructor for msg_sendcmpct() 2020-08-23 02:27:09 +02:00
John Newbery 5e8df3312e test: resort imports 2020-08-21 15:53:59 +01:00
John Newbery 85165d4332 scripted-diff: Rename mininode to p2p
-BEGIN VERIFY SCRIPT-
sed -i 's/\.mininode/\.p2p/g' $(git grep -l "mininode")
git mv test/functional/test_framework/mininode.py test/functional/test_framework/p2p.py
-END VERIFY SCRIPT-
2020-08-21 15:52:20 +01:00
John Newbery 9e2897d020 scripted-diff: Rename mininode_lock to p2p_lock
-BEGIN VERIFY SCRIPT-
sed -i 's/mininode_lock/p2p_lock/g' $(git grep -l "mininode_lock")
-END VERIFY SCRIPT-
2020-08-21 15:52:13 +01:00
MarcoFalke d254e6e795
Merge #19722: test: Add test for getblockheader verboseness
5067c5acc3 [test] Add test for getblockheader verboseness (Torhte Butler)

Pull request description:

  Improve test coverage by adding a test for getblockheader with verbose argument set to false.

ACKs for top commit:
  theStack:
    ACK 5067c5acc3

Tree-SHA512: e55593f1026a89dc7b796fa985b4cbcdb596e91d80d42dfb0660bda1692aaa35749ec29f9cd7032803f6225afb323f085df1ef6a9982de87be8e098f7253cdd5
2020-08-21 14:34:02 +02:00
Wladimir J. van der Laan 27eeb0337b
Merge #19550: rpc: Add getindexinfo RPC
124e1ee134 doc: Add release notes for getindexinfo RPC (Fabian Jahr)
c447b09458 test: Add tests for getindexinfo RPC (Fabian Jahr)
667bc7a7f7 rpc: Add getindexinfo RPC (Fabian Jahr)

Pull request description:

  As I was playing with indices a I was missing an RPC that gives information about the active indices in the node. I think this can be helpful for many users, especially since there are some new index candidates coming up (#14053, #18000) that can give a quick overview without the user having to parse the logs.

  Feature summary:
  - Adds new RPC `listindices` (placed in Util section)
  - That RPC only lists the actively running indices
  - For each index it gives the name, whether it is synced and up to which block height it is synced

ACKs for top commit:
  laanwj:
    Re-ACK 124e1ee134
  jonatack:
    Code review re-ACK 124e1ee per `git range-diff a57af89 47a5372 124e1ee` no change since my last re-ACK, rebase only

Tree-SHA512: 3b7174c87951e6457fef099f530337803906baf32fb64261410b8def2c0917853d6a1bf3059cd590b1cc1523608f8916dafb327a431d27ecbf8d7454406b5b35
2020-08-20 16:00:22 +02:00
Dhruv Mehta ed5cd12869 test: Distinguish between nodes(bitcoind) and peers(mininodes) in p2p_leak.py
Also, remove "C" prefix from class names to match new style
2020-08-18 13:13:19 -07:00
Dhruv Mehta f6f082b934 test: remove CNodeNoVersionIdle from p2p_leak.py 2020-08-18 13:06:15 -07:00
Dhruv Mehta 45cf55ccac test: remove CNodeNoVersionMisbehavior from p2p_leak.py
It's also clearer to have `no_version_disconnect_node` send a message
other than version or verack in order to reach the peer discouragement
threshold.
2020-08-18 13:06:09 -07:00
Karl-Johan Alm 7e31ea9fa0
-maxapsfee: follow-up fixes
Co-authored-by: Jon Atack <jon@atack.com>
Co-authored-by: Samuel Dobson <dobsonsa68@gmail.com>
2020-08-18 19:24:39 +09:00
Karl-Johan Alm 72ae20fc14
tests: add sync_all to fix race condition in wallet groups test 2020-08-18 10:08:49 +09:00
Torhte Butler 5067c5acc3 [test] Add test for getblockheader verboseness
Add test for getblockheader with verbose argument set to false.
2020-08-17 08:09:48 +00:00
Samuel Dobson c831e105c5
Merge #14582: wallet: always do avoid partial spends if fees are within a specified range
7f13dfb587 test: test the implicit avoid partial spends functionality (Karl-Johan Alm)
b82067bf69 wallet: try -avoidpartialspends mode and use its result if fees are below threshold (Karl-Johan Alm)

Pull request description:

  The `-avoidpartialspends` feature is normally disabled, as it may affect the optimal fee for payments. This PR introduces a new parameter `-maxapsfee` (max avoid partial spends fee) which acts on the following values:
  * -1: disable partial spend avoidance completely (do not even try it)
  * 0: only do partial spend avoidance if fees are the same or better as the regular coin selection
  * 1..∞: use APS variant if the absolute fee difference is less than or equal to the max APS fee

  For values other than -1, the code will now try partial spend avoidance once, and if that gives a value within the accepted range, it will use that.

  Example: -maxapsfee=0.00001000 means the wallet will do regular coin select, APS coin select, and then pick AKS iff the absolute fee difference is <= 1000 satoshi.

  Edit: updated this to reflect the fact we are now using a max fee.

ACKs for top commit:
  fjahr:
    tested ACK 7f13dfb587
  achow101:
    ACK 7f13dfb587
  jonatack:
    ACK 7f13dfb58, code review, debug build, verified the test fails with `AssertionError: not(2 == 1)` for the number of vouts when `-maxapsfee=0.0001` is changed to 0, and verified the new logging with an added assertion.
  meshcollider:
    Code review ACK 7f13dfb587

Tree-SHA512: 475929df57f6191bb4e36bfbcad5a280a64bb0ecd8767b76cb2e44e2301235d0eb294a3f2fac5bbf15d35d7ecfba47acb2285feadb883c9ce31c08377e3afb3c
2020-08-17 16:18:28 +12:00
Fabian Jahr c447b09458
test: Add tests for getindexinfo RPC 2020-08-16 11:15:52 +02:00
MarcoFalke a57af897ec
Merge #19564: test: p2p_feefilter improvements (logging, refactoring, speedup)
9e7894357e test: speedup p2p_feefilter.py by whitelisting peers (immediate tx relay) (Sebastian Falbesoner)
fe3f0cc44e test: use wait_until for invs matching in p2p_feefilter.py (Sebastian Falbesoner)
6d941923c3 test: add logging for p2p_feefilter.py (Sebastian Falbesoner)

Pull request description:

  This PR gives some love to the functional test `p2p_feefilter.py` by introducing the following changes:
  * add missing log messages for the `test_feefilter` subtest (the main one)
  * deduplicate code by using the utility function `wait_until` (already using the [recently introduced version](12410b1feb)) instead of a manual condition checking loop with `time.sleep`
  * improve naming of the function `matchAllInvs` (more expressive name, snake_case) and moving it from global namespace to the connection class `FeefilterConn`
  * speeding up the test significantly by the good ol' method of activating immediate tx relay (as seen on e.g. https://github.com/bitcoin/bitcoin/pull/17121, https://github.com/bitcoin/bitcoin/pull/17124, https://github.com/bitcoin/bitcoin/pull/17340, https://github.com/bitcoin/bitcoin/pull/17362, ...):
  ```
  master branch:
  $ time ./p2p_feefilter.py
  ...
  real    0m39.367s
  user    0m1.227s
  sys 0m0.571s

  PR branch:
  $ time ./p2p_feefilter.py
  ...
  real    0m9.386s
  user    0m1.120s
  sys 0m0.577s
  ```

ACKs for top commit:
  instagibbs:
    code review ACK 9e7894357e
  jonatack:
    re-ACK 9e78943 per `git range-diff 3ab2582 ea74a3c 9e78943`

Tree-SHA512: fe21c1c5413df9165fea916b5d5f609d3ba33e7b5c3364b38eb824fcc55d9e6abddf27116cbc0b325913d451a73c44542040fb916aec9c46f805c6e12f6f10cf
2020-08-16 11:04:28 +02:00
MarcoFalke 3ab2582c7f
Merge #19730: ci: Set increased --timeout-factor by default
fa330ec2fe test: Remove confusing and broken use of wait_until global (MarcoFalke)
fa6583c30b ci: Set increased --timeout-factor by default (MarcoFalke)

Pull request description:

  Assuming that tests don't have a logic error or race, setting a high timeout should not cause any issues. The tests will still pass just as fast in the fastest case, but it allows for some buffer in case of slow disks or otherwise starved ci machines.

  Fixes #19729

ACKs for top commit:
  hebasto:
    ACK fa330ec2fe, I have reviewed the code, and it looks OK, I agree it can be merged.

Tree-SHA512: 3da80ee008c7b08bab5fdaf7804d57c79d6fed49a7d37b9c54fc89756659fcb9981fd10afc4d07bd90d93c1699fd410a0110a3cd34d016873759d114ce3cd82d
2020-08-15 15:39:02 +02:00
Jon Atack cfef5a2c98
test: rpc_net.py logging and test naming improvements 2020-08-15 15:13:40 +02:00
Jon Atack 21c57bacda
test: getpeerinfo last_block and last_transaction tests 2020-08-15 15:13:13 +02:00
MarcoFalke fa330ec2fe
test: Remove confusing and broken use of wait_until global 2020-08-15 14:01:54 +02:00
Samuel Dobson a0e75bd31d
Merge #15937: Add loadwallet and createwallet load_on_startup options
642ad31b41 Add loadwallet and createwallet RPC load_on_startup options (Russell Yanofsky)

Pull request description:

  This maintains a persistent list of wallets stored in settings that will automatically be loaded on startup. Being able to load a wallet automatically on startup will be more useful in the GUI, but it's reasonable to expose this feature by RPC as well.

ACKs for top commit:
  achow101:
    re-ACK 642ad31b41 Only change is the test
  meshcollider:
    re-utACK 642ad31b41

Tree-SHA512: cca0b71bf1a83ad071830e6c459f1cd979b4add7144e899ec560da72b5910dd9bf9426e5c7d125ae96fad8990fbf81a76bc83c0459486c16086ada6cbde5eaa3
2020-08-15 12:19:48 +12:00
MarcoFalke faaa46dc20
rpc: Assert that RPCArg names are equal to CRPCCommand ones (mining) 2020-08-14 12:37:19 +02:00
Wladimir J. van der Laan 4d4bd5ed74
Merge #17204: wallet: Do not turn OP_1NEGATE in scriptSig into 0x0181 in signing code (sipa)
dca28634d7 test: ensure OP_1NEGATE satisfies BIP62 minimal push rule (Jon Atack)
e629d07199 Do not turn OP_1NEGATE in scriptSig into 0x0181 in signing code (Pieter Wuille)

Pull request description:

  A rebase of #13084 which additionally modifies the test code (unaddressed in the original, assuming sipa is too busy to deal with this at the moment).

  Relatively simple bugfix so it'd be good to have merged soon.

  Turning OP_1NEGATE into 0x0181 results in a larger-than-necessary data push instead of just actually using the OP_1NEGATE opcode (0x4f). This fails the minimal push rule of BIP 62 and makes the result non-standard.

ACKs for top commit:
  fjahr:
    Code review ACK dca28634d7
  luke-jr:
    ACK dca28634d7
  jonatack:
    ACK dca28634d7

Tree-SHA512: 706d9a2ef20c809dea923e477a873e2fd60db8d0ae64289e510b766a38005c1f31ab0b5883f16b9c7863ff0d3f705e8e413f6121320028ac196b79c3184a4113
2020-08-14 11:53:47 +02:00
Wladimir J. van der Laan dabab06a1a
Merge #19455: rpc generate: print useful help and error message
f0aa8aeea5 test: add rpc_generate functional test (Jon Atack)
92d94ffb8d rpc: print useful help and error message for generate (Jon Atack)
8d32d2011d test: consider generate covered in _get_uncovered_rpc_commands() (Jon Atack)

Pull request description:

  This was a requested follow-up to #19133 and #17700 to alleviate confusion and head-scratching by people following tutorials that use `generate`. See https://github.com/bitcoin/bitcoin/pull/19455#issuecomment-668172916 below, https://github.com/bitcoin/bitcoin/pull/19133#issuecomment-636860943 and https://github.com/bitcoin/bitcoin/pull/17700#issuecomment-566159096.

  before
  ```
  $ bitcoin-cli help generate
  help: unknown command: generate

  $ bitcoin-cli generate
  error code: -32601
  error message:
  Method not found
  ```

  after
  ```
  $ bitcoin-cli help generate
  generate ( nblocks maxtries ) has been replaced by the -generate cli option. Refer to -help for more information.

  $ bitcoin-cli generate
  error code: -32601
  error message:
  generate ( nblocks maxtries ) has been replaced by the -generate cli option. Refer to -help for more information.
  ```

  In the general help it remains hidden, as requested by laanwj.
  ```
  $ bitcoin-cli help

  == Generating ==
  generateblock "output" ["rawtx/txid",...]
  generatetoaddress nblocks "address" ( maxtries )
  generatetodescriptor num_blocks "descriptor" ( maxtries )
  ```

ACKs for top commit:
  adamjonas:
    utACK f0aa8aeea5
  pinheadmz:
    ACK f0aa8aeea5

Tree-SHA512: d083652589ad3e8228c733455245001db22397559c3946e7e573cf9bd01c46e9e88b72d934728ec7f4361436ae4c74adb8f579670b09f479011924357e729af5
2020-08-14 11:29:05 +02:00
MarcoFalke 31760bb7c9
Merge #19528: rpc: Assert that RPCArg names are equal to CRPCCommand ones (misc)
fa77de2baa rpc: Assert that RPCArg names are equal to CRPCCommand ones (misc) (MarcoFalke)
fa50bdc755 rpc: Limit echo to 10 args (MarcoFalke)
fa89ca9b5b refactor: Use C++11 range based for loops to simplify rpc code (MarcoFalke)
fa459bdc87 rpc: Treat all args after a hidden arg as hidden as well (MarcoFalke)

Pull request description:

  This is split out from #18531 to just touch the RPC methods in misc. Description from the main pr:

  ### Motivation

  RPCArg names in the rpc help are currently only used for documentation. However, in the future they could be used to teach the server the named arguments. Named arguments are currently registered by the `CRPCCommand`s and duplicate the RPCArg names from the documentation. This redundancy is fragile, and has lead to errors in the past (despite having linters to catch those kind of errors). See section "bugs found" for a list of bugs that have been found as a result of the changes here.

  ### Changes

  The changes here add an assert in the `CRPCCommand` constructor that the RPCArg names are identical to the ones in the `CRPCCommand`.

  ### Future work

  > Here or follow up, makes sense to also assert type of returned UniValue?

  Sure, but let's not get ahead of ourselves. I am going to submit any further works as follow-ups, including:

  * Removing the CRPCCommand arguments, now that they are asserted to be equal and thus redundant
  * Removing all python regex linters on the args, now that RPCMan can be used to generate any output, including the cli.cpp table
  * Auto-formatting and sanity checking the RPCExamples with RPCMan
  * Checking passed-in json in self-check. Removing redundant checks
  * Checking returned json against documentation to avoid regressions or false documentation
  * Compile the RPC documentation at compile-time to ensure it doesn't change at runtime and is completely static

  ### Bugs found

  * The assert identified issue #18607
  * The changes itself fixed bug #19250

ACKs for top commit:
  laanwj:
    Code review ACK fa77de2baa
  fjahr:
    tested ACK fa77de2baa
  theStack:
    ACK fa77de2baa
  ryanofsky:
    Code review ACK fa77de2baa. Pretty straightfoward changes

Tree-SHA512: badae1606518c0b55ce2c0bb9025d14f05556532375eb20fd6f3bfadae1e5e6568860bff8599d037e655bf1d23f1f464ca17f4db10a6ab3d502b6e9e61c7b3d3
2020-08-14 09:26:37 +02:00
Russell Yanofsky 642ad31b41 Add loadwallet and createwallet RPC load_on_startup options
This maintains a persistent list of wallets stored in settings that will
automatically be loaded on startup. Being able to load a wallet automatically
on startup will be more useful in the GUI when the option to create wallets is
added in #15006, but it's reasonable to expose this feature by RPC as well.
2020-08-13 09:44:48 -04:00
Wladimir J. van der Laan b4d0366b47
Merge #19070: p2p: Signal support for compact block filters with NODE_COMPACT_FILTERS
f5c003d3ea [test] Add test for NODE_COMPACT_FILTER. (Jim Posen)
132b30d9c8 [net] Signal NODE_COMPACT_FILTERS if we're serving compact filters. (Jim Posen)
b3fbc94d4f Apply cfilters review fixups (John Newbery)

Pull request description:

  If -peerblockfilters is configured, signal the `NODE_COMPACT_FILTERS` service bit to indicate that we are able to serve compact block filters, headers and checkpoints.

ACKs for top commit:
  MarcoFalke:
    re-review and Concept ACK f5c003d3ea
  fjahr:
    Code review ACK f5c003d3ea
  clarkmoody:
    Concept ACK f5c003d3ea
  ariard:
    Concept and Code Review ACK f5c003d
  jonatack:
    ACK f5c003d3e

Tree-SHA512: 34d1c153530a0e55d09046fe548c9dc37344b5d6d50e00af1b4e1de1e7b49de770fca8471346a17c151de9fe164776296bb3dd5af331977f0c3ef1e6fc906f85
2020-08-13 15:44:48 +02:00
Wladimir J. van der Laan 6757b3ac8f
Merge #19655: rpc: Catch listsinceblock target_confirmations exceeding block count
c133cdcdc3 Cap listsinceblock target_confirmations param (Adam Stein)

Pull request description:

  This addresses an issue brought up in #19587.

  Currently, the `target_confirmations` parameter to `listsinceblock` is not checked for being too large. When `target_confirmations` is greater than one more than the current number of blocks, `listsinceblock` fails with error code -1. In comparison, when `target_confirmations` is less than 1,  a -8 "Invalid parameter" error code is thrown.

  This PR fixes the issue by returning a -8 "Invalid parameter" error if the `target_confirmations` value corresponds to a block with more confirmations than the genesis block. This happens if `target_confirmations` exceeds one more than the number of blocks.

ACKs for top commit:
  laanwj:
    Code review ACK c133cdcdc3
  ryanofsky:
    Code review ACK c133cdcdc3. Just suggested changes since last review. Thanks!

Tree-SHA512: 02680f4cb937d2c24d5019abd0ebfa188b8a50679a1e64e9c26bfe5c17eef6aea906832e6e2d492ba8a2ea160041bf185d66795ee691e340f6793db03c21b89a
2020-08-13 12:12:33 +02:00
Samuel Dobson 8a85377cd0
Merge #18654: rpc: separate bumpfee's psbt creation function into psbtbumpfee
79d6332e9e moveonly: Fix indentation in bumpfee RPC (Andrew Chow)
431071c28a Hide bumpfee's psbt creation behavior behind -deprecatedrpc (Andrew Chow)
4638224f64 Add psbtbumpfee RPC (Andrew Chow)

Pull request description:

  Adds a new RPC `psbtbumpfee` which always creates a psbt. `bumpfee` will then only be able to create and broadcast fee bumping transactions instead of changing its behavior based on `IsWalletSet(WALLET_FLAG_DISABLE_PRIVATE_KEYS)`.

  Split from #18627

ACKs for top commit:
  Sjors:
    re-utACK 79d6332
  meshcollider:
    utACK 79d6332e9e
  fjahr:
    Code review ACK 79d6332e9e

Tree-SHA512: 1c92c4b4461bb30e78be3ee73165f624398ef33996ce36043b61a8931be667030d0fca12fd0b30097b78c56e4e9092c69582b237cbdac51d56f6be23d8c0f1bb
2020-08-13 12:21:11 +12:00
Adam Stein c133cdcdc3
Cap listsinceblock target_confirmations param
Previously, listsinceblock would fail with error code -1 when the
target_confirmations exceeded the number of confirmations of the genesis
block. This commit allows target_confirmations to refer to a lastblock
hash with more confirmations than exist in the chain by setting the
lastblock hash to the genesis hash in this case. This allows for
`listsinceblock "" 6` to not fail if the block count is less than 5
which may happen on regtest.

Includes update to the functional test for listsinceblock to test for
this case.
2020-08-12 15:16:22 -07:00
Sebastian Falbesoner 9e7894357e test: speedup p2p_feefilter.py by whitelisting peers (immediate tx relay)
Most of the test time is spent in wait_for_invs() after sending to addresses,
i.e. the bottleneck is in relaying transactions. By whitelisting the peers via
-whitelist, the inventory is transmissioned immediately rather than on average
every 5 seconds, speeding up the test significantly:

before:
$ time ./p2p_feefilter.py
...
real    0m39.367s
user    0m1.227s
sys 0m0.571s

with this commit:
$ time ./p2p_feefilter.py
...
real    0m9.386s
user    0m1.120s
sys 0m0.577s
2020-08-12 22:38:09 +02:00
Sebastian Falbesoner fe3f0cc44e test: use wait_until for invs matching in p2p_feefilter.py
additionally:
    -> rename function with snake_case (s/allInvsMatch/wait_for_invs_to_match)
    -> move it from global namespace to the class FeefilterConn
2020-08-12 22:30:38 +02:00
Sebastian Falbesoner 6d941923c3 test: add logging for p2p_feefilter.py 2020-08-12 22:21:38 +02:00
Wladimir J. van der Laan 13c4635a3e
Merge #19696: rpc: Fix addnode remove command error
a51d0ad2de rpc: Improve addnode remove command error message (Fabian Jahr)

Pull request description:

  The `addnode` RPC with the `remove` command parameter is used to remove a node from the "added nodes". It did not have test coverage and in case of failure to remove the node it responded with the confusing message "Error: Node has not been added.".

  This PR adds test coverage and introduces a new error code as well as changes the error message to something that makes sense.

ACKs for top commit:
  laanwj:
    Code review ACK a51d0ad2de
  theStack:
    Tested ACK https://github.com/bitcoin/bitcoin/commit/a51d0ad2de

Tree-SHA512: 033ef5de0d4d49d58ef4df3759b838c9d19ee9dfb0aff9f814a3a63d124ca231a442c930efa7d343fe1f65727c4b59fc23dd5e26fe6ea69f9e84fda48b5c5cc2
2020-08-12 19:29:14 +02:00
Wladimir J. van der Laan bd00d3b1f2
Merge #19658: [rpc] Allow RPC to fetch all addrman records and add records to addrman
37a480e0cd [net] Add addpeeraddress RPC method (John Newbery)
ae8051bbd8 [test] Test that getnodeaddresses() can return all known addresses (John Newbery)
f26502e9fc [addrman] Specify max addresses and pct when calling GetAddresses() (John Newbery)

Pull request description:

  Currently addrman only allows a maximum of 1000 records or 23% of all records to be returned in a call to `GetAddr()`. Relax this limit and have the client specify the max records they want. For p2p, behaviour is unchanged (but the rate limiting is set inside net_processing, where it belongs). For RPC, `getnodeaddresses` can now return the complete addrman, which is helpful for testing and monitoring.

  Also add a test-only RPC `addpeeraddress`, which adds an IP address:port to addrman. This is helpful for testing (eg #18991).

ACKs for top commit:
  naumenkogs:
    utACK 37a480e0cd
  laanwj:
    Code review and lightly manually tested ACK 37a480e0cd

Tree-SHA512: f86dcd410aaebaf6e9ca18ce6f23556e5e4649c1325577213d873aa09967298e65ab2dc19a72670641ae92211a923afda1fe124a82e9d2c1cad73d478ef27fdc
2020-08-12 15:23:06 +02:00
John Newbery 37a480e0cd [net] Add addpeeraddress RPC method
Allows addresses to be added to Address Manager for testing.
2020-08-12 09:22:10 +01:00
John Newbery ae8051bbd8 [test] Test that getnodeaddresses() can return all known addresses 2020-08-12 09:22:10 +01:00
Fabian Jahr a51d0ad2de
rpc: Improve addnode remove command error message
This also adds test coverage for the remove command which was uncovered before.
2020-08-11 14:04:02 +02:00
fanquake cb1ee1551c
Merge #19674: refactor: test: use throwaway _ variable for unused loop counters
dac7a111bd refactor: test: use _ variable for unused loop counters (Sebastian Falbesoner)

Pull request description:

  This tiny PR substitutes Python loops in the form of `for x in range(N): ...` by `for _ in range(N): ...` where applicable. The idea is indicating to the reader that a block (or statement, in list comprehensions) is just repeated N times, and that the loop counter is not used in the body, hence using the throwaway variable. This is already done quite often in the current tests (see e.g. `$ git grep "for _ in range("`). Another alternative would be using `itertools.repeat` (according to Python core developer Raymond Hettinger it's [even faster](https://twitter.com/raymondh/status/1144527183341375488)), but that doesn't seem to be widespread in use and I'm not sure about a readability increase.

  The only drawback I see is that whenever one wants to debug loop iterations, one would need to introduce a loop variable again. Reviewing this is basically a no-brainer, since tests would fail immediately if a a substitution has taken place on a loop where the variable is used.

  Instances to replace were found by `$ git grep "for.*in range("` and manually checked.

ACKs for top commit:
  darosior:
    ACK dac7a111bd
  instagibbs:
    manual inspection ACK dac7a111bd
  practicalswift:
    ACK dac7a111bd -- the updated code is easier to reason about since the throwaway nature of a variable is expressed explicitly (using the Pythonic `_` idiom) instead of implicitly. Explicit is better than implicit was we all know by now :)

Tree-SHA512: 5f43ded9ce14e5e00b3876ec445b90acda1842f813149ae7bafa93f3ac3d510bb778e2c701187fd2c73585e6b87797bb2d2987139bd1a9ba7d58775a59392406
2020-08-11 09:24:50 +08:00
Wladimir J. van der Laan be11f94e95
Merge #19631: test: Wait for 'cmpctblock' in p2p_compactblocks when it is expected
9e165d0de4 test: Wait for 'cmpctblock' in p2p_compactblocks when it is expected (Ben Woosley)

Pull request description:

  This is a more narrowly-construed wait which eliminates the possibility of the
  wait being triggered by other messages.

  Note `received_block_announcement` reflect three possible messages:
  edec7f7c25/test/functional/p2p_compactblocks.py (L34-L53)

  Prompted by looking into: #19449

ACKs for top commit:
  laanwj:
    Code review ACK 9e165d0de4
  theStack:
    ACK 9e165d0de4

Tree-SHA512: bc4a9c8bf031c8a7efb40d9625feaa3fd1f56f3b75da7034944af71ccea44328a6c708ab0c13fea85fb7cf4fd9043fe90eb94a25e95b2d42be44c2962b4904ce
2020-08-09 18:54:42 +02:00
Wladimir J. van der Laan 5e4fed9e58
Merge #19657: test: Wait until is_connected in add_p2p_connection
fa4dfd215f test: Wait until is_connected in add_p2p_connection (MarcoFalke)

Pull request description:

  Moving the wait_until from the individual test scripts to the test framework simplifies two tests

ACKs for top commit:
  jnewbery:
    Code review ACK fa4dfd215f
  theStack:
    ACK fa4dfd215f 

Tree-SHA512: 36eda7eb323614a4c4f9215f1d7b40b9f9c4036d1c08eb701ea705f3e2986fdabd2fc558965a6aadabeed861034aeaeef3c00f968ca17ed7a27e42e506cda87d
2020-08-09 18:35:31 +02:00
Wladimir J. van der Laan 214e6655c3
Merge #19649: Restore test case for p2p transaction blinding
566aada386 Test that wtxid relay peers add wtxid to reject filter (Gregory Sanders)
0fea6ede1b Restore test case for p2p transaction blinding (Gregory Sanders)

Pull request description:

  Introduced in ca10a03add then erroneously removed in 8d8099e97a. The restored line is how we are
  checking that the node will still re-request a specific txid given a witness-related failure.

ACKs for top commit:
  fjahr:
    tACK 566aada386

Tree-SHA512: be2b75b5eddb88019b79cc798f9922ca7347ccbb2210b8d4eae93fdde62e2cbb614b5247cb2fbd7ee3577dbe053875a9b62c5747aace8617f12790b8fccdeab4
2020-08-09 18:30:25 +02:00
Sjors Provoost 6d1f51343c
[rpc] fundrawtransaction, walletcreatefundedpsbt lock manually selected coins
Previously only automatically selected coins were locked when lockUnspents is set.
It now also locks selected coins.
2020-08-07 14:13:15 +02:00
fanquake 6d8543504d
Merge #19620: Add txids with non-standard inputs to reject filter
9f88ded82b test addition of unknown segwit spends to txid reject filter (Gregory Sanders)
7989901c7e Add txids with non-standard inputs to reject filter (Suhas Daftuar)

Pull request description:

  Our policy checks for non-standard inputs depend only on the non-witness
  portion of a transaction: we look up the scriptPubKey of the input being
  spent from our UTXO set (which is covered by the input txid), and the p2sh
  checks only rely on the scriptSig portion of the input.

  Consequently it's safe to add txids of transactions that fail these checks to
  the reject filter, as the witness is irrelevant to the failure. This is helpful
  for any situation where we might request the transaction again via txid (either
  from txid-relay peers, or if we might fetch the transaction via txid due to
  parent-fetching of orphans).

  Further, in preparation for future witness versions being deployed on the
  network, ensure that WITNESS_UNKNOWN transactions are rejected in
  AreInputsStandard(), so that transactions spending v1 (or greater) witness
  outputs will fall into this category of having their txid added to the reject
  filter.

ACKs for top commit:
  ajtowns:
    ACK 9f88ded82b - code review
  jnewbery:
    Code review ACK 9f88ded82b
  ariard:
    Code Review/Tested ACK 9f88ded
  naumenkogs:
    utACK 9f88ded82b
  jonatack:
    ACK 9f88ded82b

Tree-SHA512: 1e93c0a5b68cb432524780ffc0093db893911fdfed9e2ed17f888e59114cc75d2a07062aefad4e5ce2e87c9270886117a8abb3c78fb889c9b9f31967f1777148
2020-08-07 07:34:27 +08:00
Sebastian Falbesoner dac7a111bd refactor: test: use _ variable for unused loop counters
substitutes "for x in range(N):" by "for _ in range(N):"
indicates to the reader that a block is just repeated N times, and
that the loop counter is not used in the body
2020-08-06 18:39:33 +02:00
Karl-Johan Alm 7f13dfb587
test: test the implicit avoid partial spends functionality
Co-authored-by: Fabian Jahr <fjahr@protonmail.com>
2020-08-06 10:25:08 +09:00
Wladimir J. van der Laan 4644b13b44
Merge #19632: test: Catch decimal.InvalidOperation from TestNodeCLI#send_cli
82fc4017b7 test: Catch decimal.InvalidOperation from TestNodeCLI#send_cli (Ben Woosley)

Pull request description:

  `decimal.InvalidOperation` is a special case of a float parsing error, which
  presumably should be handled in the same way as a general parsing error,
  rather than blow up.

  Alternatives include: logging the error, or re-raising with more information.

  Example log output:
  ```
      File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 603, in sync_all
        self.sync_blocks(nodes)
      File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in sync_blocks
        best_hash = [x.getbestblockhash() for x in rpc_connections]
      File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in <listcomp>
        best_hash = [x.getbestblockhash() for x in rpc_connections]
      File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 571, in __call__
        return self.cli.send_cli(self.command, *args, **kwargs)
      File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 639, in send_cli
        return json.loads(cli_stdout, parse_float=decimal.Decimal)
      File "/usr/lib64/python3.6/json/__init__.py", line 367, in loads
        return cls(**kw).decode(s)
      File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib64/python3.6/json/decoder.py", line 355, in raw_decode
        obj, end = self.scan_once(s, idx)
    decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
  ```
  See: https://travis-ci.org/github/bitcoin/bitcoin/jobs/713502326

ACKs for top commit:
  laanwj:
    ACK 82fc4017b7

Tree-SHA512: 8c102b8bf831b05c5ca4b2e1feb5574dcbaed8cab0b2f22b013c5dfcb81788a38839a163dd1e2c6470ccbe5874214663b84485f45467738fd850ca38d539ae25
2020-08-05 16:27:54 +02:00
Gregory Sanders 9f88ded82b test addition of unknown segwit spends to txid reject filter 2020-08-04 13:29:40 -04:00
MarcoFalke fa4dfd215f
test: Wait until is_connected in add_p2p_connection 2020-08-04 16:13:33 +02:00
MarcoFalke 3c93623be2
Merge #19489: test: Fail wait_until early if connection is lost
faa9a74c9e test: Fail wait_until early if connection is lost (MarcoFalke)

Pull request description:

  Calling `minonode.wait_until` needs a connection to make progress (e.g. waiting for an inv), unless the mininode waits for the initial connection or for a disconnection. So for test development and failure debugging, fail early in all `wait_until`, unless opted out.

ACKs for top commit:
  jnewbery:
    Code review ACK faa9a74c9e.

Tree-SHA512: 4be850b96e23b87bc2ff42c028a5045d6f5cdbc9482ce6a6ba01cc5eb26710dab9e2ed547c363aac4bd5825151ee9996fb797261420b631bceeddbfa698d1dec
2020-08-04 11:21:13 +02:00
Wladimir J. van der Laan 14ceddd290
Merge #18991: Cache responses to GETADDR to prevent topology leaks
3bd67ba5a4 Test addr response caching (Gleb Naumenko)
cf1569e074 Add addr permission flag enabling non-cached addr sharing (Gleb Naumenko)
acd6135b43 Cache responses to addr requests (Gleb Naumenko)
7cc0e8101f Remove useless 2500 limit on AddrMan queries (Gleb Naumenko)
ded742bc5b Move filtering banned addrs inside GetAddresses() (Gleb Naumenko)

Pull request description:

  This is a very simple code change with a big p2p privacy benefit.

  It’s currently trivial to scrape any reachable node’s AddrMan (a database of all nodes known to them along with the timestamps).
  We do have a limit of one GETADDR per connection, but a spy can disconnect and reconnect even from the same IP, and send GETADDR again and again.

  Since we respond with 1,000 random records at most, depending on the AddrMan size it takes probably up to 100 requests for an spy to make sure they scraped (almost) everything.
  I even have a script for that. It is totally doable within couple minutes.

  Then, with some extra protocol knowledge a spy can infer the direct peers of the victim, and other topological stuff.

  I suggest to cache responses to GETADDR on a daily basis, so that an attacker gets at most 1,000 records per day, and can’t track the changes in real time. I will be following up with more improvements to addr relay privacy, but this one alone is a very effective. And simple!

  I doubt any of the real software does *reconnect to get new addrs from a given peer*, so we shouldn’t be cutting anyone.
  I also believe it doesn’t have any negative implications on the overall topology quality. And the records being “outdated” for at most a day doesn’t break any honest assumptions either.

ACKs for top commit:
  jnewbery:
    reACK 3bd67ba5a4
  promag:
    Code review ACK 3bd67ba5a4.
  ariard:
    Code Review ACK 3bd67ba

Tree-SHA512: dfa5d03205c2424e40a3f8a41af9306227e1ca18beead3b3dda44aa2a082175bb1c6d929dbc7ea8e48e01aed0d50f0d54491caa1147471a2b72a46c3ca06b66f
2020-08-03 14:48:52 +02:00
Gregory Sanders 566aada386 Test that wtxid relay peers add wtxid to reject filter
h/t Anthony Towns
2020-08-02 23:34:11 -04:00
Gregory Sanders 0fea6ede1b Restore test case for p2p transaction blinding
Introduced in ca10a03add then erroneously removed in
8d8099e97a. The restored line is how we are
checking that the node will still re-request a specific txid given a witness-related failure.
2020-08-02 21:07:44 -04:00
MarcoFalke fa50bdc755
rpc: Limit echo to 10 args 2020-08-02 21:32:40 +02:00
Ben Woosley 82fc4017b7
test: Catch decimal.InvalidOperation from TestNodeCLI#send_cli
decimal.InvalidOperation is a special case of a float parsing error, which
presumably should be handled in the same way as a general parsing error,
rather than blow up.

Alternatives include: logging the error, or re-raising with more information.

Example log output:
    File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 603, in sync_all
      self.sync_blocks(nodes)
    File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in sync_blocks
      best_hash = [x.getbestblockhash() for x in rpc_connections]
    File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in <listcomp>
      best_hash = [x.getbestblockhash() for x in rpc_connections]
    File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 571, in __call__
      return self.cli.send_cli(self.command, *args, **kwargs)
    File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 639, in send_cli
      return json.loads(cli_stdout, parse_float=decimal.Decimal)
    File "/usr/lib64/python3.6/json/__init__.py", line 367, in loads
      return cls(**kw).decode(s)
    File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib64/python3.6/json/decoder.py", line 355, in raw_decode
      obj, end = self.scan_once(s, idx)
  decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
2020-07-30 18:45:53 -07:00
Ben Woosley 9e165d0de4
test: Wait for 'cmpctblock' in p2p_compactblocks when it is expected
This is a more narrowly-construed wait which eliminates the possibility of the
wait being triggered by other messages.

Co-authored-by: Billy Garrison <billygarrison.btc@gmail.com>
2020-07-30 16:13:48 -07:00
Anthony Towns e65d115b72 test: request parents of orphan from wtxid relay peer 2020-07-30 13:45:02 -07:00
Gleb Naumenko 3bd67ba5a4 Test addr response caching 2020-07-30 14:38:50 +03:00
Gleb Naumenko cf1569e074 Add addr permission flag enabling non-cached addr sharing 2020-07-30 14:38:50 +03:00
MarcoFalke 2a784723f0
Merge #19597: test: test decodepsbt fee calculation (count input value only once per UTXO)
82dee87933 test: test decodepsbt fee calculation (count input value only once per UTXO) (Sebastian Falbesoner)

Pull request description:

  Fixes #19523, adding a simple test to `rpc_psbt.py` that checks that the decodepsbt fee matches the one given by the wallet (`walletcreatefundedpsbt`). This is in particular important for PSBTs with segwit inputs that have both a witness- and a non-witness-UTXO type set.

  Example test run after reverting commit 75122780e2 ("Increment input value sum only once per UTXO in decodepsbt"):

  ```
  $ test/functional/rpc_psbt.py
  2020-07-26T11:31:44.862000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test__sutcd4y
  20.00007580
  2020-07-26T11:31:47.073000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "/home/honeybadger/buidl/bitcoin_thestack/test/functional/test_framework/test_framework.py", line 118, in main
      self.run_test()
    File "test/functional/rpc_psbt.py", line 166, in run_test
      assert_equal(decoded['fee'], created_psbt['fee'])
    File "/home/honeybadger/buidl/bitcoin_thestack/test/functional/test_framework/util.py", line 49, in assert_equal
      raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
  AssertionError: not(20.00007580 == 0.00007580)
  2020-07-26T11:31:47.125000Z TestFramework (INFO): Stopping nodes
  ......
  ```

ACKs for top commit:
  achow101:
    ACK 82dee87933

Tree-SHA512: 296b8a701f851d482ef6200c6cbf0cf0257a79a828ac6dbc39b05d8c2d839c6fdb9d3f5a084015295cfa3eac7c11faa2f2d52e619c11627b04c75150eead8330
2020-07-30 09:46:16 +02:00
MarcoFalke 149eca433d
Merge #19599: test: clean message_count and last_message
2c6a02e024 Clean message_count and last_message (Troy Giorshev)

Pull request description:

  From #19580

  This PR changes comments to clarify the intended usage of `message_count` and `last_message`.  Additionally it changes the only usage of `message_count` to use `last_message` instead, bringing the code into alignment with the intended usage.

  Note: Now `message_count` is completely unused.  However, it is ready to be used (i.e. the supporting code works) and likely will be used in some test in the future.

ACKs for top commit:
  jnewbery:
    utACK 2c6a02e024

Tree-SHA512: 07c7684c9586de4f845e10d7aac36c1aab9fb56b409949c1c70d5ca705bc3971ca7d5943245a0472def4efd7b4e1c5dad2f713db5ead8fca08404daf4891e98b
2020-07-30 09:15:49 +02:00
Wladimir J. van der Laan 8db23349fe
Merge #19335: wallet: Cleanup and separate BerkeleyDatabase and BerkeleyBatch
74507ce71e walletdb: Remove BerkeleyBatch friend class from BerkeleyDatabase (Andrew Chow)
00f0041351 No need to check for duplicate fileids in all dbenvs (Andrew Chow)
d86efab370 walletdb: Move Db->open to BerkeleyDatabase::Open (Andrew Chow)
4fe4b3bf1b walletdb: track database file use as m_refcount within BerkeleyDatabase (Andrew Chow)
65fb8807ac Combine BerkeleyEnvironment::Verify into BerkeleyDatabase::Verify (Andrew Chow)

Pull request description:

  `BerkeleyBatch` and `BerkeleyDatabase` are kind of messy. The goal of this is to clean up them up so that they are logically separated.

  `BerkeleyBatch` currently handles the creation of the `BerkeleyDatabase`'s `Db` handle. This is instead moved into `BerkeleyDatabase` and is called by `BerkeleyBatch`.

  Instead of having `BerkeleyEnvironment` track each database's usage, have `BerkeleyDatabase` track this usage itself with the `m_refcount` variable that is present in `WalletDatabase`.

  Lastly, instead of having each `BerkeleyEnvironment` store the fileids of the databases open in it, have a global `g_fileids` to track those fileids. We were already checking fileid uniqueness globally (by checking the fileids in every environment when opening a database) so it's cleaner to do this with a global variable.

  All of these changes allow us to make `BerkeleyBatch` and `BerkeleyDatabase` no longer be friend classes.

  The diff of this PR is currently the same as in ##18971

  Requires #19334

ACKs for top commit:
  laanwj:
    Code review ACK 74507ce71e
  ryanofsky:
    Code review ACK 74507ce71e. No changes since last review other than rebase

Tree-SHA512: 845d84ee1a470e2bf5d2e2e3d7738183d8ce43ddd06a0bbd57edecf5779b2f55d70728b1b57f5daab0f078650a8d60c3e19dc30b75b36e7aa952ce268399d5f6
2020-07-29 18:24:16 +02:00
Troy Giorshev 2c6a02e024 Clean message_count and last_message
This commit clarifies the intended usage of message_count and
last_message.  Additionally it changes the only usage of message_count
to using last_message instead, bringing the code further along the
intended usage.
2020-07-27 07:55:49 -04:00
Sebastian Falbesoner 82dee87933 test: test decodepsbt fee calculation (count input value only once per UTXO)
Checks that the RPC decodepsbt calculates the fee correctly, in particular for
PSBTs with segwit inputs that have both a witness- and a non-witness-UTXO type
set. Before commit 75122780e2 ("Increment input
value sum only once per UTXO in decodepsbt") the values for those inputs were
double counted.
2020-07-26 13:25:16 +02:00
MarcoFalke f4cfa6d019
Merge #15935: Add <datadir>/settings.json persistent settings storage
9c69cfe4c5 Add <datadir>/settings.json persistent settings storage. (Russell Yanofsky)
eb682c5700 util: Add ReadSettings and WriteSettings functions (Russell Yanofsky)

Pull request description:

  Persistent settings are used in followup PRs #15936 to unify gui settings between bitcoin-qt and bitcoind, and #15937 to add a load_on_startup flag to the loadwallet RPC and maintain a dynamic list of wallets that should be loaded on startup that also can be shared between bitcoind and bitcoin-qt.

ACKs for top commit:
  MarcoFalke:
    Approach re-ACK 9c69cfe4c5 🌾
  jnewbery:
    utACK 9c69cfe4c5

Tree-SHA512: 39fcc6051717117c9141e934de1d0d3f739484be4685cdf97d54de967c8c816502b4fd0de12114433beaa5c5b7060c810fd8ae4e2b3ce7c371eb729ac01ba2e1
2020-07-23 18:39:42 +02:00
MarcoFalke 6ee36a263c
Merge #19473: net: Add -networkactive option
2aac093a3d test: Add test coverage for -networkactive option (Hennadii Stepanov)
3c58129b12 net: Log network activity status change unconditionally (Hennadii Stepanov)
62fe6aa87e net: Add -networkactive option (Hennadii Stepanov)

Pull request description:

  Some Bitcoin Core activity is completely local (offline), e.g., reindexing.

  The `setnetworkactive` RPC command is already present. This PR adds the corresponding command-line argument / config option, and allows to start the client with disabled p2p network by providing `-networkactive=0` or `-nonetworkactive`.

  This was done while reviewing #16981.

ACKs for top commit:
  MarcoFalke:
    re-ACK 2aac093a3d 🏠
  LarryRuane:
    ACK 2aac093a3d

Tree-SHA512: 446d791b46d7b556d7694df7b1f88cd4fbc09301fe4eaf036b45cb8166ed806156353cc03788a07b633d5887d5eee30a7c02a2d4307141c8ccc75e0a88145636
2020-07-23 18:32:59 +02:00
Andrew Chow 00f0041351 No need to check for duplicate fileids in all dbenvs
Since we have .walletlock in each directory, we don't need the duplicate
fileid checks across all dbenvs as it shouldn't be possible anyways.
2020-07-22 23:30:19 -04:00
Hennadii Stepanov 2aac093a3d
test: Add test coverage for -networkactive option 2020-07-22 22:55:48 +03:00
Wladimir J. van der Laan ccef10261e
Merge #18044: Use wtxid for transaction relay
0a4f1422cd Further improve comments around recentRejects (Suhas Daftuar)
0e20cfedb7 Disconnect peers sending wtxidrelay message after VERACK (Suhas Daftuar)
cacd85209e test: Use wtxid relay generally in functional tests (Fabian Jahr)
8d8099e97a test: Add tests for wtxid tx relay in segwit test (Fabian Jahr)
9a5392fdf6 test: Update test framework p2p protocol version to 70016 (Fabian Jahr)
dd78d1d641 Rename AddInventoryKnown() to AddKnownTx() (Suhas Daftuar)
4eb515574e Make TX_WITNESS_STRIPPED its own rejection reason (Suhas Daftuar)
97141ca442 Delay getdata requests from peers using txid-based relay (Suhas Daftuar)
46d78d47de Add p2p message "wtxidrelay" (Suhas Daftuar)
2d282e0cba ignore non-wtxidrelay compliant invs (Anthony Towns)
ac88e2eb61 Add support for tx-relay via wtxid (Suhas Daftuar)
8e68fc246d Add wtxids to recentRejects instead of txids (Suhas Daftuar)
144c385820 Add wtxids of confirmed transactions to bloom filter (Suhas Daftuar)
85c78d54af Add wtxid-index to orphan map (Suhas Daftuar)
08b39955ec Add a wtxid-index to mapRelay (Suhas Daftuar)
60f0acda71 Just pass a hash to AddInventoryKnown (Suhas Daftuar)
c7eb6b4f1f Add wtxid to mempool unbroadcast tracking (Amiti Uttarwar)
2b4b90aa8f Add a wtxid-index to the mempool (Suhas Daftuar)

Pull request description:

  Using txids (a transaction's hash, without witness) for transaction relay is problematic, post-segwit -- if a peer gives us a segwit transaction that fails policy checks, it could be because the txid associated with the transaction is definitely unacceptable to our node (regardless of the witness), or it could be that the transaction was malleated and with a different witness, the txid could be accepted to our mempool.

  We have a bloom filter of recently rejected transactions, whose purpose is to help us avoid redownloading and revalidating transactions that fail to be accepted, but because of this potential for witness malleability to interfere with relay of valid transactions, we do not use the filter for segwit transactions.  This issue is discussed at some length in #8279.  The effect of this is that whenever a segwit transaction that fails policy checks is relayed, a node would download that transaction from every peer announcing it, because it has no way presently to cache failure.  Historically this hasn't been a big problem, but if/when policy for accepting segwit transactions were to change (eg taproot, or any other change), we could expect older nodes talking to newer nodes to be wasting bandwidth because of this.

  As discussed in that issue, switching to wtxid-based relay solves this problem -- by using an identifier for a transaction that commits to all the data in our relay protocol, we can be certain if a transaction that a peer is announcing is one that we've already tried to process, or if it's something new.  This PR introduces support for wtxid-based relay with peers that support it (and remains backwards compatible with peers that use txids for relay, of course).

  Apart from code correctness, one issue to be aware of is that by downloading from old and new peers alike, we should expect there to be some bandwidth wasted, because sometimes we might download the same transaction via txid-relay as well as wtxid-relay.  The last commit in this PR implements a heuristic I want to analyze, which is to just delay relay from txid-relay peers by 2 seconds, if we have at least 1 wtxid-based peer.  I've just started running a couple nodes with this heuristic so I can measure how well it works, but I'm open to other ideas for minimizing that issue.  In the long run, I think this will be essentially a non-issue, so I don't think it's too big a concern, we just need to bite the bullet and deal with it during upgrade.

  Finally, this proposal would need a simple BIP describing the changes, which I haven't yet drafted.  However, review and testing of this code in the interim would be welcome.

  To do items:
  - [x] Write BIP explaining the spec here (1 new p2p message for negotiating wtxid-based relay, along with a new INV type)
  - [ ] Measure and evaluate a heuristic for minimizing how often a node downloads the same transaction twice, when connected to old and new nodes.

ACKs for top commit:
  naumenkogs:
    utACK 0a4f1422cd
  laanwj:
    utACK 0a4f1422cd

Tree-SHA512: d8eb8f0688cf0cbe9507bf738e143edab1f595551fdfeddc2b6734686ea26e7f156b6bfde38bad8bbbe8bec1857c7223e1687f8f018de7463dde8ecaa8f450df
2020-07-22 20:58:55 +02:00
MarcoFalke edfeaf6836
Merge #19552: test: fix intermittent failure in p2p_ibd_txrelay
12410b1feb test: fix intermittent p2p_ibd_txrelay race, add test_framework.py#wait_until (Jon Atack)

Pull request description:

  To fix these intermittent failures in Travis CI.
  ```
  162/163 - p2p_ibd_txrelay.py failed, Duration: 2 s

  stdout:
  2020-07-19T05:44:17.213000Z TestFramework (INFO):
      Check that nodes set minfilter to MAX_MONEY while still in IBD
  2020-07-19T05:44:17.216000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
    File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/test_framework/test_framework.py", line 117, in main
      self.run_test()
    File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/p2p_ibd_txrelay.py", line 30, in run_test
      assert_equal(conn_info['minfeefilter'], MAX_FEE_FILTER)
    File "/Users/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-x86_64-apple-darwin16/test/functional/test_framework/util.py", line 49, in assert_equal
      raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))

  AssertionError: not(0E-8 == 0.09170997)
  2020-07-19T05:44:17.293000Z TestFramework (INFO): Stopping nodes
  ```

  At Marco's suggestion, cherry-picked part of #19134 to nicely simplify using `wait_until`.

ACKs for top commit:
  vasild:
    ACK 12410b1fe

Tree-SHA512: 615f509883682fd693e578b259cba35a9fa0bc519f1394e88c857e8b0650bfec5397bfa856cfa9e6d5ef81d0ee6ad02e4ad2b0eb0bd530b4c281cbe3e663790b
2020-07-21 16:01:59 +02:00
MarcoFalke ea595d39f7
Merge #19205: script: previous_release.sh rewritten in python
9c34aff393 Remove previous_release.sh (Brian Liotti)
e1e5960e10 script: Add previous_release.py (Brian Liotti)

Pull request description:

  Closes #18132

  Added functionality:
  1) checks file hash before untarring when using the binary download option

ACKs for top commit:
  fjahr:
    re-ACK 9c34aff393
  Sjors:
    tACK 9c34aff393

Tree-SHA512: 323f11828736a372a47f048592de8b027ddcd75b38f312dfc73f7b495d1e078bfeb384d9cdf434b3e70f2c6c0ce2da2df48e9a6460ac0e1967c6829a411c52d5
2020-07-21 10:11:39 +02:00
Jon Atack 12410b1feb
test: fix intermittent p2p_ibd_txrelay race, add test_framework.py#wait_until 2020-07-19 13:37:54 +02:00
Fabian Jahr cacd85209e test: Use wtxid relay generally in functional tests 2020-07-19 02:10:42 -04:00
Fabian Jahr 8d8099e97a test: Add tests for wtxid tx relay in segwit test
Also cleans up some doublicate lines in the rest of the test.

co-authored-by: Anthony Towns <aj@erisian.com.au>
2020-07-19 02:10:42 -04:00
Fabian Jahr 9a5392fdf6 test: Update test framework p2p protocol version to 70016
This new p2p protocol version allows to use WTXIDs for tx relay.
2020-07-19 02:10:42 -04:00
Suhas Daftuar 97141ca442 Delay getdata requests from peers using txid-based relay
Using both txid and wtxid-based relay with peers means that we could sometimes
download the same transaction twice, if announced via two different hashes from
different peers.

Use a heuristic of delaying txid-peer-getdata requests by 2 seconds, if we have
at least one wtxid-based peer.
2020-07-19 02:10:42 -04:00
Suhas Daftuar ac88e2eb61 Add support for tx-relay via wtxid
This adds a field to CNodeState that tracks whether to relay transactions with
that peer via wtxid, instead of txid. As of this commit the field will always
be false, but in a later commit we will add a way to negotiate turning this on
via p2p messages exchanged with the peer.
2020-07-19 02:05:29 -04:00
MarcoFalke 19aaf7945e
Merge #19423: test: add functional test for txrelay during and after IBD
cb31ee01b4 [test] feefilter during and after IBD (gzhao408)

Pull request description:

  This is a followup to #19204 which uses `minfeefilter=MAX_MONEY` to effectively shut off txrelay, thereby reducing inv traffic, when nodes are in IBD. It was [missing](https://github.com/bitcoin/bitcoin/pull/19204#issuecomment-644040070) a functional test.

ACKs for top commit:
  jnewbery:
    utACK cb31ee01b4

Tree-SHA512: a9effc8193fa95fb42a2f9c66b258cc7b0941fc04c1ce3a6092f4426c9bfc7e72f702aca559b3e30e90652497f411f22fae3cf5cdb6cfd6ef6d37fed712cda67
2020-07-17 07:51:24 +02:00
Wladimir J. van der Laan c57dc566b0
Merge #16525: Dump transaction version as an unsigned integer in RPC/TxToUniv
e80259f197 Additionally treat Tx.nVersion as unsigned in joinpsbts (Matt Corallo)
970de70bdd Dump transaction version as an unsigned integer in RPC/TxToUniv (Matt Corallo)

Pull request description:

  Consensus-wise we already treat it as an unsigned integer (the
  only rules around it are in CSV/locktime handling), but changing
  the underlying data type means touching consensus code for a
  simple cleanup change, which isn't really worth it.

  See-also, https://github.com/rust-bitcoin/rust-bitcoin/pull/299

ACKs for top commit:
  sipa:
    ACK e80259f197
  practicalswift:
    ACK e80259f197
  ajtowns:
    ACK e80259f197 code review -- checked all other uses of tx.nVersion treat it as unsigned (except for policy.cpp:IsStandard anyway), so looks good.
  naumenkogs:
    ACK e80259f

Tree-SHA512: 6760a2c77e24e9e1f79a336ca925f9bbca3a827ce02003c71d7f214b82ed3dea13fa7d9f87df9b9445cd58dff8b44a15571d821c876f22f8e5a372a014c9976b
2020-07-16 21:38:09 +02:00
Fabian Jahr 36ec9801a4
test: Add chacha20 test vectors in muhash 2020-07-16 18:10:48 +02:00
Fabian Jahr 0e2b400fea
test: Add basic Python/C++ Muhash implementation parity unit test 2020-07-16 18:10:48 +02:00