Commit graph

24154 commits

Author SHA1 Message Date
practicalswift 2bcc2bd742 tests: Clarify how we avoid hitting the signed integer overflow in CFeeRate::GetFeePerK() when fuzzing 2020-04-30 14:19:49 +00:00
MarcoFalke a66ba6d029
Merge #18576: test: use unittest for test_framework unit testing
de8905adf2 test: use unittest and test_runner for test framework unit testing (Gloria Zhao)

Pull request description:

  Proposal for unit testing on test_framework functions:
  1. Use the python `unittest` library. Don't use test_framework to test itself.
  2. Put the tests inside the same file as the functions they are testing.
  3. Call the tests from `test_runner.py`. To include more Test Framework tests, add the filename to the list `TEST_FRAMEWORK_MODULES`. Don't add new files or change the list of accepted script prefixes.

  Makes these changes for `bn2vch` (followup to [this comment](https://github.com/bitcoin/bitcoin/pull/18378#pullrequestreview-377271264)).

ACKs for top commit:
  jnewbery:
    Tested ACK de8905adf2. Great stuff gzhao408 . Thanks for this!

Tree-SHA512: 91572d43e203a1864765b93a9472667994115ec38b271f2b2f9fcd0f0112b393fc24ba7d2371d5a34b0a6a4522f6b934fc5164363819aa7ed8d6c6c9a60cc101
2020-04-30 09:30:13 -04:00
practicalswift 13c1f6b24f tests: Add fuzzing harness for IsRBFOptIn(...) 2020-04-30 13:19:24 +00:00
practicalswift 3439c88a5d tests: Add fuzzing harness for CBlockPolicyEstimator 2020-04-30 13:19:24 +00:00
MarcoFalke 0000ea3265
test: Add test for GetRandMillis and GetRandMicros 2020-04-30 09:19:16 -04:00
MarcoFalke fa0e5b89cf
Add templated GetRandomDuration<> 2020-04-30 09:19:14 -04:00
MarcoFalke 00c1a4d9a9
Merge #18809: rpc: Do not advertise dumptxoutset as a way to flush the chainstate
fac0cf6e55 rpc: Do not advertise dumptxoutset as a way to flush the chainstate (MarcoFalke)

Pull request description:

  The help message leaks several implementation details: leveldb and flush.

  Neither of them are relevant to the end user and I don't see why we should make them part of the API contract.

ACKs for top commit:
  laanwj:
    ACK fac0cf6e55

Tree-SHA512: 273fb85dc5be6cdccf17c43f183fa83c57d0a1cbb30555838f32c074218b713a753930009f6c98c85659421f2285f09c0a713b22f7e34d446e56737ac03870f7
2020-04-30 07:20:14 -04:00
MarcoFalke cf5e3be5ea
Merge #18825: test: fix message for ECC_InitSanityCheck test
06e434d7d9 test: fix message for ECC_InitSanityCheck test (fanquake)

Pull request description:

  OpenSSL is long gone.

ACKs for top commit:
  laanwj:
    Good catch. ACK 06e434d7d9

Tree-SHA512: 1a920fd6493e0374ca00633407e0130f987b136bc68d2062402747bda16a1e588a12bd8b0b8cdef828c9911f210386cfbdb25d478cb9b684d52769d197032064
2020-04-30 07:09:05 -04:00
fanquake 64673b1037
Merge #18780: validation: add const for minimum witness commitment size
692f8307fc test: add test for witness commitment index (fanquake)
06442549f8 validation: Add minimum witness commitment size constant (fanquake)

Pull request description:

  16101de5f3: Per [BIP 141](https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki#Commitment_structure), the witness commitment structure is at least 38 bytes,
  OP_RETURN (0x6a) + 36 (0x24) + 4 byte header (0xaa21a9ed) + 32 byte
  SHA256 hash. It can be longer, however any additional data has no
  consensus meaning.

  54f8c48d6a: As per BIP 141, if there is more than 1 pubkey that matches the witness
  commitment structure, the one with the highest output index should be
  chosen. This adds a sanity check that we are doing that, which will fail
  if anyone tries to "optimize" GetWitnessCommitmentIndex() by returning
  early.

ACKs for top commit:
  MarcoFalke:
    ACK 692f8307fc 🌵
  jonatack:
    Code review ACK 692f830
  ajtowns:
    ACK 692f8307fc
  jnewbery:
    utACK 692f8307fc
  laanwj:
    ACK 692f8307fc

Tree-SHA512: 7af3fe4b8a52fea2cdd0aec95f7bb935351a77b73d934bc88d6625a3503311b2a062cba5190b2228f97caa76840db3889032d910fc8e318ca8e7810a8afbafa0
2020-04-30 18:50:26 +08:00
fanquake 0ae8f18dfe
build: add -Wgnu to compile flags
When compiling with Clang, this will warn when GNU extensions are
used.

Info: https://clang.llvm.org/docs/DiagnosticsReference.html#wgnu
2020-04-30 18:02:04 +08:00
Ben Woosley 3a0fd7726b
Remove use of non-standard zero variadic macros
These are a gnu extension warned against by: gnu-zero-variadic-macro-arguments
2020-04-30 18:02:04 +08:00
Ben Woosley 49f6178c3e
Drop unused LOG_TIME_MICROS helper 2020-04-30 18:02:04 +08:00
DesWurstes 5d4999951e
prevector: Avoid unnamed struct, which is a GNU extension 2020-04-30 18:02:03 +08:00
Wladimir J. van der Laan afed2e98b0
Merge #18810: doc: update rest info on block size and json
ff6549c3c8 fix: update rest info on block size and json (Chris Abrams)

Pull request description:

  Addressing the ambiguous block size text in rest docs: https://github.com/bitcoin/bitcoin/issues/18703

  Also makes sure to let developers know there is `.json` option for the rest output format.

ACKs for top commit:
  MarcoFalke:
    ACK ff6549c3c8
  promag:
    ACK ff6549c3c8.

Tree-SHA512: 9ef93c1432d650b1f9599778ba092c1ca5b084a537af257078e1c713c76c5d3a4cc4b1ede8a2489964be8ed0303ad8bea58c1cb4759bbb9b24dbdebfec8001d3
2020-04-30 11:45:06 +02:00
Wladimir J. van der Laan 35ef3c15ef
Merge #18591: Add C++17 build to Travis
c31cbe7cfe Add C++17 test to Travis (Pieter Wuille)
7829685e27 Add configure option for c++17 (Pieter Wuille)
0fbde488b2 Support conversion between Spans of compatible types (Pieter Wuille)
7cbfebbf3d Update ax_cxx_compile_stdcxx.m4 (Pieter Wuille)

Pull request description:

  This adds a `--enable-c++17` option to the configure script, fixes the only C++17 incompatibility (with a commit taken from #18468), and adds a Travis test for it.

  This is all off by default, and release builds remain C++11.

  It implements the first step of the plan in https://github.com/bitcoin/bitcoin/issues/16684.

ACKs for top commit:
  elichai:
    tACK c31cbe7cfe
  practicalswift:
    Tested ACK c31cbe7cfe
  hebasto:
    ACK c31cbe7cfe, tested on Linux Mint 19.3 both C++11 and C++17 modes. Compiled and passed tests locally.

Tree-SHA512: a4b00776dbceef9c12abbb404c6bcd48f7916ce24c8c7a14116355f64e817578b7fcddbedd5ce435322319d1e4de43429b68553f4d96d970c308fe3e3e59b9d1
2020-04-30 11:16:56 +02:00
fanquake 06e434d7d9
test: fix message for ECC_InitSanityCheck test
OpenSSL is long gone.
2020-04-30 16:57:46 +08:00
Wladimir J. van der Laan 63d5ed2fc4
Merge #18437: util: Detect posix_fallocate() instead of assuming
182dbdf0f4 util: Detect posix_fallocate() instead of assuming (Vasil Dimov)

Pull request description:

  Don't assume that `posix_fallocate()` is available on Linux and not
  available on other operating systems. At least FreeBSD has it and we
  are not using it.

  Properly check whether `posix_fallocate()` is present and use it if it
  is.

ACKs for top commit:
  laanwj:
    ACK 182dbdf0f4

Tree-SHA512: f9ed4bd661f33ff6b2b1150591e860b3c1f44e12b87c35e870d06a7013c4e841ed2bf17b41ad6b18fe471b0b23a4b5e42cf1400637180888e0bc56c254fe0766
2020-04-30 10:45:17 +02:00
John Newbery 9847e205bf [docs] Improve commenting in ProcessGetData() 2020-04-29 19:34:01 -04:00
Amiti Uttarwar 2f032556e0 [test] test that an invalid GETDATA doesn't prevent processing of future messages
Co-Authored-By: John Newbery <john@johnnewbery.com>
2020-04-29 19:34:01 -04:00
MarcoFalke 36c0abd8f6
Merge #18726: test: check misbehavior more independently in p2p_filter.py
cd543d9193 test: check misbehavior more independently in p2p_filter.py (Danny Lee)

Pull request description:

  This expands on #18672 in two ways:

  - Check positive cases (`filterload` accepted, `filteradd` accepted) in addition to the negative cases added in #18672
  - Address MarcoFalke 's [suggestion](https://github.com/bitcoin/bitcoin/pull/18672#discussion_r412101752) to successfully load a filter before testing `filteradd`

ACKs for top commit:
  theStack:
    re-ACK cd543d9193

Tree-SHA512: f82402f6287ccddf08b38b6432d5e2b2b2ef528802a981d04c24bac459022f732d9090d4849d72d3d1eb2c757161dcb18c4c036b6e11dc80114e9cd49f21c3bd
2020-04-29 18:56:25 -04:00
MarcoFalke 95a9165016
Merge #18736: test: Add fuzzing harnesses for various classes/functions in util/
32b6b386a5 tests: Sort fuzzing harnesses (practicalswift)
e1e181fad1 tests: Add fuzzing coverage for JSONRPCTransactionError(...) and RPCErrorFromTransactionError(...) (practicalswift)
103b6ecce0 tests: Add fuzzing coverage for TransactionErrorString(...) (practicalswift)
dde508b8b0 tests: Add fuzzing coverage for ParseFixedPoint(...) (practicalswift)
1532259fca tests: Add fuzzing coverage for FormatHDKeypath(...) and WriteHDKeypath(...) (practicalswift)
90b635e84e tests: Add fuzzing coverage for CHECK_NONFATAL(...) (practicalswift)
a4e3d13df6 tests: Add fuzzing coverage for StringForFeeReason(...) (practicalswift)
a19598cf98 tests: Add fuzzing harness for functions in system.h (ArgsManager) (practicalswift)

Pull request description:

  Add fuzzing harnesses for various classes/functions in `util/`.

  See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).

  Happy fuzzing :)

Top commit has no ACKs.

Tree-SHA512: d27947220850c2a202c7740f44140c17545f45522596912452ccab0c2f5379abeb07cc769982c7855cb465059425206371a2b75ee1c285b03984161c9619d0b0
2020-04-29 18:54:34 -04:00
MarcoFalke 0f204dd3f2
Merge #18727: test: Add CreateWalletFromFile test
7918c1b019 test: Add CreateWalletFromFile test (Russell Yanofsky)

Pull request description:

  Add unit test calling CreateWalletFromFile, which isn't currently called from other unit tests, with some basic checks to make sure it rescans and registers for notifications correctly.

  Motivation for this change was to try to write a test that would fail without the early `handleNotifications` call in ef8c6ca60767cac589d98ca57ee33179608ccda8 from https://github.com/bitcoin/bitcoin/pull/16426, but succeed with it:

  ef8c6ca607/src/wallet/wallet.cpp (L3978-L3986)

  However, writing a full test for the race condition that call prevents isn't possible without the locking changes from #16426. So this PR just adds as much test coverage as is possible now.

  This new test is also useful for https://github.com/bitcoin/bitcoin/pull/15719, since it detects the stale notifications.transactionAddedToMempool notifications that PR eliminates.

ACKs for top commit:
  MarcoFalke:
    ACK 7918c1b019
  jonatack:
    ACK 7918c1b019

Tree-SHA512: 44035aee698ecb722c6039d061d8fac2011e9da0b314e4aff19be1d610b53cacff99016b34d6b84669bb3b61041b2318d9d8e3363658f087802ae4aa36ca17b8
2020-04-29 15:23:39 -04:00
MarcoFalke 978c5a2122
Merge #18485: test: Add mempool_updatefromblock.py
8098dea069 test: Add mempool_updatefromblock.py (Hennadii Stepanov)

Pull request description:

  This PR adds a new test for mempool update of transaction descendants/ancestors information (count, size) when transactions have been re-added from a disconnected block to the mempool.

  It could be helpful for working on PRs like #17925, #18191.

ACKs for top commit:
  ariard:
    ACK 8098dea

Tree-SHA512: 7e808fa8df8d7d7a7dbdc3f79361049b49c7bce9b58fd5539b28c9636bedac747695537e500d7ed68dc8bdb80167ad3f1c01086f7551691405d2ba2e38ef1d06
2020-04-29 12:07:13 -04:00
MarcoFalke ab91a0e0fc
Merge #18798: ci: Fix default retry script usage
45615de26c ci: Fix default retry script usage (Hennadii Stepanov)

Pull request description:

  On master (5352d14b37) `CI_RETRY_EXE=${CI_RETRY_EXE:retry}` works as a [Substring Expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html), and that is wrong.
  If `CI_RETRY_EXE` variable was unset initially, its new value becomes an empty string, but not "retry" as one could expect. Consequently, the `${CI_RETRY_EXE} ...` command does _not_ use `ci/retry/retry` script.

  This PR makes for `CI_RETRY_EXE` variable a usual parameter expansion, i.e., `${parameter:-word}`.

  Reference: https://github.com/bitcoin/bitcoin/pull/18735#issuecomment-620095489

Top commit has no ACKs.

Tree-SHA512: 108173f6b2677979b9ddf2f9b9df4a6c56f5efa81c36543a1816bb3b984e42984bf3c83fe413ea3a5ca1e2317c4efb02fea7180a6b44863af7cfe6202e9cf94d
2020-04-29 11:14:14 -04:00
MarcoFalke e302830fae
Merge #18774: test: added test for upgradewallet RPC
66fe7b1a98 test: added test for upgradewallet RPC (Harris)

Pull request description:

  This PR adds tests for the newly merged *upgradewallet* RPC.

  Additionally, it expands `test_framework/util.py` by adding the function `adjust_bitcoin_conf_for_pre_17` to support nodes that don't parse configuration sections.

  This test uses two older node versions, v0.15.2 and v0.16.3, to create older wallet versions to be used by `upgradewallet`.

  Fixes https://github.com/bitcoin/bitcoin/issues/18767

Top commit has no ACKs.

Tree-SHA512: bb72ff1e829e2c3954386cc308842820ef0828a4fbb754202b225a8748f92d4dcc5ec77fb146bfd5484a5c2f29ce95adf9f3fb4483437088ff3ea4a8d2c442c1
2020-04-29 11:09:05 -04:00
Amiti Uttarwar e257cf71c8 [net processing] ignore unknown INV types in GETDATA messages
Co-Authored-By: John Newbery <john@johnnewbery.com>
2020-04-29 10:54:55 -04:00
Amiti Uttarwar 047ceac142 [net processing] ignore tx GETDATA from blocks-only peers
Co-Authored-By: John Newbery <john@johnnewbery.com>
2020-04-29 10:54:48 -04:00
Harris 66fe7b1a98
test: added test for upgradewallet RPC 2020-04-29 16:11:49 +02:00
MarcoFalke af2ec6b037
Merge #18759: bench: Start nodes with -nodebuglogfile
fabe44e815 bench: Start nodes with -nodebuglogfile (MarcoFalke)

Pull request description:

  For benchmarking we don't want to depend on the speed of the disk or the amount of debug logging

ACKs for top commit:
  fanquake:
    ACK fabe44e815 - This makes some of these benchmarks significantly faster to run. MempoolEviction total runtime is down from ~46s to 11s on my machine:

Tree-SHA512: d99700901650325896b9115d20b84a27042152f46266f595bf7ea1414528c0b346f4e707a12ee8b8ba99c35cf155e645e67971c1b2a679c4e609c400ff8b08ae
2020-04-29 08:30:15 -04:00
MarcoFalke ecca2ea1d5
Merge #18785: Prevent valgrind false positive in rest_blockhash_by_height
fcb7261625 Prevent valgrind false positive in rest_blockhash_by_height (Russell Yanofsky)

Pull request description:

  A bad interaction between valgrind and clang 6.0.0-1ubuntu2 with -O2 optimizations makes valgrind misleadingly imply C++ code is reading an uninitialized blockheight value in `rest_blockhash_by_height` just because that's what clang optimized code is doing. The C++ code looks like:

  ```c++
  int32_t blockheight;
  if (!ParseInt32(height_str, &blockheight) || blockheight < 0) {
  ```

  while the optimized code looks like:

  ```
  0x00000000000f97ab <+123>:   callq  0x4f8860 <ParseInt32(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int*)>
  0x00000000000f97b0 <+128>:   mov    0xc(%rsp),%ebx
  0x00000000000f97b4 <+132>:   test   %ebx,%ebx
  0x00000000000f97b6 <+134>:   js     0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
  0x00000000000f97bc <+140>:   xor    $0x1,%al
  0x00000000000f97be <+142>:   jne    0xf98aa <rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+378>
  ```

  During the rest_interface.py test:

  eef90c14ed/test/functional/interface_rest.py (L266)

  when `height_str` is empty, `ParseInt32` returns false and `blockheight` value is never assigned. The optimized code reads the uninitialized `blockheight` value in `0xc(%rsp)` before the checking the `ParseInt32` return value in `%al`, which is harmless, but triggers the following error from valgrind:

  ```
  ==30660== Thread 13 b-httpworker.2:
  ==30660== Conditional jump or move depends on uninitialised value(s)
  ==30660==    at 0x2017B6: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:614)
  ==30660==    by 0x2041B9: operator() (rest.cpp:670)
  ==30660==    by 0x2041B9: std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), StartREST(util::Ref const&)::$_1>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (std_function.h:301)
  ==30660==    by 0x3EC994: operator() (std_function.h:706)
  ==30660==    by 0x3EC994: HTTPWorkItem::operator()() (httpserver.cpp:55)
  ==30660==    by 0x3ED16D: WorkQueue<HTTPClosure>::Run() (httpserver.cpp:114)
  ==30660==    by 0x3E9168: HTTPWorkQueueRun(WorkQueue<HTTPClosure>*, int) (httpserver.cpp:342)
  ==30660==    by 0x3EDAAA: __invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:60)
  ==30660==    by 0x3EDAAA: __invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int> (invoke.h:95)
  ==30660==    by 0x3EDAAA: _M_invoke<0, 1, 2> (thread:234)
  ==30660==    by 0x3EDAAA: operator() (thread:243)
  ==30660==    by 0x3EDAAA: std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*, int), WorkQueue<HTTPClosure>*, int> > >::_M_run() (thread:186)
  ==30660==    by 0x64256DE: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25)
  ==30660==    by 0x54876DA: start_thread (pthread_create.c:463)
  ==30660==    by 0x6DC888E: clone (clone.S:95)
  ==30660==  Uninitialised value was created by a stack allocation
  ==30660==    at 0x20173A: rest_blockhash_by_height(util::Ref const&, HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (rest.cpp:608)
  ==30660==
  {
     <insert_a_suppression_name_here>
     Memcheck:Cond
     fun:_ZL24rest_blockhash_by_heightRKN4util3RefEP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
     fun:operator()
     fun:_ZNSt17_Function_handlerIFbP11HTTPRequestRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZ9StartRESTRKN4util3RefEE3$_1E9_M_invokeERKSt9_Any_dataOS1_S9_
     fun:operator()
     fun:_ZN12HTTPWorkItemclEv
     fun:_ZN9WorkQueueI11HTTPClosureE3RunEv
     fun:_ZL16HTTPWorkQueueRunP9WorkQueueI11HTTPClosureEi
     fun:__invoke_impl<void, void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
     fun:__invoke<void (*)(WorkQueue<HTTPClosure> *, int), WorkQueue<HTTPClosure> *, int>
     fun:_M_invoke<0, 1, 2>
     fun:operator()
     fun:_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvP9WorkQueueI11HTTPClosureEiES6_iEEEEE6_M_runEv
     obj:/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
     fun:start_thread
     fun:clone
  }
  ```

  This is a known bad interaction between clang and valgrind. The clang optimized code is correct but valgrind has no way of knowing that accessing the uninitialized value isn't a problem. Issue has been reported previously:

  - https://bugs.llvm.org/show_bug.cgi?id=32604#c4
  - https://github.com/Z3Prover/z3/issues/972

  This commit just sets blockheight to -1 as a workaround.

  This change was originally made in 41d5d651594c6c939add7a58b7e30c97dccdf24a from #18740 to fix the travis error there (https://travis-ci.org/github/bitcoin/bitcoin/jobs/678453061#L7157) but MarcoFalke suggested https://github.com/bitcoin/bitcoin/pull/18740#discussion_r414772851 moving to a new PR, since apparently the error's been seen on travis previously

ACKs for top commit:
  MarcoFalke:
    ACK fcb7261625
  practicalswift:
    ACK fcb7261625

Tree-SHA512: ec8abf45bd3d6c6e0e7e404d0b2a749efd43910619b84b0b5fe7dab22881598d1011a0f3ff2e146bf46320b63eb152bf63c62c06f1ab84c35dd640abc468f18f
2020-04-29 08:23:06 -04:00
João Barbosa a2e6db5c4f rpc: Add mutex to guard deadlineTimers 2020-04-29 11:47:57 +01:00
fanquake 0ef0d33f75
Merge #18038: P2P: Mempool tracks locally submitted transactions to improve wallet privacy
50fc4df6c4 [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178536 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502472 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48dc5 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f20a [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eecce3 [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a333 [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)

Pull request description:

  This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

  The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

  This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

  For privacy improvements around # 1, please see #16698.
  Thank you to gmaxwell for the idea of how to break out this subset of functionality (https://github.com/bitcoin/bitcoin/pull/16698#issuecomment-571399346)

ACKs for top commit:
  fjahr:
    Code review ACK 50fc4df6c4
  MarcoFalke:
    ACK 50fc4df6c4, I think this is ready for merge now 👻
  amitiuttarwar:
    The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits.
  jnewbery:
    utACK 50fc4df6c4.
  ariard:
    Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
  robot-visions:
    ACK 50fc4df6c4
  sipa:
    utACK 50fc4df6c4
  naumenkogs:
    utACK 50fc4df

Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
2020-04-29 16:32:37 +08:00
fanquake 692f8307fc
test: add test for witness commitment index
As per BIP 141, if there is more than 1 pubkey that matches the witness
commitment structure, the one with the highest output index should be
chosen. This adds a sanity check that we are doing that, which will fail
if anyone trys to "optimise" GetWitnessCommitmentIndex() be returning
early.
2020-04-29 11:20:31 +08:00
fanquake 06442549f8
validation: Add minimum witness commitment size constant
Per BIP 141, the witness commitment structure is atleast 38 bytes,
OP_RETURN (0x6a) + 36 (0x24) + 4 byte header (0xaa21a9ed) + 32 byte
SHA256 hash. It can be longer, however any additional data has no
consensus meaning.
2020-04-29 11:20:22 +08:00
Chris Abrams ff6549c3c8 fix: update rest info on block size and json 2020-04-28 20:17:03 -05:00
MarcoFalke fac0cf6e55
rpc: Do not advertise dumptxoutset as a way to flush the chainstate 2020-04-28 20:40:47 -04:00
Hennadii Stepanov 1e06bb68be
Drop unused CLIENT_VERSION_SUFFIX macro 2020-04-28 23:10:58 +03:00
Sebastian Falbesoner 1ad8ea2b73 net: remove is{Empty,Full} flags from CBloomFilter, clarify CVE fix 2020-04-28 19:27:22 +02:00
MarcoFalke ba348dbc51
Merge #18805: tests: Add missing sync_all to wallet_importdescriptors.py
7b2b06dfe3 tests: Add missing sync_all to wallet_importdescriptors.py (Andrew Chow)

Pull request description:

  node1 will sometimes do sendtoaddress before it has received a funding transaction which will cause the test to fail. sync_all to ensure it gets the transaction first.

  Fixes #18800

ACKs for top commit:
  instagibbs:
    ACK 7b2b06d The wallet endpoint right after is indeed node 1.

Tree-SHA512: b610a771d062b5f955cd70b34337577a1ab8dacbf4be20aa74e1e8234495b0be9faff138eb1713f29decb5574446e0583e221bc2c9a6eea13611b422ea3a296a
2020-04-28 13:14:26 -04:00
MarcoFalke 6a60bfc76c
Merge #18765: test: Fix wallet_bumpfee intermittent error
fa301fec96 test: Fix wallet_bumpfee intermittent error (MarcoFalke)

Pull request description:

  Remove incorrect and undocumented `connect_nodes(self.nodes[0], 1)`.

  Issue is that transactions are re-relayed (going full circle) between the two nodes, that have two connections between each other.

  https://travis-ci.org/github/bitcoin/bitcoin/jobs/679201559#L6992

  Also fix some pep8 while touching the file

  This bug has been introduced by accident in c1dde3a949

ACKs for top commit:
  achow101:
    ACK fa301fec96

Tree-SHA512: a6565ca30dbe44b02e3f58f159d2515c2ea4a74030118fafc1a3391ce980a4b6d4505dcf51315fda24843f72550a7dea7407b877b3b796883dd73d3b6f009e6f
2020-04-28 13:11:57 -04:00
Andrew Chow 7b2b06dfe3 tests: Add missing sync_all to wallet_importdescriptors.py
node1 will sometimes do sendtoaddress before it has received a funding
transaction which will cause the test to fail. sync_all to ensure it
gets the transaction first.
2020-04-28 12:08:33 -04:00
MarcoFalke fa301fec96
test: Fix wallet_bumpfee intermittent error 2020-04-28 11:53:00 -04:00
MarcoFalke a483ffd689
Merge #18735: ci: Add and document BASE_BUILD_DIR
fae49f6e42 ci: Add and document BASE_BUILD_DIR (MarcoFalke)

Pull request description:

  Also fixes #18768

ACKs for top commit:
  hebasto:
    re-ACK fae49f6e42, which is essentially the same as the previously [reviewed changes](https://github.com/bitcoin/bitcoin/pull/18735#pullrequestreview-400581536).

Tree-SHA512: 216565a05ccd513dd9f114b2333d3c283fd71914d32f9b05f145cb7c70633b083ff8ef60798d6f22f4be6a4d652b03806551fd74b5b596c92968501a4d9726d2
2020-04-28 11:50:21 -04:00
Carl Dong fa791da02f
nsis: Specify OutFile path only once
Previously, we would specify the makensis output file path twice:

  1. At the top of Makefile.am as BITCOIN_WIN_INSTALLER, and
  2. In share/setup.nsi.ini

This commit uses the -X flag of makensis to eliminate the need for the
second instance mentioned above, referring makensis directly to the
value of BITCOIN_WIN_INSTALLER
2020-04-28 10:36:38 -04:00
Carl Dong 14701604d0
guix: Expose GIT_COMMON_DIR in container as readonly
When using worktrees or submodules, you'll see a `.git' plain text file
at the root of your working tree instead of the usual `.git' directory.

This plain text file will point to the real GIT_DIR, under the
GIT_COMMON_DIR. From experimentation, the full GIT_COMMON_DIR is
required to exist for operations such as git-archive(1), so we expose it
as readonly inside the container.
2020-04-28 10:36:37 -04:00
Carl Dong f5a6ac4f48
guix: Make source tarball using git-archive 2020-04-28 10:36:36 -04:00
Carl Dong 395c1137f6
gitian: Limit sourced script to just assignments
Previously, the sourced script would create the source tarball. Now, it
only assigns variables and the source-ing script has more flexibility in
determining what to do with these variables.

See later commit showing how this flexibility is useful in our Guix
builds.
2020-04-28 10:36:35 -04:00
MarcoFalke 6e3fc7492a
Merge #18663: doc: mention build docs in README.md
bda62e87e6 Adding build instructions to Bitcoin Core, fixes #18658 (Saahil Shangle)

Pull request description:

  Making the instructions for building Bitcoin Core more clear in the main `README.md` will reduce confusion between the `build_msvc` and `doc` folders.

ACKs for top commit:
  laanwj:
    ACK bda62e87e6

Tree-SHA512: ee4c394661eba48d4229e3d1e9ddb67ccb79589429bfa9986cb0242cd615d1f3cc5332063562c1e89c0cdd9ae2e609f61e8bfb209926d8363d35d3da6d94ae9c
2020-04-28 08:51:57 -04:00
MarcoFalke ad04130742
Merge #18799: ci: Document why tests can not be run on mac
fa72a75102 ci: Document why tests can not be run on mac (MarcoFalke)

Pull request description:

  Fixes #18794

Top commit has no ACKs.

Tree-SHA512: 297652eda412aa8cf7255e20a6f294d22773dad8637a3d7b5204f3b638e911ce5b2e40e85f81395a34c1b5a5b497665944c2d6ea17c70c30c0c9e0ab553f956e
2020-04-28 08:25:51 -04:00
MarcoFalke fa72a75102
ci: Document why tests can not be run on mac 2020-04-28 07:10:19 -04:00