Remove "nLowestTimestamp <= chainActive.Tip()->GetBlockTimeMax()" check from
importmulti, which is always true because nLowestTimestamp is set to the
minimum of the most recent block time and all the imported key timestamps,
which is necessarily lower than the maximum block time.
Github-Pull: #9760
Rebased-From: ec1267f13b
e662af3 Use 2 hour grace period for key timestamps in importmulti rescans (Russell Yanofsky)
38d3e9e [qa] Extend import-rescan.py to test imports on pruned nodes. (Russell Yanofsky)
c28583d [qa] Extend import-rescan.py to test specific key timestamps (Russell Yanofsky)
8be0866 [qa] Simplify import-rescan.py (Russell Yanofsky)
- If the -maxsigcachesize parameter is set to zero, setup a minimum sized
sigcache (2 elements) rather than segfaulting.
- Handle maxsigcachesize being negative
- Handle maxsigcachesize being too large
A new AssertLockHeld(cs_wallet) call was added in commit a58370e
"Dedup nTimeFirstKey update logic" (part of PR #9108).
The lock held assertion will fail when loading prexisting wallets files from
before the #9108 merge that have watch-only keys.
Fixes a bug in AcceptBlock() in invoking CheckBlock() with incorrect
arguments, and restores a call to CheckBlock() from ProcessNewBlock()
as belt-and-suspenders.
Updates the (overspecified) tests to match behavior.
Because it is used inconsistently at least version 5.4.0 of g++ to
complains about methods that don't use override. There is two ways to go
about this: remove override from the methods having it, or add it to the
methods missing it. I chose the second.
d943491 qa: add a test to detect leaky p2p messages (Cory Fields)
8650bbb qa: Expose on-connection to mininode listeners (Matt Corallo)
5b5e4f8 qa: mininode learns when a socket connects, not its first action (Matt Corallo)
cbfc5a6 net: require a verack before responding to anything else (Cory Fields)
8502e7a net: parse reject earlier (Cory Fields)
c45b9fb net: correctly ban before the handshake is complete (Cory Fields)
66f861a Add a test for P2P inactivity timeouts (Matt Corallo)
b436f92 qa: Expose on-connection to mininode listeners (Matt Corallo)
8aaba7a qa: mininode learns when a socket connects, not its first action (Matt Corallo)
2cbd119 Disconnect peers which we do not receive VERACKs from within 60 sec (Matt Corallo)
266a811 Use MTP for importmulti "now" timestamps (Russell Yanofsky)
3cf9917 Add test to check new importmulti "now" value (Russell Yanofsky)
442887f Require timestamps for importmulti keys (Russell Yanofsky)
7179e7c qt: Periodic translations update (Wladimir J. van der Laan)
5e903a5 devtools: Handle Qt formatting characters edge-case in update-translations.py (Wladimir J. van der Laan)
7a8c251901 made this logic hard to follow. After that change, messages would
not be sent to a peer via SendMessages() before the handshake was complete, but
messages could still be sent as a response to an incoming message.
For example, if a peer had not yet sent a verack, we wouldn't notify it about
new blocks, but we would respond to a PING with a PONG.
This change makes the behavior straightforward: until we've received a verack,
never send any message other than version/verack/reject.
The behavior until a VERACK is received has always been undefined, this change
just tightens our policy.
This also makes testing much easier, because we can now connect but not send
version/verack, and anything sent to us is an error.
Prior to this change, all messages were ignored until a VERSION message was
received, as well as possibly incurring a ban score.
Since REJECT messages can be sent at any time (including as a response to a bad
VERSION message), make sure to always parse them.
Moving this parsing up keeps it from being caught in the
if (pfrom->nVersion == 0) check below.
7a8c251901 made a change to avoid getting into SendMessages() until the
version handshake (VERSION + VERACK) is complete. That was done to avoid
leaking out messages to nodes who could connect, but never bothered sending
us their version/verack.
Unfortunately, the ban tally and possible disconnect are done as part of
SendMessages(). So after 7a8c251901, if a peer managed to do something
bannable before completing the handshake (say send 100 non-version messages
before their version), they wouldn't actually end up getting
disconnected/banned. That's fixed here by checking the banscore as part of
ProcessMessages() in addition to SendMessages().
a60677e Pre-0.14.0 hardcoded seeds update (Wladimir J. van der Laan)
bfa9393 contrib/seeds: Update PATTERN_AGENT (Wladimir J. van der Laan)
4dfac2c Update seeds tooling to Python 3 (Wladimir J. van der Laan)
When importing a watch-only address over importmulti with a specific timestamp,
the wallet's nTimeFirstKey is currently set to 1. After this change, the
provided timestamp will be used and stored as metadata associated with
watch-only key. This can improve wallet performance because it can avoid the
need to scan the entire blockchain for watch only addresses when timestamps are
provided.
Also adds timestamp to validateaddress return value (needed for tests).
Fixes#9034.
Additionally, accept a "now" timestamp, to allow avoiding rescans for keys
which are known never to have been used.
Note that the behavior when "now" is specified is slightly different than the
previous behavior when no timestamp was specified at all. Previously, when no
timestamp was specified, it would avoid rescanning during the importmulti call,
but set the key's nCreateTime value to 1, which would not prevent future block
reads in later ScanForWalletTransactions calls. With this change, passing a
"now" timestamp will set the key's nCreateTime to the current block time
instead of 1.
Fixes#9491
These are (afaik) all long-standing races or concurrent accesses. Going
forward, we can clean these up so that they're not all individual atomic
accesses.
- Reintroduce cs_vRecv to guard receive-specific vars
- Lock vRecv/vSend for CNodeStats
- Make some vars atomic.
- Only set the connection time in CNode's constructor so that it doesn't change
a9baa6d Bugfix: Qt/Intro: Pruned nodes never require *more* space (Luke Dashjr)
93ffba7 Bugfix: Qt/Intro: Chain state needs to be stored even with the full blockchain (Luke Dashjr)
c8cee26 Qt/Intro: Update block chain size (Luke Dashjr)
618ee92 Further-enforce lockordering by enforcing directly after TRY_LOCKs (Matt Corallo)
2a962d4 Fixup style a bit by moving { to the same line as if statements (Matt Corallo)
8465631 Always enforce lock strict lock ordering (try or not) (Matt Corallo)
fd13eca Lock cs_vSend and cs_inventory in a consistent order even in TRY (Matt Corallo)
The initialization order of global data structures in different
implementation units is undefined. Making use of this is essentially
gambling on what the linker does, the so-called [Static initialization
order fiasco](https://isocpp.org/wiki/faq/ctors#static-init-order).
In this case it apparently worked on Linux but failed on OpenBSD and
FreeBSD.
To create it on first use, make the registration structure local to
a function.
Fixes#8910.
6dbfe08 [qa] test signrawtransaction merge with missing inputs (Matt Corallo)
ec4f7e4 [qa] Add second input to signrawtransaction test case (Matt Corallo)
691710a [qa] Test that decoderawtransaction throws with extra data appended (Matt Corallo)
922bea9 Better handle invalid parameters to signrawtransaction (Matt Corallo)
7ea0ad5 Fail in DecodeHexTx if there is extra data at the end (Matt Corallo)
0729102 Net: pass interruptMsgProc as const where possible (Jorge Timón)
fc7f2ff Net: Make CNetMsgMaker more const (Jorge Timón)
d45955f Net: CConnman: Make some methods const (Jorge Timón)
08bb6f4 net: log an error rather than asserting if send version is misused (Cory Fields)
7a8c251 net: Disallow sending messages until the version handshake is complete (Cory Fields)
12752af net: don't run callbacks on nodes that haven't completed the version handshake (Cory Fields)
2046617 net: deserialize the entire version message locally (Cory Fields)
80ff034 Dont deserialize nVersion into CNode, should fix#9212 (Matt Corallo)
Preserve comment, order form, and account strings from the original wallet
transaction. Also set fTimeReceivedIsTxTime and fFromMe fields for consistency
with CWallet::CreateTransaction. The latter two fields don't influence current
wallet behavior, but do record that the transaction originated in the wallet
instead of coming from the network or sendrawtransaction.
This silently skips trying to merge signatures from inputs which
do not exist from transactions provided to signrawtransaction,
instead of hitting an assert.
Since ForEach* are can be used to send messages to all nodes, the caller may
end up sending a message before the version handshake is complete. To limit
this, filter out these nodes. While we're at it, may as well filter out
disconnected nodes as well.
Delete unused methods rather than updating them.
This avoids having some vars set if the version negotiation fails.
Also copy it all into CNode at the same site. nVersion and
fSuccessfullyConnected are set last, as they are the gates for the other vars.
Make them atomic for that reason.
Once the CNode has been added to vNodes, it is possible that it is
disconnected+deleted in the socket handler thread. However, after
that we now call InitializeNode, which accesses the pnode.
helgrind managed to tickle this case (somehow), but I suspect it
requires in immensely braindead scheduler.
More accurate than simply adding one byte per input, and properly handles the
case where the original transaction happened to have very small signatures
2366180 Do not add to vNodes until fOneShot/fFeeler/fAddNode have been set (Matt Corallo)
3c37dc4 Ensure cs_vNodes is held when using the return value from FindNode (Matt Corallo)
5be0190 Delete some unused (and broken) functions in CConnman (Matt Corallo)
4b189c1 Change bumpfee result value from 'oldfee' to 'origfee'. (Alex Morcos)
0c0c63f Introduce WALLET_INCREMENTAL_RELAY_FEE (Alex Morcos)
e8021ec Use CWallet::GetMinimumFee in bumpfee (Alex Morcos)
ae9719a Refactor GetMinimumFee to give option of providing targetFee (Alex Morcos)
fe8e8ef [rpc] Add incremental relay fee to getnetworkinfo (Alex Morcos)
6b331e6 Fix to have miner test aware of new separate block min tx fee (Alex Morcos)
de6400d Fix missing use of dustRelayFee (Alex Morcos)
5b15870 Use incrementalRelayFee for BIP 125 replacement (Alex Morcos)
The result value indicates the actual fee on the transaction that was replaced. But there is an error message which uses the description 'oldfee' to refer to the original fee rate applied to the new transaction's estimated max size. It was confusing that two different uses of 'oldfee' had two different numeric values.
Have wallet's default bump value be higher than the default incrementalRelayFee to future proof against changes to incremental relay fee. Only applies when not setting the fee rate directly.
Use the wallet's fee calculation logic to properly clamp fee against minimums and maximums when calculating the fee for a bumpfee transaction. Unless totalFee is explictly given, in which case, manually check against min, but do nothing to adjust given fee.
In all cases do a final check against maxTxFee (after adding any incremental amount).
The use of mocktime in test logic means that comparisons between
GetTime() and GetTimeMicros()/1000000 are unreliable since the former
can use mocktime values while the latter always gets the system clock;
this changes the networking code's inactivity checks to consistently
use the system clock for inactivity comparisons.
Also remove some hacks from setmocktime() that are no longer needed,
now that we're using the system clock for nLastSend and nLastRecv.
094e4b3 Better document usage of SyncTransaction (Alex Morcos)
4afbde6 Introduce MemPoolConflictRemovalTracker (Alex Morcos)
ff25c32 mempool: add notification for added/removed entries (Wladimir J. van der Laan)
Analogue to ConnectTrace that tracks transactions that have been removed from the mempool due to conflicts and then passes them through SyncTransaction at the end of its scope.
Add notification signals to make it possible to subscribe to mempool
changes:
- NotifyEntryAdded(CTransactionRef)>
- NotifyEntryRemoved(CTransactionRef, MemPoolRemovalReason)>
Also add a mempool removal reason enumeration, which is passed to the
removed notification based on why the transaction was removed from
the mempool.
7ba0a00 Testing: listsinceblock should not use orphan block height. (Karl-Johan Alm)
ee5c1ce Bug-fix: listsinceblock: use closest common ancestor when a block hash was provided for a chain that was not the main chain. (Karl-Johan Alm)
This adds a comment to the new logic for setting HB peers based
on block validation (and aligns the code below to reflect the comment).
It's not obvious why we're checking mapBlocksInFlight. Add a comment to
explain.
The additional initializer is for the named arguments, which are unused
in the test (and unfilled global fields will be initialized to 0
anyhow), so this is a no-op apart from the warning.
c9f3062 Add fundrawtransactions new reserveChangeKey option to the release notes (Jonas Schnelli)
9eb325d [QA] Add test for fundrawtransactions new reserveChangeKey option (Jonas Schnelli)
9aa4e6a [Wallet] Add an option to keep the change address key, true by default (Jonas Schnelli)
The old Bitcoin alert system has long since been retired.
( See also: https://bitcoin.org/en/alert/2016-11-01-alert-retirement )
This change causes each node to send any old peers that
it connects with a copy of the final alert.
The alert it hardcode cancels all other alerts including
other final alerts.
376b3c2 Make the cs_sendProcessing a LOCK instead of a TRY_LOCK (Matt Corallo)
d7c58ad Split CNode::cs_vSend: message processing and message sending (Matt Corallo)
This command allows a user to increase the fee on a wallet transaction T, creating a "bumper" transaction B.
T must signal that it is BIP-125 replaceable.
T's change output is decremented to pay the additional fee. (B will not add inputs to T.)
T cannot have any descendant transactions.
Once B bumps T, neither T nor B's outputs can be spent until either T or (more likely) B is mined.
Includes code by @jonasschnelli and @ryanofsky
This forces the message handling thread to make another full
iteration of SendMessages prior to going back to sleep, ensuring
we announce the new block to all peers before sleeping.
4b06e41 Add unit test for FindEarliestAtLeast (Suhas Daftuar)
997a98a Replace FindLatestBefore used by importmuti with FindEarliestAtLeast. (Gregory Maxwell)
02ee4eb Make most_recent_compact_block a pointer to a const (Matt Corallo)
73666ad Add comment to describe callers to ActivateBestChain (Matt Corallo)
962f7f0 Call ActivateBestChain without cs_main/with most_recent_block (Matt Corallo)
0df777d Use a temp pindex to avoid a const_cast in ProcessNewBlockHeaders (Matt Corallo)
c1ae4fc Avoid holding cs_most_recent_block while calling ReadBlockFromDisk (Matt Corallo)
9eb67f5 Ensure we meet the BIP 152 old-relay-types response requirements (Matt Corallo)
5749a85 Cache most-recently-connected compact block (Matt Corallo)
9eaec08 Cache most-recently-announced block's shared_ptr (Matt Corallo)
c802092 Relay compact block messages prior to full block connection (Matt Corallo)
6987219 Add a CValidationInterface::NewPoWValidBlock callback (Matt Corallo)
180586f Call AcceptBlock with the block's shared_ptr instead of CBlock& (Matt Corallo)
8baaba6 [qa] Avoid race in preciousblock test. (Matt Corallo)
9a0b2f4 [qa] Make compact blocks test construction using fetch methods (Matt Corallo)
8017547 Make CBlockIndex*es in net_processing const (Matt Corallo)
Technically cs_sendProcessing is entirely useless now because it
is only ever taken on the one MessageHandler thread, but because
there may be multiple of those in the future, it is left in place
cs_vSend is used for two purposes - to lock the datastructures used
to queue messages to place on the wire and to only call
SendMessages once at a time per-node. I believe SendMessages used
to access some of the vSendMsg stuff, but it doesn't anymore, so
these locks do not need to be on the same mutex, and also make
deadlocking much more likely.
e60360e net: remove cs_vRecvMsg (Cory Fields)
991955e net: add a flag to indicate when a node's send buffer is full (Cory Fields)
c6e8a9b net: add a flag to indicate when a node's process queue is full (Cory Fields)
4d712e3 net: add a new message queue for the message processor (Cory Fields)
c5a8b1b net: rework the way that the messagehandler sleeps (Cory Fields)
c72cc88 net: remove useless comments (Cory Fields)
ef7b5ec net: Add a simple function for waking the message handler (Cory Fields)
f5c36d1 net: record bytes written before notifying the message processor (Cory Fields)
60befa3 net: handle message accounting in ReceiveMsgBytes (Cory Fields)
56212e2 net: set message deserialization version when it's actually time to deserialize (Cory Fields)
0e973d9 net: remove redundant max sendbuffer size check (Cory Fields)
6042587 net: wait until the node is destroyed to delete its recv buffer (Cory Fields)
f6315e0 net: only disconnect if fDisconnect has been set (Cory Fields)
5b4a8ac net: make GetReceiveFloodSize public (Cory Fields)
e5bcd9c net: make vRecvMsg a list so that we can use splice() (Cory Fields)
53ad9a1 net: fix typo causing the wrong receive buffer size (Cory Fields)
This disentangles the script validation skipping from checkpoints.
A new option is introduced "assumevalid" which specifies a block whos
ancestors we assume all have valid scriptsigs and so we do not check
them when they are also burried under the best header by two weeks
worth of work.
Unlike checkpoints this has no influence on consensus unless you set
it to a block with an invalid history. Because of this it can be
easily be updated without risk of influencing the network consensus.
This results in a massive IBD speedup.
This approach was independently recommended by Peter Todd and Luke-Jr
since POW based signature skipping (see PR#9180) does not have the
verifiable properties of a specific hash and may create bad incentives.
The downside is that, like checkpoints, the defaults bitrot and older
releases will sync slower. On the plus side users can provide their
own value here, and if they set it to something crazy all that will
happen is more time will be spend validating signatures.
Checkblocks and checklevel are also moved to the hidden debug options:
Especially now that checkblocks has a low default there is little need
to change these settings, and users frequently misunderstand them as
influencing security or IBD speed. By hiding them we offset the
space added by this new option.
vRecvMsg is now only touched by the socket handler thread.
The accounting vars (nRecvBytes/nLastRecv/mapRecvBytesPerMsgCmd) are also
only used by the socket handler thread, with the exception of queries from
rpc/gui. These accesses are not threadsafe, but they never were. This needs to
be addressed separately.
Also, update comment describing data flow
Similar to the recv flag, but this one indicates whether or not the net's send
buffer is full.
The socket handler checks the send queue when a new message is added and pauses
if necessary, and possibly unpauses after each message is drained from its buffer.
Messages are dumped very quickly from the socket handler to the processor, so
it's the depth of the processing queue that's interesting.
The socket handler checks the process queue's size during the brief message
hand-off and pauses if necessary, and the processor possibly unpauses each time
a message is popped off of its queue.
In order to sleep accurately, the message handler needs to know if _any_ node
has more processing that it should do before the entire thread sleeps.
Rather than returning a value that represents whether ProcessMessages
encountered a message that should trigger a disconnnect, interpret the return
value as whether or not that node has more work to do.
Also, use a global fProcessWake value that can be set by other threads,
which takes precedence (for one cycle) over the messagehandler's decision.
Note that the previous behavior was to only process one message per loop
(except in the case of a bad checksum or invalid header). That was changed in
PR #3180.
The only change here in that regard is that the current node now falls to the
back of the processing queue for the bad checksum/invalid header cases.
In spite of the name FindLatestBefore used std::lower_bound to try
to find the earliest block with a nTime greater or equal to the
the requested value. But lower_bound uses bisection and requires
the input to be ordered with respect to the comparison operation.
Block times are not well ordered.
I don't know what lower_bound is permitted to do when the data
is not sufficiently ordered, but it's probably not good.
(I could construct an implementation which would infinite loop...)
To resolve the issue this commit introduces a maximum-so-far to the
block indexes and searches that.
For clarity the function is renamed to reflect what it actually does.
An issue that remains is that there is no grace period in importmulti:
If a address is created at time T and a send is immediately broadcast
and included by a miner with a slow clock there may not yet have been
any block with at least time T.
The normal rescan has a grace period of 7200 seconds, but importmulti
does not.
0c50909 testcases: explicitly specify transaction version 1 (John Newbery)
b7e144b Add test cases to test new bitcoin-tx functionality (jnewbery)
61a1534 Add all transaction output types to bitcoin-tx. (jnewbery)
1814b08 add p2sh and segwit options to bitcoin-tx outscript command (Stanislas Marion)
There is still a call to ActivateBestChain with cs_main if a peer
requests the block prior to it being validated, but this one is
more specifically-gated, so should be less of an issue.
