Farmers carrying hay on pitchforks

hardfork_prototype_0 - an early prototype of a Bitcoin hard fork

Part 2 - Technical overview

In this part, I will give a high-level view of the prototype's design, some rationale for choices I made during its development, and an account of what I think is missing, unfinished or not well-implemented compared to how I now imagine a good, safe spin-off.

HFP0 in brief

Hard fork triggered purely on block height (separate trigger heights for mainnet/testnet/regtest)
Proof-of-work: SHA256 or satoshisbitcoin's modified scrypt, depending on build configuration option
Difficulty reset to low value at fork trigger
Difficulty algorithm after fork height: MIDAS (per-block retargeting, similar to Dash)
Block size cap: adaptive (BitPay) with floor of 2MB and ceiling of 4MB
Block propagation: Xtreme Thinblocks (from Bitcoin Unlimited)
Alert system disabled
RBF: disabled (courtesy of Classic 0.12 baseline)
Network separation: DoS score banning (no change of network magic or port numbers)
Retrofitted: BIP9, BIP65, BIP68, BIP112, BIP113 + minor cherry-picked bugfixes from various clients

A reminder: HFP0 is a prototype, not ready for production use

You are welcome to test and play with hardfork_prototype_0, but I strongly suggest you don't put any money (yours or anyone else's) at risk doing so. It's a prototype spin-off which not ready for production. As with any other fork code (or altcoin), you should run it safely.

Feel free to use whatever of it appears useful to your own development, and if you have criticism and feedback, leave comments on /r/btcfork, on the Bitcoin Forum or on any channel of the BTCfork community.

You don't have to be mad to attempt a SHA256 spin-off, but...

Cryptocurrency is a bit of a war zone, with new currencies arising and coming under attack constantly. It's very much survival of the fittest, and one of Bitcoin's advantages has been its substantial network effect that stems from being the first cryptocurrency to 'make it big'. Bitcoin is a gateway to acquiring amounts of other cryptocurrencies and still occupies pole position in the cryptocurrency competition. It has a sizeable established network and surrounding ecosystem which act as an immune system.

Unlike some altcoins which purposefully choose their own unique POW, HFP0 started out under the premise that it would not change the traditional Bitcoin SHA256 POW, specifically to enable existing miners to support it, as the strong security of Bitcoin rests on their hashpower.

A spin-off of Bitcoin which retains the POW and ledger must compete head-to-head with the existing network on its technical value proposition. This could well trigger that immune system to fight against the spin-off using a portion of its substantial hashpower and financial reserves instead of letting the market decide on more technical merits.

HFP0 was designed to survive in an environment where it would be attacked by the existing miners. Such attacks are not hypothetical - at least one existing Bitcoin Core developer has a track record of attacking other cryptocurrencies. Angel investor Chandler Guo, who backs Chinese Bitcoin enterprises, including mining-related companies, has previously expressed intention to use hashpower at his disposal to perform a 51% attack on another cryptocurrency's fork he didn't like. Therefore it is reasonable to assume that some Bitcoin miners might defend their interests with such means.

Is it possible to defend such a spin-off against these overwhelming odds? Provided that you have a well-validated client (not a prototype) and can get some pools, exchanges and miners on board from the start, I'm confident it could work. It would nevertheless require additional defensive measures which satoshisbitcoin didn't need because

it changed the POW to something ASIC-resistant
it only increased the block size very conservatively, to 2MB

Before going over the list of HFP0 additions / changes in more detail, a word on the mainnet / testnet block height triggers you will find in the code (SIZE_FORK_HEIGHT_*). They have been intentionally set to dummy figures. The regtest trigger height is set to 1000 to prevent other tests (related to BIP9-counted softforks) from interfering with the hardfork-specific tests. I wouldn't recommend lowering that value below 1000 (initially I had set it to 160 but after merging BIP9 and other BIPs I quickly found out that it was easier just to raise it than get other range-dependent tests to play nicely with the fork).

On to the other requirements that have been implemented:

1. More reactive difficulty retargeting

A block height fork needs to reset the difficulty to a low value when it triggers, so that miners can mine successfully and roughly meet the 10 minute average confirmation times to which users are accustomed. In the initial stages, the aggregate hashrate of the fork chain can be expected to be low compared to the incumbent chain, and relatively speaking, much more variable.

In this situation, there are a number of majority attacks that can be executed by miners of the existing chain.

Hash bombing is an attack that can be used against multiple "naive" forks if they don't adjust their difficulty quickly enough.

The attacking miners temporarily ramp up the difficulty on one fork, then move on to the next, leaving the previous fork chain to languish while its lesser-powered miners are unable to find blocks quick enough at the now high difficulty ... rinse and repeat.

To fend off such attacks, spin-offs need to be able to adjust their difficulty quickly in the face of hashpower changes.

HFP0 tackles this problem by replacing the historic, every-2016-blocks retargeting algorithm with a much more responsive, per-block algorithm - the Multi Interval Difficulty Adjustment System (MIDAS). MIDAS was specifically designed to help a coin survive large fluctuations in hashpower and timewarp attacks such as the Zeitgeist attack performed against Geistgeld and similar attacks against other coins.

There has been a lot of discussion about whether the "magic number" of 2016 blocks in Bitcoin's block retargeting interval actually serves some useful purpose. It equates to roughly 2 weeks (14 days @ 144 blocks/day). It has the observable effect that Bitcoin reacts quite slowly to changes in hashpower, which is good for the established chain and dominant miners, but makes life difficult for a fledgling spin-off.

It seems there isn't anything special about the 2016-block interval. One argument raised in its favor (other than "because that's what Satoshi picked") is that it provides a form of planning security for miners and pools. I'm not convinced. Surely having more up-to-date information on the actual trend in hashpower is more valuable than not having that information, and instead having a growing uncertainty as time goes on? If you have come across a more convincing logical argument, please let me know.

I don't believe Satoshi intended the long interval as a defense against rival hard forks, but I'm open to correction (please provide a citation).

What surprised me, from discussions within BTCfork, is that quite a few developers are fond enough of the 2016 number to want to keep it around, or at least re-converge to it soon after a fork. I think that's risky. There is no crystal ball that can predict when the hashrate will stabilize. Whatever number or algorithm you pick to converge back to the status quo - adversaries can exploit it after studying your fork code and waiting the required amount of time before attacking.

Recalculating on every block allows maximal responsiveness to whichever algorithm you pick, which is where the defensive strength lies. It also doesn't cost much in terms of performance. Compared to the other block validation computations, the difficulty retargeting overhead is insignificant even for a more complex algorithm such as MIDAS.

Per-block retargeting is used successfully in many well-known coins (Ethereum, Monero, Dash, Dogecoin) with various algorithms (e.g. Dash's Dark Gravity Wave v3, Dogecoin's DigiShield, etc). Some of these coins have much faster block periods than Bitcoin's 10 minute average. Therefore it is clear that per-block retargeting is not inherently flawed and scales down to even small block periods (not that I'm suggesting we should change the block period at this stage).

2. Adaptive block size cap

From the outset, HFP0 was intended to support a block size larger than Classic's 2MB.

To prevent a situation where an attacker could spam the network with huge blocks consisting only of their own transactions, I wanted to include an algorithm to dynamically adjust the maximum block size between some lower and upper bound, based on network demand.

BitPay had earlier presented their thoughts and a design for an adaptive block size algorithm which adjusted the maximum block size based on the median block size in a historic window. They implemented it to study its performance w.r.t. historical block size data and the results were quite encouraging.

I liked this simple algorithm proposal from the start, because its core design was virtually identical to the median-based algorithm used in Monero (although Monero goes further and adjusts the block reward if the block size exceeds a minimum median size). I am not a fan of flexcap-like proposals. By changing Bitcoin's rather elegant and simple incentive structure, they could introduce unforeseen economic consequences. So far, I haven't seen any studies which refute BitPay's findings that their adaptive block size proposal could work well for Bitcoin and public opinion of it seemed quite favorable since its release.

The BitPay adaptive block size algorithm is a good choice for a fork which doesn't completely abolish the block size cap, but wants to offer a pathway to significantly larger blocks. An algorithmically controlled block size cap is much easier to come to terms with than Bitcoin Unlimited's radical concept of "emergent consensus".

Nowadays, I think BU's concept is more widely understood than it was in early 2016, which is reflected in the increasing support among users and miners. An overwhelming majority of respondents in a Reddit poll recently voted BU as their favored baseline client for BTCfork's MVF implementation.

HFP0 modifies BitPay's adaptive algorithm slightly to provide a raised floor of 2MB (instead of the historic 1MB). I've also added a ceiling of 4MB, since 4MB has been described as relatively safe in the Cornell study on block propagation.

Soft-limit scaling has been disabled for simplicity reasons (the configured soft limit is used as-is). A production-ready spin-off using of BitPay's algorithm would almost certainly provide automatic soft-limit scaling.

3. Improved block propagation using Xtreme Thinblocks

The traditional block diffusion method in Bitcoin is not optimal as it transfers uncompressed block data all the time.

For a small increase in the block size, such as from 1MB to 2MB, this would be hardly noticeable. Studies have shown that as the block size increases, propagation within the network becomes increasingly problematic, although at sizes up to 4MB, 90% of the nodes measured could cope without problems.

The block propagation bottleneck had been known for quite some time and major clients have all added improved block propagation methods:

Bitcoin XT's thinblocks implementation
Bitcoin Unlimited's Xtreme Thinblocks (Xthinblocks)
Bitcoin Core's CompactBlocks

Before working on this fork, I had briefly started to look at yet another block propagation improvement idea - Jonathan Toomim's 'blocktorrent' concept. I decided that working on a hard fork was more urgent, as Xthinblocks were already in good shape and a further block propagation algorithm would provide only marginal improvement. However, the main projects were not focused on providing a guaranteed (non-elective) alternative plan to Core's soft-fork-riddled roadmap, in case the BIP109 election strategy failed or some other unforeseen disaster (e.g. Fee Event, Halving Death Spiral) were to befall due to 1MB blocks filling up as predicted.

While I was working on HFP0, Bitcoin Unlimited's Xthinblocks were already proving themselves in the field. I witnessed firsthand the bandwidth savings on my Unlimited node. I was impressed and confident that Xthinblocks could easily handle fast propagation of larger blocks, preventing adverse affects on the network even if consumer-grade connectivity was assumed. The Bitcoin Unlimited team has since rigorously demonstrated the beneficial effects on latency and bandwidth in a well-received experiment conducted across the Great Firewall of China.

Rusty Loy ported the Xthinblocks implementation over to Bitcoin Classic 0.12, which was fortunate since it meant I didn't have to port it directly from BU myself, but could instead integrate his code quite easily into HFP0.

When CompactBlocks were announced by Core, I didn't see enough additional value over Xthinblocks to merit integrating them, especially as a spin-off would at first not need to interoperate with other clients.

Minor improvements

4. More effective network separation

HFP0 separates from other-chain peers more effectively than the final version of satoshisbitcoin did. In fact this seems to have been one of the remaining open issues after satoshisbitcoin's last public test.

Recent discussions with members of the BTCfork development community have convinced me that HFP0 is still messier in this regard than it should be. I elaborate on this in the section 'Missing items' below.

5. Disabling of alert system

The alert system has been disabled. If the command line option or configuration file option is enabled, the client prints a message that the system is deprecated and disabled.

The alert system was initially considered in satoshisbitcoin as a tool to help with fork separation.

However, Core and other clients have subsequently removed it and there is no good reason for a spin-off to keep it around. It just introduces an issue of centralized alert key management. As it turned out, that didn't work out well the last time.

The obsolete alert system code in HFP0 was intended to be completely removed in a later clean-up update.

6. Backport activated soft-forks

BIPs 65, 68, 112 and 113 were considered necessary as they are already active on the main chain.

Missing items

This fork prototype lacks several features which should be implemented in a finished version.

Some of them should be considered critical - others are not so important or just nice to have.

Urgent

The missing features listed below are what I consider highly important. If left unimplemented, they would in my opinion seriously and unnecessarily endanger the viability of an attempted spin-off.

Replay attack protection (signature change)

This prototype is lacking protection against so-called 'replay attacks'. A replay attack can occur when a malicious actor can pick up transactions on the fly on one network and replay them onto the other (forked) network against the wishes of the transaction sender.

In a spin-off situation, many holders of coins will want to have the ability to transact independently with their pre-fork holdings on the various fork chains. This helps with price finding and boils down to your right as a holder to spend your money where you please, when you please.

To guard against malicious replays, a fork needs to modify the signature scheme, so that only the original sender can produce a valid transaction on either chain.

The initial design I came up with proved inadequate - it prevented replay of transactions only in one direction. It would have still been possible for transactions made using the spin-off client to be replayed on the other chain.

After discussions among BTCfork developers, a better solution seems to have been found. This will be implemented and tested in a future fork prototype (unless analysis reveals a weakness and requires a change of plans).

Proper DNS seeds, static IP seeds

Until now, I've only tested HFP0 in an isolated test environment (although it did include an isolated version of testnet).

In the released HFP0 code, all DNS seeds have been removed and IP seeds replaced by a shortlist of private IPs from my test environment. This is because:

there is no need to bother existing public peers while testing a prototype on an isolated test network
it simplified test activities

A viable spin-off should come with public seeds to get up and running, and so it would need to populate the seed lists with its own addresses.

To make it easier to re-use software builds in various test environment, it would be nice if seed configuration changes did not require a recompilation of the application. That's a nice little task for a rainy day.

Running the prototype without preconfigured peers should work fine, only requiring some additional manual actions (e.g. addnode commands) to establish peer connections.

Review and evaluation of modified POW

The modified scrypt algorithm has been exposed to some informal testing during satoshisbitcoin's public tests and private tests of HFP0.

However it is not known to have received significant code review and rigorous verification.

As this modified scrypt POW is intended to be mined using CPUs, the scrypt variant of HFP0 might be vulnerable to botnet attacks. Further analysis is definitely needed to determine weaknesses and possible improvements.

It may be preferable to exchange the modified scrypt POW algorithm for one that has already been proven in practice. There are several strong GPU algorithms that may bring existing hashpower benefits to a spin-off, however there are also existing mining entities in the Bitcoin space which have significant altcoin GPU mining hashpower alongside their Bitcoin ASIC mining operation.

Backup of pre-fork wallet

After the fork occurs, transactions involving post-fork coins can start to taint the user's wallet file, making it difficult or impossible to spend coins from the wallet with an old-chain client.

It would be sensible to create a backup for user of the wallet file as it existed just prior to the fork. That way the user would have no problem accessing pre-fork coins on the old chain as well.

HFP0 currently does not back up the wallet, instead relying on the user to create their own wallet backup before the fork triggers. Some users will inevitably forget to do such backups, leading to support problems. It's also questionable whether this can be done safely manually, without shutting down the client. Stopping for a wallet backup might not be satisfactory for users who might need to keep on transacting across the fork height. A software feature to assist users is required.

Cleaner network separation

HFP0 does not change the network magic or TCP port at fork time. It remains connected at network level to the old chain's network, processing protocol data that they may send to HFP0 peers.

While it separates slightly better than satoshisbitcoin's final version from the old chain network, the network separation is still deficient.

HFP0 separation relies on the banning of peers which send it old-version blocks after the fork height. At fork time, it switches to a new block version and accepts only blocks of that version (or higher). However, it carries on receiving new blocks even from the old chain, and looks at them to see if they are forked blocks or not. If it sees that that they have the wrong block version, it raises the ban score on the peer that sent it, until that peer eventually becomes disconnected. In turn, the old chain peer would ban the forked peer as well for misbehaving, at the latest when the fork chain peer keeps sending it blocks that have the wrong difficulty or whose has doesn't match SHA256 anymore (for the POW fork chain).

A better fork design would change the network magic. This would significantly reduce the amount of unnecessary processing it does on packets that come from the old chain. An attacker would have to spoof the network magic to feed protocol packets to the new chain, otherwise it would just not process them further.

Even better would be if the fork disconnected cleanly from non-fork peers (without the banning drama). But how to reconnect only to the peers of your fork? Can you tell them apart from the other clients that are not aware of your fork, that you may not want to connect to? It becomes a little complex.

To simplify that complexity, all fork-aware (and fork-compatible) clients could move together to a separate network port, forming their own Bitcoin peer network disconnected from the existing one. This requires a little more code - you have to cleanly disconnect, close network sockets, open new ones, reconnect to peers etc. That comes with risks of its own:

you have to have enough fork peers in your peer list, or use somewhat centralized, publicly visible discovery mechanisms (DNS seeds, perhaps some onion service) which can be attacked
at least you know that the existing Bitcoin TCP port is already available on your system because you were running on it -- a new port might already be bound by another application on some systems

Despite the complications, a cleaner separation is certainly possible, and it would be worth investing some effort to mitigate risks and avoid negative impacts on both sides of the network.

HFP0 also changes the P2P protocol version (different versions used depending on POW/non-POW variant). However, it does not use the protocol version to ban peers after forking. The protocol version was initially changed with that idea in mind, but I abandoned that as a peer should be able to talk to peers with other protocol versions as freely as possible. So the HFP0 protocol versions are just informational, like the user agent string, to be able to recognize it on the network.

Better peers/banlist datafile handling

HFP0 does not clean up or write to a separate, new peers.dat file. This is an area where satoshisbitcoin was slighly better. It switched to a new peers file (forked_peers.dat) at the fork. My view is that it should have backed up the pre-fork peers.dat file under a different name, e.g. oldpeers.dat, and then carried on using peers.dat as the active peers filename.

As I didn't like the way satoshisbitcoin implemented the switchover, I've removed it in HFP0.

I think files like peers.dat and banlist.dat are de-facto well-known interfaces which should be maintained unless there is a compelling reason. These files are widely documented using their specific names / locations, and configured in backup scripts and other miscellaneous procedures developed by users of the Bitcoin software. Changing the active locations violates the principle of least surprise. Imagine if every fork abandoned them in a different way!

In this case, it shouldn't be too hard to just back them up as-is under a different name, and then keep using the old location for the active file. It's less painful to document once where you create a backup, instead of breaking all user-related documentation.

Complete test coverage

HFP0 test coverage is inadequate, which is forgivable in the case of a prototype where someone is learning the code and trying out various things, but it would be unacceptable for a serious spin-off attempt.

Here is a list of HFP0's current shortcomings w.r.t. its own additions:

MIDAS is missing unit and integration/regression tests.
The hard fork trigger tests need to check more thoroughly around the trigger height, and in re-org circumstances.
The big block aspects need an overhaul to test right up to the 4MB ceiling.
The BitPay adaptive block size feature needs an integration test, not just unit testing.

With upcoming fork developments under the BTCfork umbrella, we intend to specify changes in terms of proper requirements and better documented design, which should make it easier to implement tests and expose areas that lack verification.

Known Bugs that need fixing

HFP0 likely contains a few bugs. Two existing tests are known to fail and need to be fixed:

bip68-sequence
pruning

The first failure (bip68-sequence) might be unrelated to the fork changes, or due to code which I might not have merged carefully enough. I haven't been able to identify the exact fixes, but similar problems were observed with that test on Classic v0.12, and have since been fixed in Classic v1.1.0.

The pruning test failure looks like it might fail due to re-org problems. These would be a consequence of fork-related changes. Troubleshooting is painfully slow because of the long running time of the test.

A possible (unconfirmed) bug is the continued usage of the old max block size when creating an instance of the ValidationCostTracker in src/main.cpp:

ValidationCostTracker costTracker(OLD_MAX_BLOCK_SIGOPS, MAX_BLOCK_SIGHASH);

At the time of adapting the code, I was unsure whether the dynamically calculated block sigops value could be safely used. I've marked that as a possible bug using a TODO.

Non-urgent omissions

These are bits and pieces which are unfinished but seem less urgent.

Scaling of soft block size limit

The optional automatic soft limit scaling feature of BitPay's algorithm was disabled due to time constraints. HFP0 uses the configured soft limit if the -blockmaxsize option is provided.

Automatic scaling would be a useful feature to re-enable - it should require limited effort to implement, test and document.

As it is, miners could work around the lack of auto-scaling by adjusting their soft limits periodically.

Removal of unnecessary debug traces

I've left debug traces in the source in case someone wishes to use them to help with understanding HFP0's workings. All new traces should be guarded by #ifdef conditionals which can be selectively enabled by definitions in consensus.h.

Some of the traces were helpful in troubleshooting certain problems, even with a debugger. A lot of things can go wrong on transactions, problems in that domain were were particularly gnarly to debug, so I added some traces for that.

Better informational output

There is no informational output available via the getblockchaininfo RPC call.

It would be nice to provide an indication about how many blocks are left before the block-height fork triggers.

Resolution of remaining TODOs

If you search the code for the regular expression 'HFP0.*TODO', you will find quite a few TODO items scattered around.

Below I summarize the more important ones which haven't been mentioned above and would require resolution for a fork:

POW and non-POW variants might benefit from having their own distinct POW limits. I have tested before on a walled-off instance of testnet, and the current POW limit (which is used to reset the difficulty at fork time) worked well enough for my desktop PC to mine blocks at a reasonable 'few minutes per block' initial rate.
POW limits should be calibrated.
Minor cleanups from Bitcoin Classic's BIP109, replacing with block-height hardfork appropriate equivalents. For example, the sizeForkTime variable is a leftover which should be removed. It is still used in code path triggered by getblockchaininfo RPC call.

Code cleanups

A few possible refactorings and cleanups would be advisable.

More global variables introduced as part of the adaptive blocksize feature.
The added debugging traces could be removed.
Clean up the mixture of C/C++ style comments where HFP0 was responsible for creating a mess.

Adapt the bitcoin-tx tool in case of signature changes

This tool would need to be enhanced to optionally emit modified transaction signatures, in case replay attack prevention is implemented using signature change. Otherwise it will not be able to generate valid transaction signatures for the post-fork chain.

In the third and final part of this article, I will look in more detail at the code changes.

Feedback

If you wish to provide feedback or ask questions about HFP0, I've opened a Reddit thread at /r/btcfork to gather comments and questions.

The BTCfork community welcomes everyone who wants to enable a hard fork to join in the discussion. We meet up on various forums and chat channels to discuss and collaborate on matters related to hard forks:

on Reddit at /r/btcfork
on our Slack (sign up here)
on GitHub at https://github.com/BTCfork

Please refer to the BTCfork website for more information on how to contribute.