Space shuttle cockpit instruments
photo of space shuttle cockpit instruments

hardfork_prototype_0 - an early prototype of a Bitcoin hard fork

Technical Details

This last part discusses aspects of the HFP0 code in more detail.

NOTE: This part is still under construction. Things may change.

Preamble

"It was a lot of fun digging into the code to figure out how to implement this fork."

- 'rocks' about satoshisbitcoin, in the Bitcoin Forum thread "Announcement - Bitcoin project to full fork to flexible blocksizes"

Before I delve into the HFP0 code, a word of thanks to rocks, who inspired me to start digging into the Bitcoin code, and the kind members at the Bitcoin Forum without whose patient explanations and words of encouragement I wouldn't have been able to get started on this journey. And what a fun journey it is!

Through it, I met a number of people who are all interested to see Bitcoin succeed, scale on-chain and fulfil the vision that Satoshi offered in the whitepaper. I'm honored to be a part of this growing Bitcoin community, together we can overcome present obstacles and bring Bitcoin to more people in this world.

Before starting on this prototype, I had never looked seriously at the Bitcoin source code. I compiled it a few times, as I usually do with most software which I want to run. Fortunately Bitcoin is free software, released under the MIT license, and no-one can really take that freedom away easily (thank you Satoshi!) As such, anyone can fork the code, at any time. It's not that difficult, so why's no-one forked Bitcoin to upgrade the block size?

The answer is: people have tried, by asking nicely. For several years. The most widely used implementation is now in the hands of developers who have decided that a block size upgrade is not part of their short-term roadmap, even though the network is running close to capacity. This is their right as maintainers of their implementation, but it's the right of people who disagree to produce the code changes they want to see and run them.

Branches in the hardfork_prototype_0 repository

The initial HFP0 code release will be placed into a separate 'snapshot' branch named 'hfp0_mXXXXXX_tYYYYYYY', where XXXXXX and YYYYYYY represent the mainnet and testnet trigger heights configured in the code.

When downloading with Git, make sure you don't select the 'develop' branch, as it only contains the historical Bitcoin code up to the commit where HFP0 development started (on March 7, 2016). Use the hfp0_m666666_t9999999 branch instead, e.g:

$ git clone -b hfp0_m666666_t9999999 https://github.com/BTCfork/hardfork_prototype_0.git hfp0_m666666_t9999999

hardfork_prototype_0 is intended to be a repository of code snapshots instead of an active development repo. Active development will take place elsewhere from now on.

I may yet add additional snapshot branches if I decide it is helpful to demonstrate something using this prototype. In particular, I am experimenting with signature changes for replay attack prevention and cleaner network separation.

GitHub issues and pull requests for this repo may be answered, but merges will not be done in hardfork_prototype_0 repo.

If someone provides a change, I will look at it, and if I deem it useful it might make it into some future snapshot.

If anyone wants to actively develop HFP0 further, please fork the snapshot you like and maintain it yourself.

Why a single huge commit?

This development was conducted privately while I was learning about the Bitcoin code, and the full commit history is long and messy. Up to the first code release there have been 317 commits since March 7, 2016. I still maintain several unclean experimental feature branches.

It would take large effort to straighten it out into a series of nice, atomic functional commits. I'm not even sure that's always entirely possible with this set of changes, at least not while keeping things compiling between certain commits.

Therefore, instead of trying to re-arrange the commit history, I have carefully marked modifications with tagged change markers that identify the functional area(s) to which a change is associated. This has been done where possible - for some files it is not possible as they permit no comments etc.

The marker tags should make it relatively easy to understand what a change is about and to focus on a set of changes of interest for review.

I agree with those who would have preferred a clean, public development with an easily understandable commit history. I'm sorry I couldn't provide that with this first prototype. My future work on improved spin-offs will be conducted using public repositories, and thus it should be easier to understand and review changes.

Why does it change so many files?

The short answer: because this is not a minimal fork.

  1. It includes some quite hefty external changes:

    • modified scrypt POW from satoshisbitcoin
    • BitPay's adaptive block size
    • MIDAS per-block difficulty retargeting algorithm
    • Xtreme Thinblocks from Bitcoin Unlimited
    • retrofit of active soft-fork BIPs 65,68,112,113
    • test framework updates from Classic v1.1.0

    A minimum viable fork (MVF) up to 2MB which does not change the POW would likely be based on a newer client version of Core/XT/Classic/BU which already has the BIPs and necessary test framework updates included. It would probably only need some difficulty algorithm adjustments from the list above.

  2. There are large amounts of HFP0 debug traces left in the provided code, which would likely be removed for a proper release.

I haven't worked out exactly how much is left if traces were to be removed, the new POW code discarded and only the minimal forking changes tallied.

It doesn't make much sense to tally that up for HFP0, because certain features are missing or not implemented to the required degree (e.g. network separation, replay attack protection) and the test coverage is incomplete.

However, it is obvious that a viable non-minimal spin-off client will have substantial code changes.

Meaning of the change markers

The following table briefly explains the meaning of the changer marker tags:

Tag Meaning
REN cosmetic renames (in docs, titles, output strings, GUI menus etc.)
FRK forking actions at triggering height
DIF difficulty algorithm update (MIDAS)
BSZ Adaptive block size algorithm based on BitPay implementation
SED DNS seeds and static IPs (emptied out / dummy values)
PER Peer handling during fork (rudimentary)
CLI Client version update
ALR Alert key disabling
POW fork to new POW code (overriden by config switch --disable-newpow)
XTB Xtreme Thinblocks implementation
DBG fork related debugging traces (removable)
CLN Cleanup related to removal of obsolete code
TST Additional tests (unit, system) or fixes to tests
CLT BIP65 OP_CHECKLOCKTIMEVERIFY
RLT BIP68 Relative Lock-time using consensus-enforced sequence number
CSV BIP112 CHECKSEQUENCEVERIFY
MTP BIP113 Median time-past as endpoint for lock-time calculations
TMP temporary settings for testing only
CRY Cherry-picked miscellaneous fixes from other clients which otherwise trouble testing

Overview of file changes

List of added files (w.r.t. Classic v0.12.0cl1)

File Comment
qa/rpc-tests/bip68-112-113-p2p.py BIP tests from recent upstream
qa/rpc-tests/bip68-sequence.py BIP tests from recent upstream
qa/rpc-tests/hardfork_bigblocks.py hardfork + big block test
qa/rpc-tests/hardfork_minebigblock.py hardfork + big block test
src/blocksizecalculator.cpp adaptive block size (BitPay), modified
src/blocksizecalculator.h adaptive block size (BitPay), modified
src/crypto/modified_scrypt_sha256.cpp modified scrypt (new POW)
src/crypto/modified_scrypt_sha256.h modified scrypt (new POW)
src/crypto/modified_scrypt_smix.cpp modified scrypt (new POW)
src/crypto/modified_scrypt_smix.h modified scrypt (new POW)
src/crypto/sysendian.h modified scrypt (new POW)
src/test/blocksizecalculator_tests.cpp C++ unit test for BSZ
src/test/thinblock_tests.cpp C++ unit tests for Xtreme Thinblocks
src/thinblock.cpp sources related to Xtreme Thinblocks
src/thinblock.h sources related to Xtreme Thinblocks
src/xthinblocks.cpp sources related to Xtreme Thinblocks
src/xthinblocks.h sources related to Xtreme Thinblocks

List of removed/replaced files

File Comment
qa/rpc-tests/bigblocks.py replaced by hardfork_bigblocks.py
qa/rpc-tests/forknotify.py removed as part of unknown-version block check removal

List of modified files

File Overview of significant changes
README.md updated to give overview of HFP0
configure.ac client rename and support for new --disable-newpow configuration parameter
contrib/gitian-descriptors/gitian-linux.yml change remote URL (dummy)
contrib/gitian-descriptors/gitian-osx-signer.yml diskimages permission fix from upstream
contrib/seeds/nodes_main.txt remove public seeds, add private seeds
contrib/seeds/nodes_test.txt remove public seeds, add private seeds
qa/pull-tester/rpc-tests.py add new tests, remove forknotify.py and rename bigblocks.py
qa/rpc-tests/maxuploadtarget.py updated as per BU0.12.1bu to use maxuploadtarget of 200MB
qa/rpc-tests/p2p-acceptblock.py set correct block version after fork
qa/rpc-tests/p2p-fullblocktest.py add reminder that this test might need updating for > 1MB blocks
qa/rpc-tests/pruning.py add scaleblocksizeoptions=0
qa/rpc-tests/sendheaders.py set correct block version after fork, add XTB changes
qa/rpc-tests/test_framework/blockstore.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/blocktools.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/comptool.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/mininode.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/netutil.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/script.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/socks5.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/test_framework.py merged test_framework update from Classic v1.1.0
qa/rpc-tests/test_framework/util.py merged test_framework update from Classic v1.1.0, read in fork height triggers from C files
src/base58.cpp fix backport (cleanse full vChTemp vector)
src/bitcoin-cli.cpp renaming only
src/bitcoind.cpp shutdown improvements for new POW (from satoshisbitcoin), renaming
src/bitcoin-tx.cpp renaming, added TODO about signature capability once replay attack protection is implemented
src/chainparams.cpp split powLimit into powLimitHistoric and powLimitResetAtFork, add nHFP0ActivateSizeForkHeight and MIDAS parameters, dummy DNS seed, removed Classic parameters
src/chainparamsseeds.h replaced public with private seeds (generated from contrib/seeds/nodes_*.txt)
src/clientversion.cpp renamings, make scrypt/sha256 variant visible in client name
src/consensus/consensus.h add conditionals (HFP0_POW, HFP0_DEBUG_* flags), new MAX_BLOCK_SIZE (2MB), add fork height constants and new POW limits for mainnet/testnet/regtestnet, add static variables for adaptive block size (BSZ), remove Classic constants
src/consensus/params.h factor some MIDAS parameters out into here, add powLimitHistoric and powLimitResetAtFork, remove Classic-related majority/expiration/grace period params and access functions, add nHFP0ActivateSizeForkHeight variable and access function
src/hash.cpp merge HashModifiedScrypt() from satoshisbitcoin
src/hash.h merge HashModifiedScrypt() from satoshisbitcoin
src/init.cpp disable alert system, add XTB debug=thin argument, add BSZ scaleblocksize argument, renamings, BSZ update in CreateNewBlock
src/main.cpp activate BSZ after fork trigger, XTB merges, BSZ initialize global variables, added lots of debug traces in CheckTransaction, BIP merges, replace MAX_STANDARD_TX_SIGOPS with BSZ dynamic maxStandardTxSigops variable, pass fork POW limit to CheckProofOfWork if fork blocks, replace Classic's DidBlockTriggerSizeFork() with DidBlockTriggerHFP0SizeFork(), replace nMaxLegacySigops by maxBlockSigops, ban peers which serve non-fork blocks after trigger height, merge Core PR#7919 bugfix, remove some obsolete Classic bits
src/main.h parameterize MaxBlockSize() and other functions with block height, add UpdateAdaptiveBlockSizeVars() to factor out BSZ updates. Some XTB, ALR, BIP changes.
src/Makefile.am add new files for POW, BSZ, XTB to compilation (see added files)
src/Makefile.test.include add new files for BSZ, XTB
src/merkleblock.cpp BSZ: replace MAX_BLOCK_SIZE constant with maxBlockSize
src/miner.cpp FRK, BSZ and POW-related changes
src/miner.h some shutdown-related improvements from satoshisbitcoin
src/net.cpp Mostly XTB, use BSZ dynamic size instead of MAX_BLOCK_SIZE
src/net.h XTB merge
src/policy/policy.cpp merge 4fea8e741ba38b2c1c4fb1c79962234f711c846e (use MAX_STANDARD_VERSION for tx version check)
src/policy/policy.h bump DEFAULT_MAX_BLOCK_SIZE to 1MB, BSZ: add DEFAULT_SCALE_BLOCK_SIZE_OPTIONS, BIPs: add STANDARD_LOCKTIME_VERIFY_FLAGS
src/pow.cpp add MIDAS
src/pow.h add MIDAS
src/primitives/block.cpp POW: add cache of past modified scrypt hashes
src/primitives/block.h add FULL_FORK_VERSION_* constants, removed Classic DEFAULT_2MB_VOTE
src/primitives/transaction.cpp BIP68 merge
src/primitives/transaction.h BIP68 and 4fea8e741ba38b2c1c4fb1c79962234f711c846e merge
src/protocol.cpp XTB merge
src/protocol.h XTB merge
src/qt/askpassphrasedialog.cpp renaming only
src/qt/bitcoin.cpp renaming only
src/qt/bitcoingui.cpp renaming only
src/qt/bitcoinstrings.cpp renaming only
src/qt/guiutil.cpp XTB add XTHIN to services
src/qt/intro.cpp renaming only
src/qt/rpcconsole.cpp renaming only
src/qt/splashscreen.cpp renaming only
src/qt/splashscreen.h renaming only
src/qt/utilitydialog.cpp renaming only
src/rpcblockchain.cpp only TODOs for more HF descriptive info
src/rpcmining.cpp FRK, DIF, BSZ changes
src/rpcnet.cpp renamings, XTB thinblockstats added
src/script/interpreter.cpp BIP112 merge
src/script/interpreter.h BIP112 merge
src/script/script_error.h BIP112 merge
src/script/script.h BIP112 merge
src/sync.cpp XTB, Classic bugfix backport (disabled onlyMaybeDeadlock assert due to too many false positives)
src/test/alert_tests.cpp only harmonize unit test name with filename
src/test/block_size_tests.cpp block size tests (incl. around fork height)
src/test/checkblock_tests.cpp only harmonize unit test name with filename
src/test/data/tx_invalid.json BIP112 merge
src/test/data/tx_valid.json BIP112 merge
src/test/miner_tests.cpp complex inherited BIP68 test changes. File marked as changed by BIP68, individual changes not marked separately.
src/test/prevector_tests.cpp only harmonize unit test name with filename
src/test/script_tests.cpp BIP68 merge
src/test/test_bitcoin.cpp reset BSZ global vars between tests, BIP112 merge
src/test/test_bitcoin.h BIP112 merge
src/test/transaction_tests.cpp BIP112 merge
src/test/txvalidationcache_tests.cpp only harmonize unit test name with filename
src/timedata.cpp renaming only
src/txdb.cpp DIF: use appropriate POW limit
src/txdb.h XTB merge
src/txmempool.cpp BIP68, BIP112 merge
src/txmempool.h BIP112 merge
src/univalue/include/univalue.h XTB merge
src/version.h split PROTOCOL_VERSION into POW_PROTOCOL_VERSION and SHA_PROTOCOL_VERSION

List of modified files without recognizable change markers

The following is a list of files which didn't permit adding formal change markers.

File Comment
contrib/gitian-descriptors/gitian-linux.yml YAML data, change is simply a deflected URL
src/test/data/tx_invalid.json JSON data, merge from CSV BIP
src/test/data/tx_valid.json JSON data, merge from CVS BIP
contrib/seeds/nodes_main.txt list of addresses processed by another script
contrib/seeds/nodes_test.txt list of addresses processed by another script

There are also some Python files in qa/rpc-tests/test_framework which have been updated based on the Classic v1.1.0 code. In these files, not every Classic-related change is marked - the files are marked generally with a single-line TST marker comment underneath the file header:

# HFP0 TST: merged test_framework update from Classic v1.1.0

Building

Build instructions for Classic 0.12.0 should work fine. Platforms I've compiled and tested on:

  • Debian 7 (x86_64 and i686), gcc 4.7.2

  • Debian 8 (x86_64), gcc 4.9.2

HFP0 adds a new configuration option: --disable-newpow .

Unless you specify this option, the client is built to switch to the new POW at the trigger height.

If you specify --disable-newpow, the fork keeps using Bitcoin's existing SHA256 after it triggers.

Test Status

What follows is a brief overview - I might update this with more detail later on.

I have tested this prototype only in an isolated test environment (using up to 4 VMs). Most of my testing was done using regtestnet, the regression test chain. It has NOT been tested on the real testnet or on mainnet.

The C++ unit tests (src/test/test_bitcoin) should all pass. You will see a message on standard output:

CreateAndProcessBlock: ProcessNewBlock, block not accepted

I added this message as a troubleshooting aid to see when a block was not accepted. It does not signify a test failure, and can be ignored.

Regression/integration tests can be run with qa/pull-tester/rpc-tests.py .

The tests bip68-sequence.py and pruning.py currently fail. For bip68-sequence, it is presumed to be a policy code issue that has been fixed in later versions of other clients (a similar issue was present on Classic 0.12 builds, but has been fixed in Classic 1.1). I have not identified the exact change that could fix it, but the test failure is likely unrelated to changes made by HFP0.

The pruning test failure might be due to a presumed bug in how the fork handles re-orgs across its trigger, but it could also be something else.

Rudimentary hardfork / bigger block QA tests are implemented, but they are lacking in some test cases (re-orgs across the fork trigger height, and fine-grained testing around the trigger height). They also specify a soft block limit of 2MB, and don't exercise the theoretical maximum of even that soft limit. A more complete test should try to test up to MAX_BLOCK_SIZE without a soft block limit. This involves a substantial overhaul of the big block tests.

Adaptive block size tests are limited to the blocksizecalculator_tests suite, and would need to be extended with more test cases and integration tests.

There are no unit tests / regression tests for MIDAS yet.

I welcome any feedback from others who are able try out this prototype in their own test environments or on the actual testnet.

Further development

I will not develop this HFP0 prototype further to production readiness.

I consider it more important that Minimum Viable Forks are produced, with solid test coverage and requirements input from the wider community.

The BTCfork community (links below) is doing just that, working with users, developers and businesses in the Bitcoin space to develop improved forks, starting from MVFs and later hopefully supporting the development of fuller-featured fork candidates. I shall be working with them to bring viable forks to maturity.

If you want to get involved, you are welcome to get in touch on any of the channels below.


Feedback on the code

If you wish to provide feedback on the code, feel free to use GitHub's comment features or raise issues.

I won't fix issues directly in the hardfork_prototype_0 repository, but I may include fixes in future code snapshot, if any take place on this repo, and I'll try to respond to any sensible comments there.

Alternatively, I've opened a Reddit thread in /r/btcfork to gather comments and questions.