 |
hardfork_prototype_0 - an early prototype of a Bitcoin hard fork
Technical Details
This last part discusses aspects of the HFP0 code in more detail.
NOTE: This part is still under construction. Things may change.
Preamble
"It was a lot of fun digging into the code to figure out how to implement this fork."
- 'rocks' about satoshisbitcoin, in the Bitcoin Forum thread "Announcement - Bitcoin project to full fork to flexible blocksizes"
Before I delve into the HFP0 code, a word of thanks to rocks, who inspired me to start digging into the Bitcoin code, and the kind members at the Bitcoin Forum without whose patient explanations and words of encouragement I wouldn't have been able to get started on this journey. And what a fun journey it is!
Through it, I met a number of people who are all interested to see Bitcoin succeed, scale on-chain and fulfil the vision that Satoshi offered in the whitepaper. I'm honored to be a part of this growing Bitcoin community, together we can overcome present obstacles and bring Bitcoin to more people in this world.
Before starting on this prototype, I had never looked seriously at the Bitcoin source code. I compiled it a few times, as I usually do with most software which I want to run. Fortunately Bitcoin is free software, released under the MIT license, and no-one can really take that freedom away easily (thank you Satoshi!) As such, anyone can fork the code, at any time. It's not that difficult, so why's no-one forked Bitcoin to upgrade the block size?
The answer is: people have tried, by asking nicely. For several years. The most widely used implementation is now in the hands of developers who have decided that a block size upgrade is not part of their short-term roadmap, even though the network is running close to capacity. This is their right as maintainers of their implementation, but it's the right of people who disagree to produce the code changes they want to see and run them.
Branches in the hardfork_prototype_0 repository
The initial HFP0 code release will be placed into a separate 'snapshot' branch named 'hfp0_mXXXXXX_tYYYYYYY', where XXXXXX and YYYYYYY represent the mainnet and testnet trigger heights configured in the code.
When downloading with Git, make sure you don't select the 'develop' branch, as it only contains the historical Bitcoin code up to the commit where HFP0 development started (on March 7, 2016).
Use the hfp0_m666666_t9999999 branch instead, e.g:
$ git clone -b hfp0_m666666_t9999999 https://github.com/BTCfork/hardfork_prototype_0.git hfp0_m666666_t9999999
hardfork_prototype_0 is intended to be a repository of code snapshots instead of an active development repo. Active development will take place elsewhere from now on.
I may yet add additional snapshot branches if I decide it is helpful to demonstrate something using this prototype. In particular, I am experimenting with signature changes for replay attack prevention and cleaner network separation.
GitHub issues and pull requests for this repo may be answered, but merges will not be done in hardfork_prototype_0 repo.
If someone provides a change, I will look at it, and if I deem it useful it might make it into some future snapshot.
If anyone wants to actively develop HFP0 further, please fork the snapshot you like and maintain it yourself.
Why a single huge commit?
This development was conducted privately while I was learning about the Bitcoin code, and the full commit history is long and messy. Up to the first code release there have been 317 commits since March 7, 2016. I still maintain several unclean experimental feature branches.
It would take large effort to straighten it out into a series of nice, atomic functional commits. I'm not even sure that's always entirely possible with this set of changes, at least not while keeping things compiling between certain commits.
Therefore, instead of trying to re-arrange the commit history, I have carefully marked modifications with tagged change markers that identify the functional area(s) to which a change is associated. This has been done where possible - for some files it is not possible as they permit no comments etc.
The marker tags should make it relatively easy to understand what a change is about and to focus on a set of changes of interest for review.
I agree with those who would have preferred a clean, public development with an easily understandable commit history. I'm sorry I couldn't provide that with this first prototype. My future work on improved spin-offs will be conducted using public repositories, and thus it should be easier to understand and review changes.
Why does it change so many files?
The short answer: because this is not a minimal fork.
It includes some quite hefty external changes:
- modified scrypt POW from satoshisbitcoin
- BitPay's adaptive block size
- MIDAS per-block difficulty retargeting algorithm
- Xtreme Thinblocks from Bitcoin Unlimited
- retrofit of active soft-fork BIPs 65,68,112,113
- test framework updates from Classic v1.1.0
A minimum viable fork (MVF) up to 2MB which does not change the POW would likely be based on a newer client version of Core/XT/Classic/BU which already has the BIPs and necessary test framework updates included. It would probably only need some difficulty algorithm adjustments from the list above.
There are large amounts of HFP0 debug traces left in the provided code, which would likely be removed for a proper release.
I haven't worked out exactly how much is left if traces were to be removed, the new POW code discarded and only the minimal forking changes tallied.
It doesn't make much sense to tally that up for HFP0, because certain features are missing or not implemented to the required degree (e.g. network separation, replay attack protection) and the test coverage is incomplete.
However, it is obvious that a viable non-minimal spin-off client will have substantial code changes.
Meaning of the change markers
The following table briefly explains the meaning of the changer marker tags:
| Tag | Meaning |
|---|---|
REN |
cosmetic renames (in docs, titles, output strings, GUI menus etc.) |
FRK |
forking actions at triggering height |
DIF |
difficulty algorithm update (MIDAS) |
BSZ |
Adaptive block size algorithm based on BitPay implementation |
SED |
DNS seeds and static IPs (emptied out / dummy values) |
PER |
Peer handling during fork (rudimentary) |
CLI |
Client version update |
ALR |
Alert key disabling |
POW |
fork to new POW code (overriden by config switch --disable-newpow) |
XTB |
Xtreme Thinblocks implementation |
DBG |
fork related debugging traces (removable) |
CLN |
Cleanup related to removal of obsolete code |
TST |
Additional tests (unit, system) or fixes to tests |
CLT |
BIP65 OP_CHECKLOCKTIMEVERIFY |
RLT |
BIP68 Relative Lock-time using consensus-enforced sequence number |
CSV |
BIP112 CHECKSEQUENCEVERIFY |
MTP |
BIP113 Median time-past as endpoint for lock-time calculations |
TMP |
temporary settings for testing only |
CRY |
Cherry-picked miscellaneous fixes from other clients which otherwise trouble testing |
Overview of file changes
List of added files (w.r.t. Classic v0.12.0cl1)
| File | Comment |
|---|---|
qa/rpc-tests/bip68-112-113-p2p.py |
BIP tests from recent upstream |
qa/rpc-tests/bip68-sequence.py |
BIP tests from recent upstream |
qa/rpc-tests/hardfork_bigblocks.py |
hardfork + big block test |
qa/rpc-tests/hardfork_minebigblock.py |
hardfork + big block test |
src/blocksizecalculator.cpp |
adaptive block size (BitPay), modified |
src/blocksizecalculator.h |
adaptive block size (BitPay), modified |
src/crypto/modified_scrypt_sha256.cpp |
modified scrypt (new POW) |
src/crypto/modified_scrypt_sha256.h |
modified scrypt (new POW) |
src/crypto/modified_scrypt_smix.cpp |
modified scrypt (new POW) |
src/crypto/modified_scrypt_smix.h |
modified scrypt (new POW) |
src/crypto/sysendian.h |
modified scrypt (new POW) |
src/test/blocksizecalculator_tests.cpp |
C++ unit test for BSZ |
src/test/thinblock_tests.cpp |
C++ unit tests for Xtreme Thinblocks |
src/thinblock.cpp |
sources related to Xtreme Thinblocks |
src/thinblock.h |
sources related to Xtreme Thinblocks |
src/xthinblocks.cpp |
sources related to Xtreme Thinblocks |
src/xthinblocks.h |
sources related to Xtreme Thinblocks |
List of removed/replaced files
| File | Comment |
|---|---|
qa/rpc-tests/bigblocks.py |
replaced by hardfork_bigblocks.py |
qa/rpc-tests/forknotify.py |
removed as part of unknown-version block check removal |
List of modified files
| File | Overview of significant changes |
|---|---|
README.md |
updated to give overview of HFP0 |
configure.ac |
client rename and support for new --disable-newpow configuration parameter |
contrib/gitian-descriptors/gitian-linux.yml |
change remote URL (dummy) |
contrib/gitian-descriptors/gitian-osx-signer.yml |
diskimages permission fix from upstream |
contrib/seeds/nodes_main.txt |
remove public seeds, add private seeds |
contrib/seeds/nodes_test.txt |
remove public seeds, add private seeds |
qa/pull-tester/rpc-tests.py |
add new tests, remove forknotify.py and rename bigblocks.py |
qa/rpc-tests/maxuploadtarget.py |
updated as per BU0.12.1bu to use maxuploadtarget of 200MB |
qa/rpc-tests/p2p-acceptblock.py |
set correct block version after fork |
qa/rpc-tests/p2p-fullblocktest.py |
add reminder that this test might need updating for > 1MB blocks |
qa/rpc-tests/pruning.py |
add scaleblocksizeoptions=0 |
qa/rpc-tests/sendheaders.py |
set correct block version after fork, add XTB changes |
qa/rpc-tests/test_framework/blockstore.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/blocktools.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/comptool.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/mininode.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/netutil.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/script.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/socks5.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/test_framework.py |
merged test_framework update from Classic v1.1.0 |
qa/rpc-tests/test_framework/util.py |
merged test_framework update from Classic v1.1.0, read in fork height triggers from C files |
src/base58.cpp |
fix backport (cleanse full vChTemp vector) |
src/bitcoin-cli.cpp |
renaming only |
src/bitcoind.cpp |
shutdown improvements for new POW (from satoshisbitcoin), renaming |
src/bitcoin-tx.cpp |
renaming, added TODO about signature capability once replay attack protection is implemented |
src/chainparams.cpp |
split powLimit into powLimitHistoric and powLimitResetAtFork, add nHFP0ActivateSizeForkHeight and MIDAS parameters, dummy DNS seed, removed Classic parameters |
src/chainparamsseeds.h |
replaced public with private seeds (generated from contrib/seeds/nodes_*.txt) |
src/clientversion.cpp |
renamings, make scrypt/sha256 variant visible in client name |
src/consensus/consensus.h |
add conditionals (HFP0_POW, HFP0_DEBUG_* flags), new MAX_BLOCK_SIZE (2MB), add fork height constants and new POW limits for mainnet/testnet/regtestnet, add static variables for adaptive block size (BSZ), remove Classic constants |
src/consensus/params.h |
factor some MIDAS parameters out into here, add powLimitHistoric and powLimitResetAtFork, remove Classic-related majority/expiration/grace period params and access functions, add nHFP0ActivateSizeForkHeight variable and access function |
src/hash.cpp |
merge HashModifiedScrypt() from satoshisbitcoin |
src/hash.h |
merge HashModifiedScrypt() from satoshisbitcoin |
src/init.cpp |
disable alert system, add XTB debug=thin argument, add BSZ scaleblocksize argument, renamings, BSZ update in CreateNewBlock |
src/main.cpp |
activate BSZ after fork trigger, XTB merges, BSZ initialize global variables, added lots of debug traces in CheckTransaction, BIP merges, replace MAX_STANDARD_TX_SIGOPS with BSZ dynamic maxStandardTxSigops variable, pass fork POW limit to CheckProofOfWork if fork blocks, replace Classic's DidBlockTriggerSizeFork() with DidBlockTriggerHFP0SizeFork(), replace nMaxLegacySigops by maxBlockSigops, ban peers which serve non-fork blocks after trigger height, merge Core PR#7919 bugfix, remove some obsolete Classic bits |
src/main.h |
parameterize MaxBlockSize() and other functions with block height, add UpdateAdaptiveBlockSizeVars() to factor out BSZ updates. Some XTB, ALR, BIP changes. |
src/Makefile.am |
add new files for POW, BSZ, XTB to compilation (see added files) |
src/Makefile.test.include |
add new files for BSZ, XTB |
src/merkleblock.cpp |
BSZ: replace MAX_BLOCK_SIZE constant with maxBlockSize |
src/miner.cpp |
FRK, BSZ and POW-related changes |
src/miner.h |
some shutdown-related improvements from satoshisbitcoin |
src/net.cpp |
Mostly XTB, use BSZ dynamic size instead of MAX_BLOCK_SIZE |
src/net.h |
XTB merge |
src/policy/policy.cpp |
merge 4fea8e741ba38b2c1c4fb1c79962234f711c846e (use MAX_STANDARD_VERSION for tx version check) |
src/policy/policy.h |
bump DEFAULT_MAX_BLOCK_SIZE to 1MB, BSZ: add DEFAULT_SCALE_BLOCK_SIZE_OPTIONS, BIPs: add STANDARD_LOCKTIME_VERIFY_FLAGS |
src/pow.cpp |
add MIDAS |
src/pow.h |
add MIDAS |
src/primitives/block.cpp |
POW: add cache of past modified scrypt hashes |
src/primitives/block.h |
add FULL_FORK_VERSION_* constants, removed Classic DEFAULT_2MB_VOTE |
src/primitives/transaction.cpp |
BIP68 merge |
src/primitives/transaction.h |
BIP68 and 4fea8e741ba38b2c1c4fb1c79962234f711c846e merge |
src/protocol.cpp |
XTB merge |
src/protocol.h |
XTB merge |
src/qt/askpassphrasedialog.cpp |
renaming only |
src/qt/bitcoin.cpp |
renaming only |
src/qt/bitcoingui.cpp |
renaming only |
src/qt/bitcoinstrings.cpp |
renaming only |
src/qt/guiutil.cpp |
XTB add XTHIN to services |
src/qt/intro.cpp |
renaming only |
src/qt/rpcconsole.cpp |
renaming only |
src/qt/splashscreen.cpp |
renaming only |
src/qt/splashscreen.h |
renaming only |
src/qt/utilitydialog.cpp |
renaming only |
src/rpcblockchain.cpp |
only TODOs for more HF descriptive info |
src/rpcmining.cpp |
FRK, DIF, BSZ changes |
src/rpcnet.cpp |
renamings, XTB thinblockstats added |
src/script/interpreter.cpp |
BIP112 merge |
src/script/interpreter.h |
BIP112 merge |
src/script/script_error.h |
BIP112 merge |
src/script/script.h |
BIP112 merge |
src/sync.cpp |
XTB, Classic bugfix backport (disabled onlyMaybeDeadlock assert due to too many false positives) |
src/test/alert_tests.cpp |
only harmonize unit test name with filename |
src/test/block_size_tests.cpp |
block size tests (incl. around fork height) |
src/test/checkblock_tests.cpp |
only harmonize unit test name with filename |
src/test/data/tx_invalid.json |
BIP112 merge |
src/test/data/tx_valid.json |
BIP112 merge |
src/test/miner_tests.cpp |
complex inherited BIP68 test changes. File marked as changed by BIP68, individual changes not marked separately. |
src/test/prevector_tests.cpp |
only harmonize unit test name with filename |
src/test/script_tests.cpp |
BIP68 merge |
src/test/test_bitcoin.cpp |
reset BSZ global vars between tests, BIP112 merge |
src/test/test_bitcoin.h |
BIP112 merge |
src/test/transaction_tests.cpp |
BIP112 merge |
src/test/txvalidationcache_tests.cpp |
only harmonize unit test name with filename |
src/timedata.cpp |
renaming only |
src/txdb.cpp |
DIF: use appropriate POW limit |
src/txdb.h |
XTB merge |
src/txmempool.cpp |
BIP68, BIP112 merge |
src/txmempool.h |
BIP112 merge |
src/univalue/include/univalue.h |
XTB merge |
src/version.h |
split PROTOCOL_VERSION into POW_PROTOCOL_VERSION and SHA_PROTOCOL_VERSION |
List of modified files without recognizable change markers
The following is a list of files which didn't permit adding formal change markers.
| File | Comment |
|---|---|
contrib/gitian-descriptors/gitian-linux.yml |
YAML data, change is simply a deflected URL |
src/test/data/tx_invalid.json |
JSON data, merge from CSV BIP |
src/test/data/tx_valid.json |
JSON data, merge from CVS BIP |
contrib/seeds/nodes_main.txt |
list of addresses processed by another script |
contrib/seeds/nodes_test.txt |
list of addresses processed by another script |
There are also some Python files in qa/rpc-tests/test_framework which have been
updated based on the Classic v1.1.0 code. In these files, not every Classic-related
change is marked - the files are marked generally with a single-line TST marker comment
underneath the file header:
# HFP0 TST: merged test_framework update from Classic v1.1.0
Building
Build instructions for Classic 0.12.0 should work fine. Platforms I've compiled and tested on:
Debian 7 (x86_64 and i686), gcc 4.7.2
Debian 8 (x86_64), gcc 4.9.2
HFP0 adds a new configuration option: --disable-newpow .
Unless you specify this option, the client is built to switch to the new POW at the trigger height.
If you specify --disable-newpow, the fork keeps using Bitcoin's existing SHA256 after it triggers.
Test Status
What follows is a brief overview - I might update this with more detail later on.
I have tested this prototype only in an isolated test environment (using up to 4 VMs). Most of my testing was done using regtestnet, the regression test chain. It has NOT been tested on the real testnet or on mainnet.
The C++ unit tests (src/test/test_bitcoin) should all pass.
You will see a message on standard output:
CreateAndProcessBlock: ProcessNewBlock, block not accepted
I added this message as a troubleshooting aid to see when a block was not accepted. It does not signify a test failure, and can be ignored.
Regression/integration tests can be run with qa/pull-tester/rpc-tests.py .
The tests bip68-sequence.py and pruning.py currently fail.
For bip68-sequence, it is presumed to be a policy code issue
that has been fixed in later versions of other clients (a similar issue
was present on Classic 0.12 builds, but has been fixed in Classic 1.1).
I have not identified the exact change that could fix it, but the
test failure is likely unrelated to changes made by HFP0.
The pruning test failure might be due to a presumed bug in how the fork handles re-orgs across its trigger, but it could also be something else.
Rudimentary hardfork / bigger block QA tests are implemented, but they are lacking in some test cases (re-orgs across the fork trigger height, and fine-grained testing around the trigger height). They also specify a soft block limit of 2MB, and don't exercise the theoretical maximum of even that soft limit. A more complete test should try to test up to MAX_BLOCK_SIZE without a soft block limit. This involves a substantial overhaul of the big block tests.
Adaptive block size tests are limited to the blocksizecalculator_tests
suite, and would need to be extended with more test cases and integration
tests.
There are no unit tests / regression tests for MIDAS yet.
I welcome any feedback from others who are able try out this prototype in their own test environments or on the actual testnet.
Further development
I will not develop this HFP0 prototype further to production readiness.
I consider it more important that Minimum Viable Forks are produced, with solid test coverage and requirements input from the wider community.
The BTCfork community (links below) is doing just that, working with users, developers and businesses in the Bitcoin space to develop improved forks, starting from MVFs and later hopefully supporting the development of fuller-featured fork candidates. I shall be working with them to bring viable forks to maturity.
If you want to get involved, you are welcome to get in touch on any of the channels below.
- Slack: sign up here: https://btcforks.signup.team/
- Reddit: https://www.reddit.com/r/btcfork/
- GitHub: https://github.com/BTCfork
- ConsiderIt (issue voting): https://btcfork.consider.it/
- Twitter: https://twitter.com/btcfork
Feedback on the code
If you wish to provide feedback on the code, feel free to use GitHub's comment features or raise issues.
I won't fix issues directly in the hardfork_prototype_0 repository,
but I may include fixes in future code snapshot, if any take place on this repo,
and I'll try to respond to any sensible comments there.
Alternatively, I've opened a Reddit thread in /r/btcfork to gather comments and questions.