Weekly musing #5 - Testing, Testing, ...
 |
Bitcoin Unlimited
I've been testing and raising issues on Github for some failing
tests that we are troubleshooting. (I guess it started
with the failing pruning.py, but we encountered more
during recent updates meant to close some post-1.0.1.4 gaps.
Among them:
I tested another one with sickpig (validateblocktemplate).
Most if not all these test failures were apparently random failures, some involving infinite loops.
Random failures can slip through Travis CI undetected, only to surface later. This seems to have already happened, based on subsequent investigations.
This is a prime reason to run the full test suite (including extended tests) multiple times a day, on as many platforms as possible. Fortunately BU's Travis account has been upgraded to a paid one, at least this means the project will be able to run more tests.
Running tests repeatedly
Our parallel testing wrapper for unit tests has an -r option for
repeating tests, but our regression test wrapper (rpc-tests.py) currently
does not. This is something I'd like to enhance, it is frequently needed
because a lot of failures are not deterministic.
One can of course always write a little shell script. I used this one which runs the test repeatedly using a timeout per run, logs failures to a temporary file and prints out the total number of failures at the bottom.
#!/bin/bash
TIMEOUT=100 # seconds, need to adapt to specific test
TEST_COMMAND="qa/pull-tester/rpc-tests.py abandonconflict"
for i in {1..100}
do
echo "`date`: start iteration $i"
timeout --foreground $TIMEOUT $TEST_COMMAND || echo "this is a failure"
killall bitcoind
done | tee /tmp/testfailures.log | grep -c "this is a failure"
Messy, but it works.
Hopefully, an improved rpc-tests.py will do this for us in future, and spit out more statistics e.g. average time per run and maybe the standard deviation of that.
Bisections
Once you know a test is failing randomly on a certain revision, you need to find out what introduced the problem.
If you cannot tell easily, then you can usually get some idea by testing backwards from a known bad commit (usually the head of the branch you are working off) to find the first commit that broke the test, and see what it changed.
If the source of the problem possibly lies back a long time, going back over each revision takes a long time. You can try doing a bisection which is much quicker (but you need to know a working revision sometime in the past).
I submitted a PR for a helper script that can be used in conjunction
with git bisect to run a bisection on a test that is failing
randomly and/or getting stuck in a loop.
I committed the original version
of this script in my personal repo under GPLv3, but I submitted it
in improved form in PR532.
That PR introduces an additional documentation file with some tips in doc/bug-triage.md.
The script has so far proved quite useful, but it does have limitations, for example when the build breaks in some past commits. Be sure to read the file header of the script and the mentioned document if you plan to use it.
The main benefit is that this script can triage randomly failing and looping tests. It can be used in "one-shot" mode without running a bisection, of course, just to either
- run a test command repeatedly (set a huge timeout if you know the test does not time out)
- catch a test which times out (although you can also use the timeout(1) command in Linux)
parity-bitcoin
I've never dabbled in Rust before, so I was surprised that it was not too difficult to get to a running version of parity-bitcoin. Fortunately I did have a little help though from Christian Nyumbayire (Chritchens on the forum and Reddit). He is working on an adaptation of Parity that he called 'parity-unlimited' and is currently taking shape. Initial steps will hopefully demonstrate that parity can handle a hardcoded bigger block size (e.g. 2MB) well.
Here is his announcement thread, in which he also left valuable information on how to install Parity (some information on installing dependencies got me past initial hurdles).
https://bitco.in/forum/threads/multi-implementation-client-development.2092/
I was able to successfully run the unmodified parity-bitcoin unit test suite (one test was ignored but all others passed). Unfortunately, my initial attempt to run the regression test suite failed - it got stuck with bitcoinj receiving an empty block header.
Whether this was a problem with my build (I did not build the 'release' build) or still a bug in the regression tests is something that future trials will show.
It is already promising that it got very far in the regression tests. Hopefully it can soon become a big-block capable neighbor of Bitcoin Unlimited on the actual network.
BTCfork
I managed to resolve the little issues that were troubling Travis, and so complete the merge of recent upstream BU 'dev' into MVF-BU.
Two long running tests (excessive.py and parallel.py) had to be moved to the 'extended' set of tests which are not automatically run. This is because BTCfork still operates on a free Travis instance with a cutoff time of 50 minutes.
The fork code introduces some long-running tests of its own, and these have to be tested more frequently. However, when touching code that modifies block handling, it will be necessary to manually run the relocated tests, and occasionally swap them into Travis for testing across different build platforms (which I don't have).
I will be starting work on a checkpoint feature next, but more details about that once it is ready.
Bitcoin-ABC
BU contributor deadalnix created a repository of a Bitcoin Core 0.14 client which has the SegWit and RBF removed.
This can serve as a basis for future Adjustable Blocksize Cap modification, hence he called it: Bitcoin-ABC!
I have started assisting on this effort by helping to resolve some failing regression tests. Most of the issues appear due to incomplete removal of SegWit test code, and should not take too long to clean up.
It would be exciting to have a recent Core with a configurable limit! Any volunteers are more than welcome to assist this effort.
Footnotes
[1] Fixings Problems comic thanks to https://xkcd.com/1739/ (CC BY-NC 2.5)