Ava Chow

Building and Syncing Bitcoin 0.5.0 in 2022

Tue, 29 Mar 2022 17:42:00 -0400

Building and Syncing Bitcoin 0.5.0 in 2022

A couple of weeks ago, Jameson Lopp came to me with a question: Could I figure out how to build old versions of Bitcoin and give him instructions for doing so? This intrigued me, and so I obliged. Thankfully, the task was not particularly difficult as the Gitian build system was in use, and it seemed to work. So I built down to 0.3.20, put the instructions into a repository on GitHub, and called it a day.

A week later, Jameson put out a blog post detailing his attempts to sync these old versions to the main chain. But he noted that we was unable to sync and had gotten stuck at various blocks with all 4 major versions he attempted to sync. This of course, nerd sniped me once again as the failures all looked like the BDB locks problem that had a workaround published in 2013. But Jameson had tried that, and it didn’t work.

First Sync Attempt

So of course the first thing that I had to do was to try syncing an old node myself. I chose to use 0.5.0 as this was the oldest version I could easily build (I didn’t want to deal with the extra steps 0.4 required). As with Jameson, my attempt at syncing got stuck at block 258,354. That confirms that this is a persistent issue with 0.5, not a spurious failure.

The first thought is the BDB locks problem. But instead of jumping to using that solution, I wanted to dig a bit more to figure out what actually happened. Conveniently, in additon to the debug.log file, we also have db.log file which outputs messages from BDB itself. Usually this second log file is empty, but this time it was not:

Lock table is out of available lock entries

This led me to reading BDB’s documentation on the locking subsystem, which led me to the page on configuring sizes where we see the parameter the workaround mentions: set_lk_max_locks. Seeing that, I figured the correct course of action is to do as the workaround suggests and so I created a DB_CONFIG file in the datadir with the line set_lk_max_locks 1000000. I use 1000000 as I figured that a number larger than the suggestion in the workaround will be needed to sync the rest of the blockchain.

Second Attempt

With the parameter set, I restarted bitcoind and found that it still wasn’t syncing! Looking at the debug.log, I kept seeing a few lines get spammed over and over:

socket closed
disconnecting node 127.0.0.1:8333
trying connection 127.0.0.1:8333 lastseen=-429948.4hrs lasttry=-457726.1hrs
connected 127.0.0.1:8333
version message: version 70016, blocks=728263

That’s quite odd. The node is configured to only connect to a local node on this machine. It worked fine earlier, so why is getting disconnected now? I restarted my modern node but turned on network debug logging with -debug=net, where I saw the problem:

2022-03-20T21:50:33Z Header error: Size too large (headers, 5134836 bytes), peer=0
2022-03-20T21:50:33Z disconnecting peer=0

It seems that 0.5.0 pre-dates the implementation of a maximum size, currently 4 MiB, for p2p protocol messages. This is a headers message responding to a getheaders message from the modern node. Due to a bug in how 0.5.0 responds to getheaders messages, it ends up trying to respond with a significant portion of the blockchain that it knows about, and this exceeds the size limit. Resolving this is simple - remove the protocol message limit on my modern node.

On my modern node, I applied the following diff:

diff --git a/src/net.cpp b/src/net.cpp
index 955eec46e3..001283dc71 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -749,7 +749,7 @@ int V1TransportDeserializer::readHeader(Span<const uint8_t> msg_bytes)
     }
 
     // reject messages larger than MAX_SIZE or MAX_PROTOCOL_MESSAGE_LENGTH
-    if (hdr.nMessageSize > MAX_SIZE || hdr.nMessageSize > MAX_PROTOCOL_MESSAGE_LENGTH) {
+    if (hdr.nMessageSize > MAX_SIZE) {
         LogPrint(BCLog::NET, "Header error: Size too large (%s, %u bytes), peer=%d\n", SanitizeString(hdr.GetCommand()), hdr.nMessageSize, m_node_id);
         return -1;
     }

And now I could continue syncing without having to resync blocks.

Third Attempt

With the p2p problem out of the way, I figured the sync would continue as I have also increased the BDB locks. But to my surprise, it did not. This time though, the db.log had something different to say:

Lock table is out of available object entries

Looking back at the BDB documentation, there’s another option that mentions “objects” - set_lk_max_objects. I added set_lk_max_objects 1000000 to my DB_CONFIG and restarted.

Fourth Attempt

Finally I am able to get past block 258,354. In fact, this attempt was able to sync the entire blockchain! I just had to wait about two weeks.

Conclusion

0.5.0 was released in November 2011. It’s remarkable that Bitcoin has actually maintained backwards compatibility over the past 10 years. Furthermore, this is proof that Bitcoin has indeed only had soft forks since 2011.

Overall, this took two weeks in order to sync the blockchain. While 0.5.0 did not end up requiring multiple minutes to validate a single block, around block 395,000, each block started to take multiple seconds. This slowdown persisted through to the chain tip and ended up making the sync extraordinarily slow. In the end, I was observing four seconds per block.

Besides the long time it took to sync the blockchain, 0.5.0 also ended up temporarily having a much larger datadir than a modern Bitcoin node. I observed that, while it was running, it consumed 800 GB of disk space. It likely would have consumed more if I had not had to restart it because my computer ran out of disk space. But it turns out that a lot of that disk space is freed when the node shuts down. This is because the database uses journaling and those journal files take up 400+ GB. On shutdown, those journal files are compacted into the main database and cleaned up, freeing a ton of space.

The 0.5.0 datadir ended up being 400 GB after it was shut down when syncing completed. 301 GB of that are the block data, stored in blk*.dat files. Interestingly, this is less than the 372 GB of block data for my modern node. This discrepancy is of course due to witness data which 0.5.0 does not download. The modern node also stores “undo data” in the rev*.dat files, which conssume another 52 GB. However 0.5.0’s databases are far larger, at 99 GB, whereas the modern node only requires 4.6 GB for the chainstate, and 35 GB for the transaction index (which is optional but I have enabled anyways). So while modern nodes require a little bit more storage than 0.5.0, they still have more compact databases (which also do not grow to insane sizes while running) and sync far faster.

0.5.0 is certainly not the oldest Bitcoin software, it was just the most convenient old version for me to test. This experiment should be repeated in the future with even older versions to test how compatible Bitcoin really is.

URI Argument Injection Vulnerability in Bitcoin Core 0.18 and Earlier

Mon, 01 Feb 2021 10:00:00 -0500

URI Argument Injection Vulnerability in Bitcoin Core 0.18 and Earlier

This article describes an URI Argument Injection vulnerability discovered in Bitcoin Core 0.18 and earlier. The vulnerability was fixed in Bitcoin Core 0.19.0. Bitcoin Core 0.19.0 and later are not vulnerable. This vulnerability only affects Windows and Linux.

Background

In 2019, I was browsing the pwnie award nominations and found an entry titled “RCE in Qt5-Based GUI Apps.” As Bitcoin Core’s GUI (bitcoin-qt) uses Qt5, this immediately piqued my interest. The linked article describes a vulnerability in multiple Qt5 based software could have malicious code executed on a user’s computer after clicking a malicious URI. I proceeded to investigate whether this same vulnerability affects and found that it was theoretically possible. However it seems that this vulnerability is unlikely to actually be exploited.

URI Handlers

Understanding this vulnerability first requires that we understand how URIs are handled. Windows and Linux handles this in a similar manner. MacOS handles URIs differently so it is not affected by this vulnerability.

In Windows, the operating system executes a pre-registered command with the URI as an argument. This command is stored in the Windows registry and contains format specifiers for the location within in the command to insert the URI. Windows replaces the format specifier %1 with the URI and then executes the command string. For Bitcoin Core, the registry key is

HKEY_CLASSES_ROOT\bitcoin\shell\open\command

and the command is

"C:\Program Files\Bitcoin\bitcoin-qt.exe" "%1"

In Linux, the desktop environment searches for a .desktop file which contains a line like MimeType=x-scheme-handler/bitcoin. It then executes the command specified by the Exec= line. A common .desktop file for Linux has:

Exec=bitcoin-qt %u

This command is, like Windows, a string with format specifiers. The desktop environment will replace the format specifier %u with the URI and execute the command string.

URI Argument Injection

Since URI handling in both Windows and Linux basically amounts to creating a string containing the URI and then executing it, a maliciously crafted URI could make the string such that it looks like a command with additional command line arguments. For example, consider the URI bitcoin:BC1QXUFYM7NNUSQZW9R2U5G98USKDAT7XQN4VYZ63S" "-argument. With the Window’s URI command, the simple string replacement it does would create the command

"C:\Program Files\Bitcoin\bitcoin-qt.exe" "bitcoin:BC1QXUFYM7NNUSQZW9R2U5G98USKDAT7XQN4VYZ63S" "-argument"

Notice how this looks like the URI is followed by a command line argument -argument. In fact, it not only looks like that, it is interpreted by the OS that way and -argument is provided to the software as a separate command line argument instead of as part of the URI as it should be. In this way, a malicious URI could cause the software to start with a dangerous command line argument.

Injecting Arguments into Bitcoin Core

Since URI argument injection is a known issue, software developers have ways to avoid them. Bitcoin Core’s argument parser ignores any arguments that follow a URI. So it is not possible to use inject Bitcoin Core specific arguments through a URI. However, as noted earlier, the problem is with Qt.

The Qt framework specifies a set of Qt specific command line arguments. When Qt is initialized, the command line arguments are be passed into Qt in order for those arguments to be processed by Qt. This is done before Bitcoin Core processes the command line arguments. Qt will look through the argument list, find the arguments it can handle, and remove them from the list as it handle them. Then the arguments get passed to Bitcoin Core’s argument parser. This processing by Qt does not recognize URIs and will process any arguments regardless of their position in the arguments list. Thus it is possible to use a URI to inject Qt specific arguments.

Dangerous Command Line Arguments

Most of the Qt arguments are benign and Qt’s documentation specifies the available arguments. These arguments primarily affect the way that the application looks. However there is one specific argument that is problematic: -platformpluginpath. -platformpluginpath loads some plugin from a path and executes it. With some other tricks (described in the original “RCE in Qt5” article), it is possible to get a malicious plugin onto the victim’s computer, and once they click a malicious URI that injects -platformpluginpath, have that plugin be executed. As a malicious plugin can be executed, this is classified as a remote code execution vulnerability.

URI Argument Injection Mitigations

Of course, operating systems, browser developers, and software developers do not necessarily do the naive implementation. As previously mentioned, Bitcoin Core ignores command line arguments found after a URI. So a URI that injects arguments cannot cause Bitcoin Core to start with any additional Bitcoin Core specific command line arguments. Additionally, Bitcoin Core specifically registers a URI handler that surrounds the URI with double quotes so that an argument cannot be injected with just spaces. The attacker has to make a URI that itself contains double quotes to do the injection.

But browser developers are aware of the double quote trick, so they will sanitize the URI before passing it to the operating system to be handled. Browsers will escape the double quotes, and often do other sanitizing such as replacement of spaces, to ensure that the OS treats the entire URI as a single URI argument, not as multiple command line arguments. For example, for the URI bitcoin:BC1QXUFYM7NNUSQZW9R2U5G98USKDAT7XQN4VYZ63S" "-argument, a possible string actually used would be bitcoin:BC1QXUFYM7NNUSQZW9R2U5G98USKDAT7XQN4VYZ63S\" \"-argument and the final command

`"C:\Program Files\Bitcoin\bitcoin-qt.exe" "bitcoin:BC1QXUFYM7NNUSQZW9R2U5G98USKDAT7XQN4VYZ63S\" \"-argument"

Since the double quotes are escaped by being preceded with a backslash (\), they are not treated as delimiters between arguments but instead as part of a single argument string. Another method of sanitizing includes using URL encoding of special characters instead of escaping.

It is important to note that browsers must do this sanitizing in Windows. Windows itself does not sanitize the URI, so a browser which does not sanitize the URI would allow for arguments to be injected. However most of the browsers in use today will sanitize the URI in some way which prevents this vulnerability from being exploited.

When it comes to Linux, the %u specifier specifically means that the entire URI is to be treated as a single argument. Properly implemented desktop environments will do this correctly and not allow part of the URI to be treated as separate arguments. I do not know of any desktop environment that does not do this, so on Linux, the desktop environments prevent this vulnerability from being exploited.

Thus, in practice, argument injection via a URI is almost impossible to execute with Bitcoin Core and modern browsers and desktop environments. It is because of these mitigations that I believe that this vulnerability cannot actually be exploited.

Bitcoin Core’s Fix

Even though I believe that this vulnerability cannot actually be exploited, it is still prudent to fix the issue. The fix for this vulnerability is to disallow Qt from processing command line arguments. Instead of initializing Qt with the real list of command line arguments, we initialize it with an empty list of arguments. This was implemented in PR #16578 which was merged in August 2019. While this does fix the problem, it does also have the downside that users who do wish to use Qt’s other command line arguments to customize how it looks will be unable to do so via command line arguments. Instead this will need to be done through environment variables. A future followup would be to allow only some of Qt’s command line arguments to be used and/or disallow the dangerous ones.

Demonstration

Video on twitch.tv

CVE

This vulnerability has been assigned CVE-2021-3401.

Timeline

2019-08-02: Vulnerability discovered
2019-08-03: Vulnerability reported to security@bitcoincore.org
2019-08-03 - 2019-08-09: Further vulnerability investigation, disclosure to additional developers, and development of fix
2019-08-09: PR #16578 opened
2019-08-15: PR #16578 merged
2019-11-24: 0.19.0.1 released containing fix
2021-02-01: 0.18.0 End of Life
2021-02-01: Disclosure published

Announcing the Bitcoin Core Usage Survey

Tue, 19 Jan 2021 14:30:00 -0500

Announcing the Bitcoin Core Usage Survey

Today I am launching a survey aimed at gathering information about how people use Bitcoin Core.

Click here to take the survey

This survey is generously made possible by a grant from MIT’s Digital Currency Initiative and is being run with the help and collaboration of the researchers at Maiden Labs. A special thanks to Square Crypto grant recipient Jamaal Montasser for his input throughout the design as well.

The survey will remain up for 6 weeks, until March 2nd, 2021.

Goals

The goal of this survey is to learn why people use Bitcoin Core, how they use it, and what features are used. The results of the survey will be used to inform what features should be focused on and prioritized. By understanding how our users behave and use Bitcoin Core, we will be better able to write software that suits their needs.

Additionally the survey will allow users to provide feedback, such as feature requests, so that we can include things that people want.

Results

The plan is to share the results of this survey publicly. At the very least, the results will be shared with other Bitcoin Core developers so that many of the people involved will have data that can be used.

Privacy

As many people involved in Bitcoin are very privacy conscious, this survey was designed in order to preserve survey taker’s privacy as much as possible. The survey uses Alchemer and is configured so that no information about the survey taker is stored. While the survey does ask for location, survey source, and email, these are all entirely optional and can be left blank if so desired. Additionally IP information is not collected and the survey does not make use of any kind of tracking cookies. Even so, if you want to preserve your privacy as much as possible, I recommend using the Tor browser to take it. I have tested it with Tor and it does not appear to have any issues with the website; no annoying CAPTCHAs to fill out and the web page works as expected.

Survey Breakdown

In this section, I will go through the various questions being asked and what the purpose of asking them is. There are some questions which change which questions are asked. These are intended to accommodate the wide variety of participants that we expect so that only relevant questions will be answered.

Meta and Screener Questions

The questions in this section just provide some additional context about our survey participants as well as determining which questions are relevant to the participant.

Which country are you based in?
Where did you hear about this study?

These two questions allow us to control for potential bias due to the source of the participants (e.g. users that come from Reddit may be different from those who come from Twitter) as well as gain some insight into how users are distributed around the world.

Do you run a node with Bitcoin Core?
Do you use the Bitcoin Core Wallet?

These two questions direct the participant to different questions. Depending on the answer, different questions will be asked as some are not relevant to those who, for example, do not use the Bitcoin Core wallet.

Currently Node Operator Questions

Why do you own and/or use Bitcoin?
Why do you run a Bitcoin Core node?

These questions aim to provide us with the motivations for users to run a node. This will help us change how the node behaves so that it can better fit with why people are using Bitcoin Core as a node.

How long have you run a Bitcoin Core node?
How often do you update your Bitcoin Core node?
What version are you running?

These questions give us information about how frequently people actually update their node software as well as telling us what versions are actually in use. This will help us with deciding when and whether a new bug fix release should be made.

Select core features you’ve used on Bitcoin Core’s node over the last six months
If applicable, select all of the settings for which you have changed the default value

These two questions inform us as to what features people are using. This will help us know where to prioritize ongoing development. For some features, this could be informative as to whether a feature should be removed or changed.

What other Bitcoin software or hardware projects do you often use, if any?

Knowing what other software is being used can help us learn where there are deficiencies in Bitcoin Core that should be remedied. This would let us know what features we may want to implement as people are using other software for those features.

When you run into problems with your Core node, how do you usually troubleshoot those problems?
What’s the primary way you stay informed of Bitcoin Core development?

These questions allow us to learn where users are getting their information. This will help us with knowing where to go to distribute information in the event that something needs to be widely known. Additionally this lets us know where developers may need to increase their presence in order to better aid users.

Previously Operated Node Questions

Many of the questions for this section are the same as for the Current Node Operator questions.

When did you stop running a Bitcoin Core node?
Why did you run a Bitcoin Core node?
What’s the reason why you stopped running a Bitcoin Core node?

These questions will inform us what stops people from running a Bitcoin Core node so that we can improve.

Currently Use the Wallet Questions

What is most important to you when looking for a new wallet?

This question allows us to know what users want so that we can focus on desired features.

How did you find yourself primarily using the Bitcoin Core wallet?
How do you primarily use the Bitcoin Core wallet?
Is Bitcoin Core the only wallet you use?
What other wallets do you use and why do you use them instead of the Core wallet?

These two questions tell us how users use the wallet. With this information, we will be able to decide which set of features should be prioritized or more actively worked on. Knowing what other software is being used also lets us know what is missing in Bitcoin Core that we may want to emulate.

What does the Bitcoin Core wallet do better than other bitcoin wallets you have tried, if any?
Select features you’ve used on Bitcoin Core’s wallet over the last six months?
What feature would you be most disappointed to see removed from the Bitcoin Core wallet?
What features or actions did you wish the Bitcoin Core wallet supported better?

This questions lets us know what features we should prioritize, keep, and improve.

How many UTXOs are in your Bitcoin Core wallet?

This question is specifically to aid with the design of our coin selection algorithm. The coin selection algorithm is what decides what inputs to use in a transaction and is an integral part of the wallet. It can be optimized to different kinds of wallets. This question allows us to know what kind of wallets the coin selection algorithm should be targeting to better serve the majority of our users.

Previously Used the Wallet Questions

Many of the questions for this section are the same as for the Currently Use the Wallet questions.

How long were you using the Bitcoin Core wallet for?
When did you stop using the Bitcoin Core wallet?

These questions are to learn how long people tend to use the wallet for and whether stopping the use of the wallet is related to the node.

What’s the reason you no longer use the Bitcoin Core wallet?
What wallet did you switch to?

We want to know what turned users off to another wallet so that we can make the wallet better to meet the needs that others were fulfilling.

Never Used the Wallet Questions

What wallet do you use?
How did you decide which wallet to use?
Did you consider using the Bitcoin Core wallet?
Do you remember why you decided to not use the Bitcoin Core wallet?
What is most important to you when looking for a new wallet?
What feature would you be most disappointed to see removed from your current wallet?

Although users who see these questions do not use Bitcoin Core for their wallet, we would like to know why and whether there are things from the wallet that they use that we should implement in Bitcoin Core.

Never Used Bitcoin Core Questions

Do you own Bitcoin?
What wallet do you use to store Bitcoin?
Why do you own and/or use Bitcoin?
Have you considered running a Bitcoin Node?
What’s stopping you from running a node?

Although users who see these questions do not use Bitcoin Core, it is still useful for us to know why they do not so that we can hopefully address those concerns and onboard new users.

Acknowledgements

Special thanks to MIT’s DCI for generously sponsoring this survey, and for the researchers at Maiden Labs, particularly Zach Herring and Shira Frank, who helped me put this survey together. Additional thanks to Jamaal Montasser, Samuel Dobson, Sjors Provoost, and Jeremy Rubin who all provided additional insight into the questions and goals of this survey.

What's Coming To The Bitcoin Core Wallet in 0.21

Wed, 21 Oct 2020 14:30:00 -0400

What’s Coming To The Bitcoin Core Wallet in 0.21

Bitcoin Core 0.21 brings about a turning point to the Bitcoin Core wallet with the introduction of two major features and an eventual breaking of compatibility. The landmark wallet features of this release are Descriptor Wallets and the SQLite database backend. In this article, I will be explaining what these features are, what else is changing, and what the plan for the future is.

Upcoming Features

Descriptor Wallets

The headlining wallet feature for Bitcoin Core 0.21 is Descriptor Wallets. Descriptor Wallets store Output Script Descriptors in the wallet and use these to generate the addresses that users can use. Legacy Wallets (the non-descriptor wallet type, and the only type of wallet previous versions would create) instead used private keys to generate addresses. For even more detailed information on Descriptor Wallets, please read the tentative release notes.

The Problem with Legacy Wallets

Legacy Wallets were initially designed by Satoshi himself. As this was when Bitcoin was first created, the understanding of what Bitcoin could do and how Bitcoin can be used was not as well understood as it is today. As such, the wallet was built around the private keys and everything based around private keys even though Bitcoin has a scripting language that supports a lot more than just private keys. So Legacy Wallets primarily contain private keys, and from these private keys, addresses are made.

Even with just private keys, the Legacy Wallet was designed long before BIP 32 Hierarchical Deterministic Wallets were invented. It was designed to use a random number generator to create private keys, not deriving them from a seed. While the Legacy Wallet does support BIP 32 now, it is still not up to the level of support that other wallets have. This support is done by replacing the RNG with BIP 32 derivation. But there is no support for watching extended public keys (xpubs) and pubkeys cannot be derived, nor are pubkeys stored by the wallet.

Since Bitcoin was first created, we have learned that there are many more things that output scripts can do than just single keys. The scripting system in Bitcoin is powerful, but Bitcoin Core’s wallet cannot make use of it because more than just keys are involved in scripts. Even just the introduction of P2SH introduced some issues for the Bitcoin Core wallet, and supporting the watching of scripts is largely bolted on and not integrated very well. Watch-only in Legacy Wallets does not quite work as one would expect, and having a mix of watch-only and non-watch-only things in a Legacy Wallet only makes things more confusing. Ultimately, extending the Legacy Wallet has primarily been hacking in new features rather than having a truly well designed wallet that can make the full use of Bitcoin.

Benefits of Descriptor Wallets

In contrast with Legacy Wallets, Descriptor Wallets are designed to support the Bitcoin scripting system through the use of descriptors. Descriptors explicitly give an output script (and thus address) as well as all of the keys and scripts necessary to sign them. This essentially means that Descriptor Wallets are a script based wallet, while Legacy Wallets are key based.

A Descriptor Wallet will then be able to support any kind of descriptor. Newly introduced descriptors for new script types can be easily added to the wallet by adding a new descriptor. For example, the Taproot proposal introduces a new address type and output scripts. This can be easily added to the Bitcoin Core wallet by implementing a new descriptor.

Alternative ways of deriving keys and scripts can be added by implementing them in the simpler descriptor module. For example, Miniscript will eventually be added to the descriptor module. This will allow descriptors to describe arbitrary scripts, which means that the wallet will be able to create addresses for arbitrary scripts.

Descriptor Wallets also bring a simpler and saner way to do watch-only wallets in Bitcoin Core. The confusing behavior of having mixed watch-only and non-watch-only things in Legacy Wallet is not present in Descriptor Wallets. Descriptor Wallets can only have descriptors imported into them as well, so the scripts that it will watch for is explicit and can be computed. There will no longer be the Legacy Wallet behavior of watching for something related to an import but was not explicitly imported. Notably, because descriptors support xpub derivation, watch-only Descriptor Wallets can watch for BIP 32 wallets such as hardware wallets. This paves the way for full hardware wallet support in a future release of Bitcoin Core.

SQLite Database Backend

As Descriptor Wallets are a wholly new type of wallet which is backwards incompatible, it made sense to introduce a new database backend at the same time. So newly created Descriptor Wallets will now use SQLite as the database backend, instead of Berkeley DB that is used for Legacy Wallets. This will eventually lead to the outright removal of Berkeley DB. The introduction of a new database backend is motivated by the fact that Berkeley DB is not suitable for our use case, and the age of the version of Berkeley DB that we use.

The problem with Berkeley DB

Berkeley DB is not very suitable for our use case as an application data file; it was not designed to be used in this way. Instead it was designed for large, multithreaded databases, which the wallet is not. Because of this, the Legacy Wallet has several hacks to force Berkeley DB to behave like an application data file database, and these hacks can result in undesirable behavior. Notably, Berkeley DB wallet files can be easily corrupted which can result in the loss of private keys, which is not good.

Furthermore, Berkeley DB requires a database environment and it produces extra files which typically need to be moved with the database file if it is moved elsewhere. Without those extra environment files, data can be lost. This means that Berkeley DB is not portable, but that is unsuitable for a wallet file. Many users expect it to be a singular file that can be transferred to other computers or to other directories for backups. The Berkeley DB environment requirement also forces us to have directories for each wallet. This is confusing and unintuitive to users.

Lastly, the version of Berkeley DB used by Bitcoin Core is more than 10 years old. We use Berkeley DB 4.8 which was released in 2010. The requirement on this aging version of Berkeley DB is because backwards incompatible changes were introduced to the database environment files. This means that wallets loaded with a newer version of Berkeley DB cannot be loaded with an older version of Berkeley DB, and this breaks downgrade compatibility. This age has thus introduced additional issues for us. For example, to compile Berkeley DB using C++11, a patch is required. Berkeley DB 4.8 also cannot currently be compiled on the macOS Big Sur beta which is problematic for users who use (and will use) that OS and want to self compile Bitcoin Core. So a new database backend is desired.

Why SQLite

SQLite was chosen to be the new database backend because the SQLite developers provide several guarantees with regards to compatibility, support, and testing. The SQLite developers advertise that SQLite is suitable as an application data file, which is what we want for the wallet. SQLite is also very well tested and very widely used, so bugs and corruption issues tend to be found and fixed quickly. Furthermore, SQLite makes few file format changes, and all of these file format changes are feature dependent and backwards compatible. This means that if we don’t use a particular feature, then the file format will not change and it will be possible to upgrade and downgrade SQLite versions. Even though we currently use the latest version of SQLite, we have found that it is possible to use a version of SQLite from 2013.

SQLite is also designed to be portable and does not require a database environment. While it does temporarily produce extraneous files, those files are removed as soon as it is done with them, and everything is written to the database file when a database write completes. This is in contrast to Berkeley DB where a completed write does not guarantee that the data was written to the database file, it may have been written to an environment file. The lack of the environment will also allow us to move to single wallet files instead of wallet directories.

Other databases, such as LMDB, were also in consideration however none of the databases investigated had all of the same guarantees of SQLite. For example, LMDB is not portable across different CPU architectures, so this was determined to not be suitable. Many other databases have database environments and environment files, so these were not suitable for use either. SQLite was the only widely used database system that had all of the features that we wanted for a wallet file.

No Longer Creating The Default Wallet

The Bitcoin Core wallet used to always create a wallet for the user when one did not already exist. With the introduction of the multiwallet feature and the ability to create and open specifically named wallets, the default created wallet started to become more of an issue rather than a useful feature. In particular, users would sometimes find that their wallet appeared to disappear when some configuration changed. Of course, the wallet would not actually disappear, just that the data directory changed and a new default wallet was created. This can be very confusing and extra stressful to the user as a new blank wallet is created. This situation can be hard to debug, hard to explain, and difficult for non-technical users to fix.

So The default wallet will no longer be created if it doesn’t exist. Instead users will be prompted to create a new wallet or open an existing wallet. In this way, when a configuration mistake occurs, a new empty wallet won’t be created so users won’t think that all of their Bitcoin has disappeared. Furthermore, not creating a default wallet means that disk space is not unnecessarily used. Users can create a wallet of the type and name that they want instead of having an unnamed default wallet taking up space on disk.

Since users expect that their wallets be loaded when they start, there have been changes to the configuration system as well. These changes, while not wallet related, allow for Bitcoin Core to save which wallets to load again on the next startup. So the wallets that users create and load from the GUI will automatically be loaded again on the next startup unless they were closed during the session. For RPC users, a load_on_startup argument has been added to the createwallet, loadwallet, and unloadwallet RPCs to specify whether a wallet should be loaded on startup.

Compatibility

An important change to the wallet is that compatibility will be breaking. Backwards compatibility has always been a key pillar in the Bitcoin Core wallet. Changes have always had to consider users who decided to downgrade after an upgrade and users who bring in old wallets to a new release. But maintaining backwards compatibility is difficult, and almost impossible to do when making large sweeping changes.

Unfortunately, to make progress, sometimes compatibility needs to be broken. Sometimes the past needs to be left behind. As the Bitcoin Core wallet moves to making Descriptor Wallets the default, and later, only, wallet type, a future release of Bitcoin Core will no longer be able to open Legacy Wallets.

Of course this does not mean that the change will be overnight and occur in one release. Nor does this mean that Bitcoin Core will begin to make gratuitous backwards compatibility breaking changes. Instead we will be making it easy to move from Legacy Wallets to Descriptor Wallets. We will be making every effort to make Descriptor Wallets backwards compatible with previous versions that have them.

As such, there will be a long deprecation cycle for Legacy Wallets and Berkeley DB. The proposed timeline is to only have the final removal occur in late 2023. In the meantime, we will be introducing ways to migrate a Legacy Wallet to a Descriptor Wallet. It is, and will be, possible to take an existing Legacy Wallet, and make a Descriptor Wallet which exactly matches it. There will be tooling that allows users to do this. Even after the removal of Legacy Wallets and Berkeley DB, it will be possible to migrate Legacy Wallets to Descriptor Wallets through the use of a minimally dependent migration tool.

As we get closer to the actual removal date, there will also be warnings and instructions on how to migrate a wallet. So even users who don’t read the release notes and don’t pay attention to the Bitcoin Core news will know that they should migrate, and how to do so. Even though compatibility will be breaking, we are not leaving users out to dry and we are doing everything that we can to make sure that the change to Descriptor Wallets only is as smooth as possible.

Atomic Swaps Across Bitcoin Chain Splits

Wed, 11 Oct 2017 22:00:00 -0400

Atomic Swaps Across Bitcoin Chain Splits

Updates

A previous version of this suggested the use of OP_CHECKLOCKTIMEVERIFY but after futher thought, I do not believe that is necessary.

There has been a bit of interest regarding a way to do atomic swaps between two blockchains which will split from each other with a hard fork. This post describes a way to do so with neither party needing to trust each other or a third party. This post is written in the context of the Segwit2x hard fork (and an example will be given using that) but can be applied to any such hard forks.

Requirements

First, let’s establish what we want to do

Allow two parties, Alice and Bob, to swap their coins in the future at or after the time of the hard fork without trusting each other a third party
Allow both parties to back out of the agreement at any time and neither lose coins.

The Method

The Multisig Address

To do this, we need both parties to approve the transaction, so we use a simple 2-of-2 mulltisig address. This way the transactions that pay from the address must be signed by both parties. Once we create the address, both Alice and Bob will fund the address with the coins they wish to swap.

The Spending Transactions

The first transaction will send the entirety of the coins in the above address to Alice on chain A. To do this, we simply create a transaction which sends the coins to an Alice’s address and include the replay protection that is employed by chain A (e.g. a specific output that chain B will reject). We then set the nLocktime value of the transaction to be that of the fork activation height or time. We will actually set this to <fork activation height/time> + 2 days to allow for the backing out transaction. Alice should have a copy of this transaction, and Bob can as well.

The second transaction is very similar to the first transaction. It will send the entirety of the coins in the above address to Bob on chain B. To do this, we create a transaction which sends the coins to Bob’s address and include the replay protection that is employed by chain B (e.g. a new parameter that makes this transaction invalid on chain A). We then set the nLocktime value of the transaction to be that of the fork activation height or time. We will actually set this to <fork activation height/time> + 2 days to allow for the backing out transaction. Bob should have a copy of this transaction, and Alice can as well.

Backing out

If Alice or Bob decides that they really don’t want to go through with the swap, they have a way to back out after the fork activation height. To back out, Alice and Bob create a third transaction which sends the amount that they put in (minus transaction fees) back to themselves on their respective addresses. This transaction does not have a nLocktime value at all so it can be broadcast at any time. If Alice or Bob wants to back out of the atomic swap at any time, this back out transaction can be broadcast and once it confirms, the prior two transactions will be invalidated. Both Alice and Bob should have a copy of this transaction.

Fork Day Events

When the hard fork activates, if Alice or Bob wishes to back out, they should broadcast the back out transaction immediately. Because it was done in advance, the transaction fees may not be enough and a Child-Pays-For-Parent transaction may be needed. It will have two days to confirm.

If Alice and Bob wish to continue with the swap, then they will wait until 2 days after the fork occurs and broadcast their respective transactions. Alice will broadcast here transaction to the network using chain A and Bob will broadcast his transaction to the network using chain B. Because both transactions employ replay protection, they are not at risk of being replayed on either network and thus losing coins.

An Example with Segwit2x

Now we will have an example atomic swap with Segwit2x. The examples will be done with Bitcoin Core.

Suppose Alice wishes to swap 10 B2X for 10 of Bob’s BTC. This means that after the swap is complete, Alice will have 20 BTC and Bob will have 20 B2X.

First they will create their addresses. Alice generates a new address and retrieves its public key:

$ bitcoin-cli getnewaddress
1Hs12LwwdXu2bhiuqyPfmWqUmTcZSBqEiS
$ bitcoin-cli validateaddress 1Hs12LwwdXu2bhiuqyPfmWqUmTcZSBqEiS
{
  "isvalid": true,
  "address": "1Hs12LwwdXu2bhiuqyPfmWqUmTcZSBqEiS",
  "scriptPubKey": "76a914b8f6d4790d053d6b781f5fe229ecab8aab881d4388ac",
  "ismine": true,
  "iswatchonly": false,
  "isscript": false,
  "iswitness": false,
  "pubkey": "02fd4dbc0f5076881586ce7ae84a3bd2854d2c2ca7782e7d3de9d4cfe64c20daa9",
  "iscompressed": true,
  "account": "",
  "timestamp": 1505671741,
  "hdkeypath": "m/0'/0'/3'",
  "hdmasterkeyid": "5daeb23c6fda331386be884baab609d78531292e"
}

And Bob generates a new address and retrieves its public key:

$ bitcoin-cli getnewaddress
1JngAG1EfbjLqcfe3qcHNeV4Aeqc3NK4v3
$ bitcoin-cli validateaddress 1JngAG1EfbjLqcfe3qcHNeV4Aeqc3NK4v3
{
  "isvalid": true,
  "address": "1JngAG1EfbjLqcfe3qcHNeV4Aeqc3NK4v3",
  "scriptPubKey": "76a914c31d8ba0a258f80eabade40a705f5831c8d28d6488ac",
  "ismine": true,
  "iswatchonly": false,
  "isscript": false,
  "iswitness": false,
  "pubkey": "0208fe052d79b9feeb2d5b12ba72d5fab857a055275d1c550573c61c251751a643",
  "iscompressed": true,
  "account": "",
  "timestamp": 1505671741,
  "hdkeypath": "m/0'/0'/4'",
  "hdmasterkeyid": "5daeb23c6fda331386be884baab609d78531292e"
}

Now they are going to create the multisig address:

$ bitcoin-cli addmultisigaddress 2 '["02fd4dbc0f5076881586ce7ae84a3bd2854d2c2ca7782e7d3de9d4cfe64c20daa9","0208fe052d79b9feeb2d5b12ba72d5fab857a055275d1c550573c61c251751a643"]'
34sQK7HNe7eXwhduxcCNTxXaRJFyBWH6K4

Now Alice and Bob are going to fund that address with transactions $TX_A and $TX_B. These have outputs $VOUT_A and $VOUT_B which fund the above address.

We now need to create our spending transactions. First we create Alice’s spending transaction. The replay protection that we employ here is an OP_RETURN output with the string RP=!>1x as implemented in this Pull Request to btc1:

$ bitcoin-cli createrawtransaction '[{"txid":"$TX_A","vout":$VOUT_A},{"txid":"$TX_B","vout":$VOUT_B}]' '{"1Hs12LwwdXu2bhiuqyPfmWqUmTcZSBqEiS":19.999,"data":"52503d213e3178"}' 495072
0200000002..e08d0700

Next we create Bob’s spending transaction. The replay protection that we employ here is that the transaction will be larger than 999920 bytes and less than 1000000 bytes so that it is too large to fit in a Bitcoin block but just small enough to fit in a Segwit2x block. To do so, we create 1886 outputs which have a maximum script size of 520 bytes. Note that we actually give it 516 bytes of data for the 1882 outputs because three bytes are used to represent the size of the data and one byte for the OP_RETURN. Unfortunately this transaction is going to be nonstandard, but it is the only known way to make a transaction only valid on the Segwit2x chain without using UTXO tainting.

$ bitcoin-cli createrawtransaction '[{"txid":"$TX_A","vout":$VOUT_A},{"txid":"$TX_B","vout":$VOUT_B}]' '{"1JngAG1EfbjLqcfe3qcHNeV4Aeqc3NK4v3":19.999,"data":"<516 bytes of data>",<repeat 1882 times>}' 495072
0200000002..e08d0700

And then we create our back out transaction:

$ bitcoin-cli createrawtransaction '[{"txid":"$TX_A","vout":$VOUT_A},{"txid":"$TX_B","vout":$VOUT_B}]' {"1JngAG1EfbjLqcfe3qcHNeV4Aeqc3NK4v3":9.999,"1Hs12LwwdXu2bhiuqyPfmWqUmTcZSBqEiS":9.999}' 494785
0200000002..c18c0700

Lastly we sign these transactions:

$ bitcoin-cli signrawtransaction 0200000002..e08d0700
0200000002..e08d0700
$ bitcoin-cli signrawtransaction 0200000002..e08d0700
0200000002..e08d0700
$ bitcoin-cli signrawtransaction 0200000002..c18c0700
0200000002..c18c0700

And broadcast them:

$ bitcoin-cli sendrawtransaction 0200000002..e08d0700
{TXID}
$ bitcoin-cli sendrawtransaction 0200000002..e08d0700
{TXID}
$ bitcoin-cli sendrawtransaction 0200000002..c18c0700
{TXID}

Software For This Example

Bash Script to create Bob’s spending transaction

Acknowledgements

I think Greg Maxwell first posted about this method on Reddit but did not go into as much detail or with an example as I have done here. I cannot find the post for it though.

Bitcointalk Account Name Change

Sat, 06 Aug 2016 11:36:00 -0400

Bitcointalk Accounts Name Change

My primary account (290195) on Bitcointalk.org has had its name changed from knightdk to achow101. My secondary account (uid 466100) has had its name changed from achow101 to achow101_alt.

Bitcoin Core Troubleshooting FAQ and Tips

Sun, 24 Jul 2016 23:00:00 -0400

Bitcoin Core Troubleshooting FAQ and Tips

Over the past year, I have been hanging out on various Bitcoin forums, the most frequented being Bitcointalk, and helping people out with various tech support issues with Bitcoin Core. I decided to compile the most frequent issues and troubleshooting tips into this post here, partially to help people troubleshoot their install, and partially to help me not have to keep posting the same thing over and over again.

This document will continue to grow as I hear of more and more troubleshooting issues.

Troubleshooting FAQ
Troubleshooting Tips

A transaction is stuck, how do I make it confirm?

In Bitcoin Core, it is fairly easy to remove a transaction from your wallet so that you can resend the transaction with a higher fee. There are two methods to do so, the abandontransaction command and the -zapwallettxes startup option.

To use the abandontransaction RPC, you first must be running Bitcoin Core 0.12 or later. Then open up the debug console and use the command

abandontransaction <txid>

where <txid> is the transaction id of your stuck transaction. Do this for every stuck transaction. If the command is successful, there will be no error and no output.

If abandontransaction did not work or if you do not have Bitcoin Core 0.12 or later, then you can use the -zapwallettxes startup option. To do so, just follow the instrucionts to start Bitcoin Core with an option where the option you want to use is -zapwallettxes.

Wallet Balance is incorrect

If your wallet balance is incorrect, a number of things could have gone wrong. First double check that you are fully synced. Check a block explorer to see the latest block height. Next open Bitcoin Core and hover your mouse over the check mark at the bottom right hand corner. It should pop up with a little info box that says how many blocks Bitcoin Core has processed. This number should match the latest block height in the block explorer. If it does not, wait for Bitcoin Core to finish syncing. If you are synced, try starting Bitcoin Core with the -rescan option. If that does not fix the problem, next check your receiving addresses at File > Receiving addresses. Make sure that you are not missing any addresses. If you are, check whether the addresses you are missing are still in your wallet. If the address is missing from your wallet, then you will have to restore from a recent backup in order to recover it. Otherwise there is nothing that can be done.

Wallet is empty

If your wallet is empty and you see no familiar transactions or addresses, it is likely that the wallet file Bitcoin Core is currently using is not your actual wallet. If you had recently pressed the Reset Options button in Settings > Options, then your data directory may have been reverted to the default. To check, go to Help > Debug Window and go to the Information tab. In the field labeled Datadir, the path to your data directory will be listed. If that is not what you expect (custom if it was set, or default if not), then this is why your wallet is empty. If you had a custom data directory, you must start Bitcoin Core with the -datadir=<path> option where <path> is the path to your data directory.

Troubleshooting tips

Using the Debug Console
Start Bitcoin Core with options
Check for a private key in the wallet
Open the debug.log file

Enter commands into the debug console

To open the debug console, go to Help > Debug Window in Bitcoin Core. In the new window that pops up, click on the Console tab. This is the debug console. You type the commands in the small box at the bottom of the window. The command help will give you a list of all commands available. Typing help <command name>, where <command name> is one of the commands, will give you the usage for that command.

The debug window will let you know if something was successful through the output of each command. Most of the commands will have an output. If the output is in black text, then your command was successfully run. If the output is in red text and has some sort of error code, then the command failed and there was an error.

The commands also take arguments in JSON format, so entering the commands with proper JSON format and escaping is essential.

Start Bitcoin Core with a startup option

Starting Bitcoin Core with startup options can be very useful. The method of doing so is dependent on your Operating System

Windows
Linux
Mac OSX

Windows

Right click the shortcut that you use for starting Bitcoin Core. Click on Properties in that menu. In the Properties window, go to the box labeled Target. Click the box and move your cursor all the way to the right, past what is already in there. Then just type the options you want, making sure that there is a space between what is already in the box and your option, and a space between each option. Then just click OK and double click the shortcut to start Bitcoin Core. When Bitcoin Core is fully started, you can repeat this process and remove the options that you added.

Linux

Open the terminal in Linux. If you did not install Bitcoin Core and are just running the binary, navigate to the directory you have the bitcoin-qt file in. Then type

bitcoin-qt <options>

where <options> are your startup options. Just make sure to have a space between each option. Press enter and Bitcoin Core will start with those options.

Mac OSX

Disclaimer: I don’t have a Mac so I am not 100% sure that this works, but it should.

Open the Mac terminal. Then type

open Bitcoin-Qt.app --args <options>

where <options> are your startup options. Just make sure to have a space between each option. Press enter and Bitcoin Core will start with those options.

Check if the private key to an address is in the wallet

Open the debug console and use the command

dumpprivkey <address>

where <address> is the Bitcoin address whose private key you want to check is in the wallet. If the command was successful, it will print the private key to the console. DO NOT SHARE THE PRIVATE KEY WITH ANYONE. This key should begin with a ‘5’, ‘K’, or ‘L’. If it does not, it is not a private key. If the command failed with an error, the private key to the address is not in your wallet.

Get the debug.log file

The debug.log file is a log file that is very useful for troubleshooting. It does not leak any information about your private keys so your Bitcoin is always safe. To get the debug.log file, open Bitcoin Core and go to Help > Debug Window and go to the Information tab. Near the bottom right hand corner of the window is a button labeled Open with Debug log file. Click this to access the file. Now you can save the file elsewhere or copy the contents to send to someone.

If Bitcoin Core is unable to open for some reason, go to the data directory. If you set a custom data directory on first startup or with the --datadir option, then you must go there. The default locations are described at https://en.bitcoin.it/wiki/Data_directory#Default_Location. Once you are at the data directory, find the file named debug.log. This is the file we are looking for. You can open this with any text editor and share the file with others.

Announcing the Bitcoin Fork Monitor

Sun, 08 May 2016 10:00:00 -0400

Updates

July 18th 2017: The site has been revived and rewritten in python and django in anticipation of the BIP 91 and BIP 148 forks.

June 19th 2016: The site has been shut down due to the expense of maintaining it.

The Bitcoin Fork Monitor

I have created a new website, http://btcforkmonitor.info, which monitors the blockchain for the status of planned forks. It will continuously be updated as new forks are planned and deployed. It will only support forks that are assigned a BIP number and supported by a significant proportion of the Bitconi community. The site automatically follows and updates as new blocks are added and it will determine whether the block is supporting a specific fork.

How it works

The website uses a bitcoind in the background to receive, validate, and relay blocks to the actual site process. The site’s servlet backend will receive the block and examine the raw block data to determine which fork, if any, that the block supports.

The source code

This website is open source. The source code can be found at https://github.com/achow101/ForkMonitor. It is written in Java and utilizes the Google Web Toolkit, ObjectDB, and ZeroMQ libraries. The site and the source code is licensed under the Affero Gneral Public License.

Clearing the FUD around Segwit

Fri, 01 Apr 2016 16:00:00 -0400

Clearing the FUD around Segregated Witness

Lately I have been noticing on a variety of online bitcoin communities such as bitcointalk.org, /r/bitcoin, /r/btc, bitco.in, and bitcoin.com that many users do not understand what Segregated Witness (commonly known as segwit) is and what it does. I have seen that there are plenty of either uninformed or misinformed users spreading FUD (fear, uncertainty, doubt) about segwit. I am here to clear this up.

What is SegWit?

Segregated Witness is a solution created by the Bitcoin Core developers. It is spearheaded by Pieter Wuille, Johnson Lau, and Eric Lombrozo. The general high-level idea of segwit is that the signatures in a transaction (also called the witness data) are skipped when calculating the transaction id. By doing so it solves many issues that Bitcoin has. These benefits are laid out here.

Clearing up the myths

In this section, I will clear up many of the myths and misinformation that surround segwit. This will grow and change as people respond to me.

Myth: Segwit is primarily for the Lightning Network

This is simply not true. The true and original purpose of segwit was to prevent transaction malleability. Since it removes the signatures from the data that is hashed to become the transaction id, there is then simply no way for a third party to change the signature data so that it was still valid but produced a different transaction id from the original. This attack is the High-S/Low-S attack which works because the signatures which protect the rest of the transaction from being modified cannot protect itself. With this separation, segwit is no longer vulnerable to these transaction malleability attacks as the txid is then calculated from data that cannot be changed at all by a third party.

Although it began as a transaction malleability fix, it has evolved to something a little more complex. Segwit introduces a new hash pre-image generation algorithm which will make signature hashing operations scale linearly. The hash pre-image is the data that is to be hashed. This hash will be signed and that is the signature for the transaction. The change enables the use of hash midstates which allows for faster and more efficient hashing. This makes the relationship between the number of signature hashes and the time to generate them linear instead of the former quadratic relationship.

Additionally, to maintain backwards compatibility, the size of the transaction that counts for part of the block size is the size of the data with the old transaction serialization. This means that the witness data is not included in this calculation. This means that a significant portion of the data in a transaction is not counted in the block size more transactions can then fit in a block. This is effectively a capacity increase. However full nodes will still be downloading all of the witness data to verify blocks and transactions and they will still have the same network bandwidth cost as an increased block size to support a similar capacity would have.

Myth Segwit as a soft fork is more dangerous than a hard fork

Segwit deployed as a soft fork is actually safer than a hard fork. A soft fork means that backwards and forwards compatibility is maintained. Old versions of Bitcoin software will be able to function with no ill effect when a soft fork is deployed. In contrast, a hard fork requires that every single Bitcoin user upgrade their software to support the new consensus rules. This has the effect of being not backwards compatible and thus forcing users to upgrade to the latest version or risk being kicked off of the Bitcoin network.

Segwit was in fact originally planned as a hard fork, but the developers realized that it could be done as a soft fork and doing so would be safer than a hard fork. The only differences between segwit as a soft fork and as a hard fork is where the witness root hash goes and whether the witness data is counted in the block size or not. With the witness root hash, it could have either been in the block header or in the coinbase transaction. Segwit chose to put it in the coinbase transaction as an OP_RETURN output. This is safer because this allows for segwit to be done as a softfork even though the data is still the same size as it would have been if it were in the header. The other difference is whether to include the witness data in the block size count, and to maintain backwards compatibility while also having the capacity increase, it was decided to not do so.

Myth: Segwit is kludgy hacked together software that is not ready.

No, this is very far from the truth. The Bitcoin Core developers have very strict testing and require that a test exist to test every single scenario. When new functionality is introduced, an automated test must also exist. The developers will also review the code and test it themselves to guarantee that it works as intended. Segwit is also written by some of very good developers who both came up with the concept and then implemented it. Before being added to Bitcoin Core, it will be tested very thoroughly over several days or even weeks to make sure that it is all ready. The implementation now is already being tested, with it being run by several people and on its fourth iteration of the segregated witness test network.

There are however, a few hacks which are designed to allow segwit nodes to maintain compatibility with older software. The witness root hash being placed in the Coinbase transaction and the witness data not being counted towards the block size are two most prominent hacks that segwit employs. These are necessary to maintain the compatibility with non-upgraded nodes.

Myth: Segwit is much more complex than a super simple hard fork.

While Segwit is indeed more complex and introduces many changes, but it is a relatively simple conceptual change, similar to a hard fork to increase the blocksize limit. On the surface, they appear simple. Segwit ignores the signatures when calculating the transactions, but as a soft fork, some additional changes must be made to make segwit transactions compatible with non-segwit nodes. These changes then have side effects which can be beneficial to Bitcoin. It also contains more functionality than a hard fork increasing the block size limit. The hard fork to increase the blocksize limit also appears simple, but additional changes need to be made to support the deployment and to solve the quadratic hashing issue with transactions.

Additionally the BIPs and the Segwit Wallet Dev Guide are detailed and can help developers implement Segwit properly.

Myth: When segwit activates, I won’t be able to send or receive my Bitcoin anymore if I don’t upgrade.

With the way that segwit is designed, people can still send their transactions using Pay-to-Pubkey-Hash, Pay-to-Script-Hash, and Pay-to-Pubkey outputs as the current software do. These outputs and they way to spend from these outputs will not change. The signatures for verifying these transactions will not be moved and will remain in the scriptsig.

What segwit does change is that it adds two new output types, Pay-to-Witness-Pubkey-Hash and Pay-To-Witness-Script-Hash. These two new output types require that the signatures to spend from those outputs are in a new Script Witness array which is not included in the hash of the transaction. The scriptsig of the inputs that spend those output types are empty. These output types are still compatible with non-upgraded nodes because they will always validate as true to those non-upgraded nodes because when they validate those transactions, the stack is not empty and not zero. Upgraded nodes will validate them fully because those nodes have access to the signatures and recognize that what the output format is.

Segwit also creates a new type of Pay-to-Script-Hash address. The redeemscript of the p2sh address is one of the two new witness output types. The redeemscript is hashed and that becomes a p2sh address. This allows for senders who have not yet upgraded to send to a person who has upgraded and the recipient, when he spends from this p2sh output, can take advantage of segwit’s benefits.

Overall, the changes that segwit makes do maintain compatibility with non-upgraded nodes. With these changes, a non-upgraded user can send Bitcoin using the current output types, thus able to send to both upgraded and non-upgraded users. Because the transaction uses the current output types, the following transaction cannot take advantage of segwit. However, a non-upgraded user can send to a segwit output nested in a p2sh output which allows them to send to upgraded users who can then take advantage of segwit for the following transaction. Upgraded users can also spend from any of the output types and still create the current output types so they can send to non-upgraded users as well.

Myth: If I don’t upgrade, I can be attacked by people sending me transactions with segwit outputs

If you don’t upgrade, your wallet will not know of the segwit outputs. It will not know that these outputs are meant for you. This can of course create problems if an upgraded user decides to create a segwit output when sending to a non-upgraded user. The non-upgraded user will not recognize the transaction is for him and this can create disputes. Unfortunately this flaw cannot be fixed through the consensus rules but rather through software implementation. For this, the best thing to do is for software to not use the segwit output types at all but rather continue to use the currently used p2pkh, p2sh, and p2pk output types. This can still take advantage of segwit because the new segwit output types can be nested inside of a p2sh address. New wallet addresses should be of this type; the p2wpkh or p2wsh output is placed as the redeemscript of a p2sh address. Thus users can still send to each other without issue and the upgraded users can still take advantage of segwit.

Myth: Miners who don’t upgrade to segwit will be forced off of the network

This is not true, miners will not be forked off of the network. Segwit will be deployed using BIP9 versionbits which uses a 95% threshold. If a miner had less than 5% of the hash power and did not upgrade, he would not run into significant trouble so long as he follows certain rules (the standardness rules) which most miners do actually follow today. If this miner follows the standardness rules of today which are widely accepted by everyone, then he does not risk being forked. His blocks would simply not contain an transactions that spend from the output types nor have transactions that create the new output types. His blocks do not need to have the witness root hash in the coinbase so long as the block does not contain any transactions that have witnesses.

However, if the miner did not follow the standardness rules, then he could end up including transactions with witnesses but he wouldn’t have the witnesses nor would he have the witness root hash in the coinbase. This would be an invalid block and it would be orphaned. Because the vast majority of the hash power supports segwit, this miner’s small fork would end up orphaned and he would switch back to the longest blockchain which was extended on by the segwit miners.

Additionally, the miner has an incentive to upgrade as less and less transactions will continue to be the current normal transactions. More and more will have the segwit outputs which and spend from them which reduces the fees that this miner could get. Eventually, there will be some point where there are no transactions that this miner could include a block and thus he has an incentive to upgrade in order to get the transaction fees. If he did not, he would only earn from the block subsidy.

Myth: The witness discounts are to incentivise people to use segwit

The primary idea behind the discounts are to incentivise wallets to manage change differently and clean up the UTXO set. This is best explained by Adam Back here

Welcome to my new website!

Sat, 27 Feb 2016 11:42:00 -0500

Welcome to my new website! This website is still hosted on Github but now I am using Jekyll which allows me to update the website more easily and add more more pages and info about stuff that I am doing.

Ava Chow

Building and Syncing Bitcoin 0.5.0 in 2022

Building and Syncing Bitcoin 0.5.0 in 2022

First Sync Attempt

Second Attempt

Third Attempt

Fourth Attempt

Conclusion

URI Argument Injection Vulnerability in Bitcoin Core 0.18 and Earlier

URI Argument Injection Vulnerability in Bitcoin Core 0.18 and Earlier

Background

URI Handlers

URI Argument Injection

Injecting Arguments into Bitcoin Core

Dangerous Command Line Arguments

URI Argument Injection Mitigations

Bitcoin Core’s Fix

Demonstration

CVE

Timeline

Announcing the Bitcoin Core Usage Survey

Announcing the Bitcoin Core Usage Survey

Goals

Results

Privacy

Survey Breakdown

Meta and Screener Questions

Currently Node Operator Questions

Previously Operated Node Questions

Currently Use the Wallet Questions

Previously Used the Wallet Questions

Never Used the Wallet Questions

Never Used Bitcoin Core Questions

Acknowledgements

What's Coming To The Bitcoin Core Wallet in 0.21

What’s Coming To The Bitcoin Core Wallet in 0.21

Upcoming Features

Descriptor Wallets

The Problem with Legacy Wallets

Benefits of Descriptor Wallets

SQLite Database Backend

The problem with Berkeley DB

Why SQLite

No Longer Creating The Default Wallet

Compatibility

Atomic Swaps Across Bitcoin Chain Splits

Atomic Swaps Across Bitcoin Chain Splits

Updates

Requirements

The Method

The Multisig Address

The Spending Transactions

Backing out

Fork Day Events

An Example with Segwit2x

Software For This Example

Acknowledgements

Bitcointalk Account Name Change

Bitcointalk Accounts Name Change

Bitcoin Core Troubleshooting FAQ and Tips

Bitcoin Core Troubleshooting FAQ and Tips

Frequently Asked Questions

A transaction is stuck, how do I make it confirm?

Wallet Balance is incorrect

Wallet is empty

Troubleshooting tips

Enter commands into the debug console

Start Bitcoin Core with a startup option

Windows

Linux

Mac OSX

Check if the private key to an address is in the wallet

Get the debug.log file

Announcing the Bitcoin Fork Monitor

Updates

The Bitcoin Fork Monitor

How it works

The source code

See also

Clearing the FUD around Segwit