txJoinmarket: Twisted Joinmarket

txJoinmarket - Joinmarket on Twisted

Introduction

Warning: This is Alpha code. While it appears to be working, the author is a notorious cowboy.

All txJoinmarket activity is currently on @ghtdak's txjoinmarket branch

First, the name: Many projects which are built on Twisted use the tx prefix to indicate. txJoinmarket uses Twisted, treq and txsocksx.

Why this refactoring of Joinmarket into txJoinmarket. First is the belief that threads are bad... (they really are). I love hardware and the wonderful things we can do with parallelism and I even enjoy playing around with the mechanisms of threading (atomics, spin locks, transactional memory)... but for everyday use, threads are a disaster. I recommend Glyph's Unyielding article for a full explanation.

Asynchronous

Asynchronous programming can be difficult to grasp at first, but one quickly comes to appreciate the power and elegance of the approach. Async on Python is truly amazing as one can write code in a mostly linear style. Google "twisted inlinecallbacks" and there's plenty of results. (one can start here)

Joinmarket is, by necessity, entirely callback driven. Architecturally, refactoring it to use Twisted was straightforward though it required some effort. Twisted has good IRC support so a major change was swapping out irc.py for txirc.py

Threads

Removing the threads in Joinmarket was also straightforward as their use was primarily for timeout management. A good example is the changes to taker.py.

Comms

Http

This commit shows the conversion of a block of http communication code to treq

JsonRpc

JsonRpc has only been partially refactored as there are large call chains involved. Fortunately, a few key calls were trivial as they don't use the return result. Therefore, I could use asynchronous http calls making Joinmarket much faster.

Mulitcast

Multicast capability has been introduced for local communications. In the past, when a client received the http alerts from bitcoind, they had to be rebroadcast locally for other Joinmarket peers on the user's machine. Twisted has support for multicast so clients now listen for both http and multicast updates. If one receives http, it is rebroadcast locally as multicast which is a non-blocking in addition to it being one to many.

ZeroMQ

ZeroMQ has been added to Bitcoin core for broadcast of raw transactions and blocks, as well as their hashes. If enabled, txJoinmarket registers for raw transactions and uses this as alternative to the walletnotify entry in bitcoin.conf. The advantage is avoiding the JsonRPC round trip which is required when using the walletnotify mechanism. If ZeroMQ is used, multicast rebroadcast is disabled though, architecturally, an argument could be made that rebroadcasting locally would be advantageous. This is an open issue and configuration options will evolve.

Software Architecture

Multiple Maker / Taker / Observers

Finally, there was some general housekeeping. The use of global variables for dynamic state has been eliminated allowing multiple Maker / Taker / Observer objects to live in the same process. This is generally useful as well as being great for testing.

Refactoring

Major changes to the project's software organization were undertaken both to be consistent with Twisted as well as a general upgrade. The files themselves follow Exarkun's filesystem structure recommendations. Other specific tricks and code were borrowed from Twisted including mechanisms to manage paths.

Many changes to the core objects have taken place.

• Increased use of interface by inheritance and significantly minimized use of function pointer passing
• Preference for Deferred where applicable
• Removal of all threads. The code can run with 15+ Joinmarket peers in a single thread