Exercise 1

Write the is_coinbase method of the Tx class.

ScriptSig

The coinbase transaction has no previous output that it’s spending, so the input is not unlocking anything. So what’s in the ScriptSig?

The ScriptSig of the coinbase transaction is set by whoever mined the transaction. The main restriction is that the ScriptSig has to be at least 2 bytes and no longer than 100 bytes. Other than those restrictions and BIP0034 (described in the next section), the ScriptSig can be anything the miner wants as long as the evaluation of the ScriptSig by itself, with no corresponding ScriptPubKey, is valid. Here is the ScriptSig for the genesis block’s coinbase transaction:

4d04ffff001d0104455468652054696d65732030332f4a616e2f32303039204368616e63656c6c
6f72206f6e206272696e6b206f66207365636f6e64206261696c6f757420666f722062616e6b73

This ScriptSig was composed by Satoshi and contains a message that we can read:

>>> from io import BytesIO
>>> from script import Script
>>> stream = BytesIO(bytes.fromhex('4d04ffff001d0104455468652054696d6573203033\
2f4a616e2f32303039204368616e63656c6c6f72206f6e206272696e6b206f66207365636f6e64\
206261696c6f757420666f722062616e6b73'))
>>> s = Script.parse(stream)
>>> print(s.cmds[2])
b'The Times 03/Jan/2009 Chancellor on brink of second bailout for banks'

This was the headline from the Times of London on January 3, 2009. This proves that the genesis block was created some time at or after that date, and not before. Other coinbase transactions’ ScriptSigs contain similarly arbitrary data.

BIP0034

BIP0034 regulates the first element of the ScriptSig of coinbase transactions. This was due to a network problem where miners were using the same coinbase transaction for different blocks.

The height is interpreted as a little-endian integer and must equal the height of the block (that is, the number of blocks since the genesis block). Thus, a coinbase transaction cannot be byte-wise the same across different blocks, since the block height will differ. Here’s how we can parse the height from the coinbase transaction in Figure 9-1:

>>> from io import BytesIO
>>> from script import Script
>>> from helper import little_endian_to_int
>>> stream = BytesIO(bytes.fromhex('5e03d71b07254d696e656420627920416e74506f6f\
6c20626a31312f4542312f4144362f43205914293101fabe6d6d678e2c8c34afc36896e7d94028\
24ed38e856676ee94bfdb0c6c4bcd8b2e5666a0400000000000000c7270000a5e00e00'))
>>> script_sig = Script.parse(stream)
>>> print(little_endian_to_int(script_sig.cmds[0]))
465879

A coinbase transaction reveals the block it was in! Coinbase transactions in different blocks are required to have different ScriptSigs and thus different transaction IDs. This rule continues to be needed as it would otherwise allow the duplicate coinbase transaction IDs across multiple blocks.

Exercise 2

Write the coinbase_height method for the Tx class.

Exercise 3

Write the parse method for Block.

Exercise 4

Write the serialize method for Block.

Exercise 5

Write the hash method for Block.

Version

Version in normal software refers to a particular set of features. For a block, this is similar, in the sense that the version field reflects the capabilities of the software that produced the block. In the past this was used as a way to indicate a single feature that was ready to be deployed by the block’s miner. Version 2 meant that the software was ready for BIP0034, which introduced the coinbase transaction block height feature mentioned earlier in this chapter. Version 3 meant that the software was ready for BIP0066, which enforced strict DER encoding. Version 4 meant that the software was ready for BIP0065, which specified OP_CHECKLOCKTIMEVERIFY.

Unfortunately, the incremental increase in version number meant that only one feature was signaled on the network at a time. To alleviate this, the developers came up with BIP0009, which allows up to 29 different features to be signaled at the same time.

BIP0009 requires that 95% of blocks signal readiness in a given 2,016-block epoch (the period for a difficulty adjustment; more on that later in this chapter) before the soft fork feature gets activated on the network. Soft forks that used BIP0009 as of this writing have been BIP0068/BIP0112/BIP0113 (OP_CHECKSEQUENCEVERIFY and related changes) and BIP0141 (Segwit). These BIPs used bits 0 and 1 for signaling, respectively. BIP0091 used something like BIP0009 but with an 80% threshold and a smaller block period, so it wasn’t strictly using BIP0009. Bit 4 was used to signal BIP0091.

>>> from io import BytesIO
>>> from block import Block
>>> b = Block.parse(BytesIO(bytes.fromhex('020000208ec39428b17323fa0ddec8e887b\
4a7c53b8c0a0a220cfd0000000000000000005b0750fce0a889502d40508d39576821155e9c9e3\
f5c3157f961db38fd8b25be1e77a759e93c0118a4ffd71d')))
>>> print('BIP9: {}'.format(b.version >> 29 == 0b001))  
BIP9: True
>>> print('BIP91: {}'.format(b.version >> 4 & 1 == 1))  
BIP91: False
>>> print('BIP141: {}'.format(b.version >> 1 & 1 == 1))  
BIP141: True

The >> operator is the right bit-shift operator, which throws away the rightmost 29 bits, leaving just the top 3 bits. The 0b001 is a way of writing a number in binary in Python.

The & operator is the “bitwise and” operator. In our case, we right-shift by 4 bits first and then check that the rightmost bit is 1.

We shift 1 to the right because BIP0141 was assigned to bit 1.

Exercise 6

Write the bip9 method for the Block class.

Exercise 7

Write the bip91 method for the Block class.

Exercise 8

Write the bip141 method for the Block class.

Previous Block

All blocks have to point to a previous block. This is why the data structure is called a blockchain. Blocks link back all the way to the very first block, or the genesis block. The previous block field ends in a bunch of 00 bytes, which we will discuss more later in this chapter.

The Merkle root encodes all the ordered transactions in a 32-byte hash. We will discuss how this is important for simplified payment verification (SPV) clients and how they can use the Merkle root along with data from the server to get a proof of inclusion in Chapter 11.

Timestamp

The timestamp is a Unix-style timestamp taking up 4 bytes. Unix timestamps are the number of seconds since January 1, 1970. This timestamp is used in two places: for validating timestamp-based locktimes on transactions included in the block and for calculating a new bits/target/difficulty every 2,016 blocks. The locktimes were at one point used directly for transactions within a block, but BIP0113 changed the behavior to not use the current block’s timestamp directly, but the median time past (MTP) of the past 11 blocks.

Bits is a field that encodes the proof-of-work necessary in this block. This will be discussed more in the next section.

Nonce stands for “number used only once,” or n-once. This number is what is changed by miners when looking for proof-of-work.

How a Miner Generates New Hashes

Where does the miner get new numerical dirt to process to see if it satisfies proof-of-work? This is where the nonce field comes in. The miners can change the nonce field at will to change the hash of the block header.

Unfortunately, the 4 bytes or 32 bits of the nonce field (or 2³² possible nonces that a miner can try) is insufficient for the required proof-of-work. Modern ASIC equipment can calculate way more than 2³² different hashes per second. The AntMiner S9, for example, calculates 12 terahashes per second (Th/s). That is approximately 2⁴³ hashes per second, which means that the entire nonce space can be consumed in just 0.0003 seconds.

What miners can do when the nonce field is exhausted is change the coinbase transaction, which then changes the Merkle root, giving miners a fresh nonce space each time. The other option is to roll the version field or use overt ASICBOOST. The mechanics of how the Merkle root changes whenever any transaction in the block changes will be discussed in Chapter 11.

The Target

Proof-of-work is the requirement that the hash of every block header in Bitcoin must be below a certain target. The target is a 256-bit number that is computed directly from the bits field (in our example, e93c0118). The target is very small compared to an average 256-bit number.

>>> from helper import little_endian_to_int
>>> bits = bytes.fromhex('e93c0118')
>>> exponent = bits[-1]
>>> coefficient = little_endian_to_int(bits[:-1])
>>> target = coefficient * 256**(exponent - 3)
>>> print('{:x}'.format(target).zfill(64))  
0000000000000000013ce9000000000000000000000000000000000000000000

We are purposefully printing this number as 64 hexadecimal digits to show how small the number is in 256-bit terms.

A valid proof-of-work is a hash of the block header that, when interpreted as a little-endian integer, is below the target number. Proof-of-work hashes are exceedingly rare, and the process of mining is the process of finding one of these hashes. To find a single proof-of-work with the preceding target, the network as a whole must calculate 3.8 × 10²¹ hashes, which, when this block was found, could be done roughly every 10 minutes. To give this number some context, the best GPU mining card in the world would need to run for 50,000 years on average to find a single proof-of-work below this target.

We can check that this block header’s hash satisfies the proof-of-work as follows:

>>> from helper import little_endian_to_int
>>> proof = little_endian_to_int(hash256(bytes.fromhex('020000208ec39428b17323\
fa0ddec8e887b4a7c53b8c0a0a220cfd0000000000000000005b0750fce0a889502d40508d3957\
6821155e9c9e3f5c3157f961db38fd8b25be1e77a759e93c0118a4ffd71d')))
>>> print(proof < target)  
True

target is calculated above.

We can see that the proof-of-work is lower by lining up the numbers in 64 hex characters:

TG: 0000000000000000013ce9000000000000000000000000000000000000000000

ID: 0000000000000000007e9e4c586439b0cdbe13b1370bdd9435d76a644d047523

Exercise 9

Write the bits_to_target function in helper.py.

Difficulty

Targets are hard for human beings to comprehend. The target is the number that the hash must be below, but as humans, it’s not easy to see the difference between a 180-bit number and a 190-bit number. The first is a thousand times smaller, but from looking at targets, such large numbers are not easy to contextualize.

>>> from helper import little_endian_to_int
>>> bits = bytes.fromhex('e93c0118')
>>> exponent = bits[-1]
>>> coefficient = little_endian_to_int(bits[:-1])
>>> target = coefficient*256**(exponent-3)
>>> difficulty = 0xffff * 256**(0x1d-3) / target
>>> print(difficulty)
888171856257.3206

We already learned that proof-of-work can be calculated by computing the hash256 of the block header and interpreting this as a little-endian integer. If this number is lower than the target, we have a valid proof-of-work. If not, the block is not valid as it doesn’t have proof-of-work.

In Bitcoin, each group of 2,016 blocks is called a difficulty adjustment period. At the end of every difficulty adjustment period, the target is adjusted according to this formula:

The time_differential is calculated so that if it’s greater than 8 weeks, 8 weeks is used, and if it’s less than 3.5 days, 3.5 days is used. This way, the new target cannot change more than four times in either direction. That is, the target will be reduced or increased by four times at the most.

>>> from block import Block
>>> from helper import TWO_WEEKS  
>>> last_block = Block.parse(BytesIO(bytes.fromhex('00000020fdf740b0e49cf75bb3\
d5168fb3586f7613dcc5cd89675b0100000000000000002e37b144c0baced07eb7e7b64da916cd\
3121f2427005551aeb0ec6a6402ac7d7f0e4235954d801187f5da9f5')))
>>> first_block = Block.parse(BytesIO(bytes.fromhex('000000201ecd89664fd205a37\
566e694269ed76e425803003628ab010000000000000000bfcade29d080d9aae8fd461254b0418\
05ae442749f2a40100440fc0e3d5868e55019345954d80118a1721b2e')))
>>> time_differential = last_block.timestamp - first_block.timestamp
>>> if time_differential > TWO_WEEKS * 4:  
...     time_differential = TWO_WEEKS * 4
>>> if time_differential < TWO_WEEKS // 4:  
...     time_differential = TWO_WEEKS // 4
>>> new_target = last_block.target() * time_differential // TWO_WEEKS
>>> print('{:x}'.format(new_target).zfill(64))
0000000000000000007615000000000000000000000000000000000000000000

Note that TWO_WEEKS = 60*60*24*14 is the number of seconds in 2 weeks: 60 seconds × 60 minutes × 24 hours × 14 days.

This makes sure that if it took more than 8 weeks to find the last 2,015 blocks, we don’t decrease the difficulty too much.

This makes sure that if it took less than 3.5 days to find the last 2,015 blocks, we don’t increase the difficulty too much.

Note that we only need the headers to calculate what the next block’s target should be. Once we have the target, we can convert the target to bits. The inverse operation looks like this:

def target_to_bits(target):
    '''Turns a target integer back into bits'''
    raw_bytes = target.to_bytes(32, 'big')
    raw_bytes = raw_bytes.lstrip(b'\x00')  
    if raw_bytes[0] > 0x7f:  
        exponent = len(raw_bytes) + 1
        coefficient = b'\x00' + raw_bytes[:2]
    else:
        exponent = len(raw_bytes)  
        coefficient = raw_bytes[:3]  
    new_bits = coefficient[::-1] + bytes([exponent])  
    return new_bits

Get rid of all the leading zeros.

The bits format is a way to express really large numbers succinctly and can be used with both negative and positive numbers. If the first bit in the coefficient is a 1, the bits field is interpreted as a negative number. Since the target is always positive for us, we shift everything over by 1 byte if the first bit is 1.

The exponent is how long the number is in base 256.

The coefficient is the first three digits of the base 256 number.

The coefficient is in little-endian and the exponent goes last in the bits format.

If the block doesn’t have the correct bits calculated using the difficulty adjustment formula, then we can safely reject that block.

Exercise 12

Calculate the new bits given the first and last blocks of this 2,016-block difficulty adjustment period:

• Block 471744:

  000000203471101bbda3fe307664b3283a9ef0e97d9a38a7eacd88000000000000000000
  10c8aba8479bbaa5e0848152fd3c2289ca50e1c3e58c9a4faaafbdf5803c5448ddb84559
  7e8b0118e43a81d3

• Block 473759:

  02000020f1472d9db4b563c35f97c428ac903f23b7fc055d1cfc26000000000000000000
  b3f449fcbe1bc4cfbcb8283a0d2c037f961a3fdf2b8bedc144973735eea707e126425859
  7e8b0118e5f00474

Exercise 13

Write the calculate_new_bits function in helper.py.