Skip to content

Commit

Permalink
Merge pull request #60 from jrakibi/comapctsize-feerate
Browse files Browse the repository at this point in the history 
Add 3 new topics: compactSize, Fee rate
  • Loading branch information
@jrakibi
jrakibi authored Jan 22, 2025
2 parents d9d5a48 + fae7690 commit d89db6a
Show file tree
Hide file tree
Showing 3 changed files with 372 additions and 3 deletions.
160 changes: 159 additions & 1 deletion decoding/compact-size.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,162 @@ images:
parent: "technical-foundation"
---

(coming soon)
Compact Size (also known as CompactSize or var_int) is a variable-length integer encoding used in Bitcoin to efficiently represent numbers while minimizing space usage. It's primarily used in transaction data and network messages to indicate:

- Number of inputs/outputs in a transaction
- Script sizes
- Number of witness elements
- Length of upcoming data fields

## Structure

The encoding uses a prefix byte to determine how to read the subsequent bytes:

<div className="overflow-x-auto">
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
<thead className="bg-orange-100 dark:bg-orange-900">
<tr>
<th className="px-6 py-3 text-left text-sm font-semibold">Prefix</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Format</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Range</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Total Size</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap"><code>≤ 0xFC</code></td>
<td className="px-6 py-4">Direct value</td>
<td className="px-6 py-4">0-252</td>
<td className="px-6 py-4">1 byte</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap"><code>0xFD</code></td>
<td className="px-6 py-4">Next 2 bytes (LE)</td>
<td className="px-6 py-4">253-65,535</td>
<td className="px-6 py-4">3 bytes</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap"><code>0xFE</code></td>
<td className="px-6 py-4">Next 4 bytes (LE)</td>
<td className="px-6 py-4">65,536-4,294,967,295</td>
<td className="px-6 py-4">5 bytes</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap"><code>0xFF</code></td>
<td className="px-6 py-4">Next 8 bytes (LE)</td>
<td className="px-6 py-4">4,294,967,296-2^64-1</td>
<td className="px-6 py-4">9 bytes</td>
</tr>
</tbody>
</table>
</div>

### How to Read It

Let's analyze real Bitcoin transactions to understand compact size encoding:

1. **Transaction with 1 Input**
-> <a href="https://mempool.space/tx/5e6e1a9b4ce3f9f39467d7129e3ecfbe6c81c08dd377aac666fedc9a758fe614" target="_blank">5e6e1a9b4ce3f9f...fedc9a758fe614</a>
- Looking at input count: `01`
- Since 1 < 252, it's a direct value
- Meaning: Transaction has 1 input
- Total bytes used: 1

2. **ScriptSig of 107 bytes**
-> <a href="https://mempool.space/tx/c27c4d2236fce2a7542e024408d7f89b95e50e42a2c3d10be499c3102ccb45ef" target="_blank">c27c4d2236fce2a...99c3102ccb45ef</a>
- Looking at scriptsig size: `6b`
- Since 107 (0x6b) < 252, it's a direct value
- Meaning: ScriptSig is 107 bytes long
- Total bytes used: 1

3. **ScriptPubKey of 4,026 bytes**
-> <a href="https://mempool.space/tx/e411dbebd2f7d64dafeef9b14b5c59ec60c36779d43f850e5e347abee1e1a455" target="_blank">e411dbebd2f7d64...347abee1e1a455</a>
- Looking at scriptpubkey size: `fd ba 0f`
- First byte is 0xFD, so read next 2 bytes
- Next bytes: `ba 0f` (in little-endian)
- Convert from little-endian: `0fba` = 4,026 in decimal
- Meaning: ScriptPubKey is 4,026 bytes long
- Total bytes used: 3

<ExpandableAlert title="Understanding Little-Endian" type="info">
In the third example, we see `ba 0f` instead of `0f ba` because of little-endian encoding.
Think of it as reading the bytes from right to left: `0fba` is the actual number in hexadecimal.
</ExpandableAlert>

<ExpandableAlert title="Tip" type="warning">
You can verify these values yourself by looking at the raw transaction data on the block explorer.
Click the transaction IDs above and look for the "Raw Transaction" section.
</ExpandableAlert>

## Implementation

Here's a Python implementation for reading and writing compact size integers:

<CodeSnippet
code={`def read_varint(s):
'''Reads a variable integer from a stream
The first byte determines the format:
- If < 0xfd: directly contains the number
- If 0xfd: next 2 bytes contain number
- If 0xfe: next 4 bytes contain number
- If 0xff: next 8 bytes contain number
'''
i = s.read(1)[0]
if i == 0xfd:
return int.from_bytes(s.read(2), 'little')
elif i == 0xfe:
return int.from_bytes(s.read(4), 'little')
elif i == 0xff:
return int.from_bytes(s.read(8), 'little')
else:
return i
def encode_varint(i):
'''Encodes an integer as a compact size
Returns bytes object with the encoded number
'''
if i < 0xfd:
return bytes([i])
elif i < 0x10000:
return b'\\xfd' + i.to_bytes(2, 'little')
elif i < 0x100000000:
return b'\\xfe' + i.to_bytes(4, 'little')
else:
return b'\\xff' + i.to_bytes(8, 'little')`}
language="python"
/>

## Common Uses in Bitcoin

1. **Transaction Structure**
- Input count
- Output count
- ScriptSig size
- ScriptPubKey size
- Witness element count

<TransactionCreation enabledFields={["input_count", "output_count", "scriptsig_size", "scriptpubkey_size"]} />

2. **Block Data**
- Number of transactions in a block

3. **Network Messages**
- Length of upcoming message data
- Number of inventory items


<ExpandableAlert title="Warning" type="warning">
The compact size format is part of Bitcoin's consensus rules. Incorrect encoding
or decoding can lead to invalid transactions or network messages.
</ExpandableAlert>

<ExpandableAlert title="Important Notes" type="info">
- Most transactions use single-byte encoding (≤ 252)
- The FF prefix (8-byte numbers) is rarely used in practice
- All multi-byte numbers must be encoded in little-endian format
- This encoding has been part of Bitcoin since its first release
</ExpandableAlert>

This variable-length integer encoding plays a crucial role in keeping transaction and block data compact while maintaining flexibility for larger values when needed.
59 changes: 58 additions & 1 deletion decoding/endianness.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Big-endian is considered more "human-readable" because the data is stored in the

Little-endian stores the **least significant byte** first. This might feel counterintuitive to humans but is more efficient for modern processors.

Using the same number **12345678** (`0x00BC614E`), here’s how it looks in **little-endian**:
Using the same number **12345678** (`0x00BC614E`), here's how it looks in **little-endian**:

<CodeSnippet
language="Hex"
Expand Down Expand Up @@ -146,3 +146,60 @@ Most modern CPUs are little-endian and the network protocols typically use big-e
This duality requires developers to frequently convert between the two formats when working with Bitcoin data.

---

## 4. Working with Endianness in Code

When working with Bitcoin data, you'll frequently need to convert between little-endian and big-endian formats. Here are some common Python functions for handling endianness:

### Little-Endian to Integer

<CodeSnippet
language="python"
code={`def little_endian_to_int(b):
'''Convert little-endian bytes to integer'''
return int.from_bytes(b, 'little')
# Example usage
bytes_data = bytes.fromhex('4E61BC00') # 12345678 in little-endian
number = little_endian_to_int(bytes_data)
print(number) # Output: 12345678`}
/>

### Integer to Little-Endian

<CodeSnippet
language="python"
code={`def int_to_little_endian(n, length):
'''Convert integer to little-endian bytes of specified length'''
return n.to_bytes(length, 'little')
# Example usage
number = 12345678
bytes_data = int_to_little_endian(number, 4)
print(bytes_data.hex()) # Output: 4e61bc00`}
/>

### Common Gotchas

1. **Byte Order Confusion**: When reading transaction IDs or hashes:

<CodeSnippet
language="python"
code={`# INCORRECT - reading txid directly from hex
txid = "4e61bc0000000000000000000000000000000000000000000000000000000000"
# CORRECT - reverse the bytes for proper txid display
txid_bytes = bytes.fromhex(txid)
txid_display = txid_bytes[::-1].hex()`}
/>

2. **Length Specification**: When converting to little-endian, always specify the correct byte length:

<CodeSnippet
language="python"
code={`# For Bitcoin amounts (8 bytes)
amount = int_to_little_endian(12345678, 8) # Correct
amount = int_to_little_endian(12345678, 4) # Wrong - too short!`}
/>

---
156 changes: 155 additions & 1 deletion decoding/fee-rate.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,158 @@ images: ["/bitcoin-topics/static/images/topics/thumbnails/musig-thumbnail.webp"]
parent: "fee-calculation"
---

(coming soon)
When miners select transactions for inclusion in a block, they don't just look at the raw fee amount, they evaluate transactions based on their _fee rate_.

This is how we calculate the fee rate:

<div className="text-center my-4 text-orange-500">
$$\mathbf{Fee\ Rate = \frac{Transaction\ Fee}{Transaction\ Size}}$$
</div>

Miners use the fee rate to maximize their profit by comparing the profitability of including different transactions in their blocks.

For example, consider these two transactions:

1. Transaction A: 50,000 sats fee, 250 vbytes size
2. Transaction B: 30,000 sats fee, 100 vbytes size

**Fee Rates:**
- Transaction A: 200 sats/vbyte (50,000 ÷ 250)
- Transaction B: 300 sats/vbyte (30,000 ÷ 100)

<ExpandableAlert title="Fee Rate Priority" type="info">
Even though Transaction A pays a higher absolute fee, Transaction B is more profitable for miners because it provides more fees per unit of block space.
</ExpandableAlert>

## Fee Rate Units

There are three common ways to measure fee rates:

1. **sats/vbyte (Most Common)**
2. **sats/wu (Internal)**
3. **sats/byte (Legacy)**


<div className="overflow-x-auto">
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
<thead className="bg-orange-100 dark:bg-orange-900">
<tr>
<th className="px-6 py-3 text-left text-sm font-semibold">Unit</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Description</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Usage</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Example</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/vbyte</code></td>
<td className="px-6 py-4">Virtual bytes account for witness discount</td>
<td className="px-6 py-4">Most common unit used by wallets and block explorers</td>
<td className="px-6 py-4">50 sats/vbyte</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/wu</code></td>
<td className="px-6 py-4">Weight units used internally by Bitcoin Core</td>
<td className="px-6 py-4">Internal Bitcoin Core (4M WU block limit)</td>
<td className="px-6 py-4">12.5 sats/wu</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/byte</code></td>
<td className="px-6 py-4">Deprecated since SegWit activation</td>
<td className="px-6 py-4">Legacy software only</td>
<td className="px-6 py-4">50 sats/byte</td>
</tr>
</tbody>
</table>
</div>

## Understanding Virtual Bytes (vBytes)

Virtual bytes (vBytes) are a way to express transaction size that accounts for the SegWit discount. The formula is simple:

<div className="text-center my-4 text-orange-500">
$$\mathbf{Virtual\ Bytes = Transaction\ Weight ÷ 4}$$
</div>

<div className="overflow-x-auto">
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
<thead className="bg-orange-100 dark:bg-orange-900">
<tr>
<th className="px-6 py-3 text-left text-sm font-semibold">Transaction Type</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Weight Units</th>
<th className="px-6 py-3 text-left text-sm font-semibold">Virtual Bytes</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4">Legacy (non-SegWit)</td>
<td className="px-6 py-4">size × 4</td>
<td className="px-6 py-4">Same as raw size</td>
</tr>
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
<td className="px-6 py-4">Native SegWit</td>
<td className="px-6 py-4">(base × 4) + witness</td>
<td className="px-6 py-4">~25% smaller than raw size</td>
</tr>
</tbody>
</table>
</div>

<ExpandableAlert title="Example Calculation" type="info" expandable={true} initialLines={2}>
Consider a SegWit transaction with:
- Base size: 200 bytes
- Witness size: 100 bytes

1. Weight = (200 × 4) + 100 = 900 weight units
2. Virtual Size = 900 ÷ 4 = 225 vBytes

Even though the raw size is 300 bytes (200 + 100), the virtual size is only 225 vBytes thanks to the SegWit discount!
</ExpandableAlert>

## Minimum Relay Fee Rate

Nodes enforce a minimum fee rate to prevent spam:

- Default minimum: 1 sat/vbyte
- Transactions below this rate are typically rejected
- You can check your node's setting:

<CodeSnippet
language="bash"
code={`$ bitcoin-cli getmempoolinfo
{
"loaded": true,
"size": 47293,
"bytes": 32883821,
"usage": 148075552,
"maxmempool": 300000000,
"mempoolminfee": 0.00001000, # 1 sat/vbyte
...
}`}
/>

<ExpandableAlert title="NOTE" type="info">
💡 The minimum relay fee is a policy rule, not a consensus rule. While
theoretically possible, getting a below-minimum fee transaction mined is
extremely unlikely unless you have a direct connection to a miner.
</ExpandableAlert>

## RPC: Current Fee Rates

You can estimate current fee rates using Bitcoin Core:

<CodeSnippet
language="bash"
code={`$ bitcoin-cli estimatesmartfee 6
{
"feerate": 0.00008124, # BTC/kB
"blocks": 6
}`}
/>

Or query popular block explorers:
- mempool.space
- blockstream.info
- bitcoinfees.earn.com

Remember that fee rates are dynamic and can change rapidly based on network demand.

0 comments on commit d89db6a

Please sign in to comment.