-
Notifications
You must be signed in to change notification settings - Fork 8
2. Database
The daemon tracks each logical FAT chain in its own sqlite3 database. The databases are designed such that they are self-contained and independent of one another. This means that it is possible to copy a single chain database out of the database folder and any instance of fatd can completely validate the correctness of that single chain database and operate on it independently. After validating a database, the latest EBlock KeyMR can be looked up against a factomd node to completely confirm the correctness of the database.
You will note that package db does not make any network calls. This is by design. The engine is responsible for making network calls to factomd to retrieve the correct blockchain data to pass to the db package to sync databases.
Chain databases are named after their hex encoded Chain ID and use the ".sqlite3" suffix.
The exact schema of a chain database can be found in db/schema.go.
Chain databases include the following tables:
-
address_transactions-- Relates addresses with the transactions they are involved in -
addresses-- Addresses and their balances -
eblocks-- Raw data, key MR, and timestamp of allfactom.EBlocks -
entries-- Raw data, hash and timestamp of allfactom.Entries -
metadata-- Single row table with Chain metadata such as Issuer Identity -
nf_token_address_transactions-- [FAT-1] View relatingfactom.NFTokenswith address_transactions -
nf_token_transactions-- [FAT-1] Relatesfactom.NFTokenswith transactions -
nf_tokens-- [FAT-1]factom.NFTokens, their owner, and metadata -
nf_tokens_addresses-- [FAT-1] Relatesfactom.NFTokensand addresses that have ever owned them
At the end of the day, all state has to be stored in the database. This inextricably links the database and the state of a given FAT token chain. For this reason, the state transition logic is located in the db package (db/apply.go), not in the respective fat packages. The fat packages include data validation logic, but cannot validate a transaction with respect to the state of the chain. For example, a transaction may have valid structure and signatures, but its final validity depends on whether there are sufficient balances for the inputs. This is something that can only be evaluated in the context of the state of a FAT token chain.
It is impractical to keep the entire state for a chain around in memory. For one, the number of possible addresses is unbounded. Additionally, anything we keep in memory we must keep synced with the database, which increases code complexity. Finally, the sqlite3 database engine itself does a fine job of caching, so there isn't all that much benefit to keeping a subset of the addresses in memory anyway. So we rely on the database to keep track of address balances, NFTokens, Entries, and the relationships between them. There are a few pieces of chain metadata, that don't change very often but may be often read, that we keep in memory. This data gets stored in the db.Chain type and is loaded when we first open the db.Chain database connection.
It is critical that all state transitions are atomic at both the Entry and the EBlock level. This avoids corrupted state and ensures that a fatd instance that encounters an error will always leave the database correct up to the last saved EBlock.
To validate a chain database, we must confirm 3 things:
- The chain is complete. -- No EBlocks or Entries are missing, nor added, up to the last recorded EBlock in the database.
- The chain is correct. -- All Entry and EBlock data is identical to what is stored on the Factom blockchain.
- The FAT state is correct. -- All balances are correct up to the last recorded entry.
We can validate completeness and correctness by validating every EBlock KeyMR and Entry Hash, and then confirming with factomd that it knows about our last recorded KeyMR. It is important to validate the order of Entries while recomputing the KeyMR for each EBlock. In doing so we cryptographically guarantee that the data is correct and complete. For this reason, all EBlocks and Entries, regardless of whether the Entry represents a valid FAT transaction, must be stored.
To validate the FAT state, we have to reset the FAT state tables and re-validate each entry and apply each valid transaction to the state. If the resulting state differs from what was originally recorded, the database was corrupted in some way to begin with.
The validation code can be found in db/validate.go. The daemon uses sqlite3's Sessions feature to record a changeset while re-applying state. Since changesets only record the resulting change, and not any intermediary operations, it serves as a simple diff on the database. If the changeset is empty, then the database did not change after re-applying all entries, thus was not corrupted to begin with.
A proper migration management system for the database schema is still needed.
The database can be examined and manipulated using sqlite3. However, if you alter a chain database in any way, even down to the schema, fatd will detect it and refuse to start.