Origin Subsquid

Design Decisions Notion Document

See docs/ for specific tasks.

Release Checklist

Ensure we don't miss anything on a release by following this checklist.

• Go to github repository and create a release
• Create a new tag following the format v<VERSION_NUMBER> where VERSION_NUMBER is the squid version of the processor (i.e. v54)
• Generate the release notes automatically or fill them manually
• Publish the release
• Wait for processing to complete
• Set to production at https://app.subsquid.io/squids
• Check functionality of front-ends
• Update Grafana Subsquid datasource: https://origindefi.grafana.net/connections/datasources
• Notify in #defi-data if important
• Hibernate previous version(s)
• Delete old versions (keep recently hibernated version as a backup)

Env Options

DEBUG_PERF=true sqd process:oeth     # Run with performance numbers
BLOCK_FROM=18421105 sqd process:oeth # Start processing at block 18421105
BLOCK_TO=18421105 sqd process:oeth   # Process up to block 18421105

Frequent Commands

yarn codegen           # Generate Schema code
sqd typegen            # Generate ABI code
sqd down               # Bring down squid container
sqd up                 # Bring up squid container
sqd process:oeth       # Run OETH processor
sqd process:ousd       # Run OUSD processor
sqd process:mainnet      # Run misc processor
sqd migration:generate # Generate migration (sqd down && sqd up first)

Quickstart

# 0. Install @subsquid/cli a.k.a. the sqd command globally
npm i -g @subsquid/cli

# 1. Retrieve the template
sqd init my_squid_name -t evm
cd my_squid_name

# 2. Install dependencies
npm ci

# 3. Start a Postgres database container and detach
sqd up

# 4. Build and start the processor
sqd process:oeth

# 5. The command above will block the terminal
#    being busy with fetching the chain data,
#    transforming and storing it in the target database.
#
#    To start the graphql server open the separate terminal
#    and run
sqd serve

A GraphiQL playground will be available at localhost:4350/graphql.

Dev flow

1. Define database schema

Start development by defining the schema of the target database via schema.graphql. Schema definition consists of regular graphql type declarations annotated with custom directives. Full description of schema.graphql dialect is available here.

2. Generate TypeORM classes

Mapping developers use TypeORM EntityManager to interact with target database during data processing. All necessary entity classes are generated by the squid framework from schema.graphql. This is done by running yarn codegen command.

NOTE: We don't directly use the sqd codegen command because we generate our schema.graphql file first in the yarn codegen script.

3. Generate database migrations

All database changes are applied through migration files located at db/migrations. squid-typeorm-migration(1) tool provides several commands to drive the process.

## drop create the database
sqd down
sqd up

## replace any old schemas with a new one made from the entities
sqd migration:generate

See docs on database migrations for more details.

4. Import ABI contract and generate interfaces to decode events

It is necessary to import the respective ABI definition to decode EVM logs. One way to generate a type-safe facade class to decode EVM logs is by placing the relevant JSON ABIs to ./abi, then using squid-evm-typegen(1) via an sqd script:

sqd typegen

See more details on the squid-evm-typegen doc page.

Project conventions

Squid tools assume a certain project layout:

• All compiled js files must reside in lib and all TypeScript sources in src. The layout of lib must reflect src.
• All TypeORM classes must be exported by src/model/index.ts (lib/model module).
• Database schema must be defined in schema.graphql.
• Database migrations must reside in db/migrations and must be plain js files.
• sqd(1) and squid-*(1) executables consult .env file for environment variables.

Deploy a new version

• Visit Squid deploy dashboard
• Auth with sqd auth -k sqd_XXX (key is on squid deploy page)
• Update squid.yaml to set the correct version
• Run sqd deploy .
• Make branch for new version (eg v9) and push to origin
• Switch back to main branch

Reset cloud dev version (v999)

Useful if you made a schema change or need to reload data.

• Check squid.yaml to make sure you're on v999
• sqd deploy . --update --hard-reset