* update dictionary validation to validate chainID

* bug fixes

* get evmChainId only one

* Bug fix

Co-authored-by: Scott Twiname <[email protected]>

* Fix unwrapping dictionary response

Co-authored-by: Scott Twiname <[email protected]>

* add header to manifest network

* Improve validation of runner name

* add all search params to connection headers

Co-authored-by: Scott Twiname <[email protected]>

* add unfinalizedBlock service

* add unit tests for unfinalized blocks

* modify log levels of best and finalised blocks

* fix: incomplete ds data in same block (#1370)

Co-authored-by: Orion <[email protected]>

* [SKIP CI] Prerelease

* Best Block (#1308)

* Draft

* update changes

* rebase changes

* fix

* fix (#1329)

* disable best blocks for workers until we support it

* Rename bestBlock to unfinalizedBlocks, other clean up

* Clean up

* Further clean up

* Dedupe reindex function

* Fix tests

* Clean up

* Update checking finalization to use parent hash

* Rename logger

* Refactor unfinalized blocks

* Use header rather than full block, improve detecting forks

* Verify unfinalized blocks when disabled.

Use sorted array for storing unfinalized blocks

* Clean up logs

* Fix not indexing unfinalized blocks right away, exit if historical not enabled

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Fix bugs with unfinalized blocks (#1374)

* Fix bugs with unfinalized blocks

- Fix using wrong hash for unfinalized blocks
- Fix not removing unfinalized blocks when the latest unfinalized block < finalized

* Fix tests getting stuck

* Fix issue finding where fork occurred

* [SKIP CI] Prerelease

* [release] 20221028 (#1372)

* hot fix tests (#1360)

* update tests: api.service.spec, init-controller.test, publish-controller.spec

* update tests following comments

* [SKIP CI] Prerelease

* add ethereum to CLI and Validator (#1378)

* Imporve dictionary query (#1371)

* wip

* wip

* wip

* broken wip

* wip refactoring

* seperate dictionaryQueryEntries

* fix logic for setDicitonaryQueryEntries

* cleaning up

* move dictionaryEntry.ts to node-core

* remove comments

* create class for dictionaryqueryentries

* update tests

* move dictionaryQueryService into dicitonaryService

* refactor

* update logic for useDicitonary

* relocated dictionaryQuery funcs

* add test for dictquerymap

* tests failing on windows?

* test getDictionaryQueryEntries

* test fixed for getDictioanryQueryEntries

* added test for sorting

* conflict fixed ?

* fix conflict_2

* fix conflict_3

* add delete temp ds records back

* clean up with new logic

* clean up, add comments for test

* fix

* update logic

* add generic type

* [SKIP CI] Prerelease

* Enable for better inheritance of generated entity modeld (#1377)

* refactor: enable for entity inheritance

* refactor: include also filed getters

* [SKIP CI] Prerelease

* fix comments issue with new package (#1380)

* fix comments issue with new package

* moved yaml package

* [SKIP CI] Prerelease

* fix logic with reindex and unfinalized height and dynamic ds (#1382)

* fix logic with reindex and unfinalized height

* fix

* include fix for #1379

* update polkadot to 9.7.1 (#1384)

* [release] 20221107

* Fix remove alter table (#1387)

* remove migrate alter table

* remove

* [SKIP CI] Prerelease

* [release] 20221108 (#1388)

* fix missing sequelize sync (#1389)

* [SKIP CI] Prerelease

* [release] 20221108 patch (#1390)

* reindex bind (#1391)

* [SKIP CI] Prerelease

* [release] 20221109 (#1393)

* Handle fetch errors, then retry (#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (#1394)

* Fix tests hanging (#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (#1410)

* [SKIP CI] Prerelease

* fix row estimate (#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: HZ <[email protected]>

* Fix missing speChanged logic with getRuntime (#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (#1375)

* [SKIP CI] Prerelease

* remove enums (#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (#1430)

* [SKIP CI] Prerelease

* fix missing query (#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (#1433)

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* update

Co-authored-by: Scott Twiname <[email protected]>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (#1484)

* [SKIP CI] Prerelease

* update fetch.module dictionary service

* add bypassBlocks

Co-authored-by: hariu-starfish <[email protected]>
Co-authored-by: Orion <[email protected]>
Co-authored-by: Jay Ji <[email protected]>
Co-authored-by: Scott Twiname <[email protected]>
Co-authored-by: Naveen V <[email protected]>
Co-authored-by: Filippo <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: HZ <[email protected]>
Co-authored-by: MOZGIII <[email protected]>
Co-authored-by: James Bayly <[email protected]>
Co-authored-by: Mingyang Li <[email protected]>

* Ensure addresses are checksum addresses

* Add a test for api

* Fixes

---------

Co-authored-by: JQQQ <[email protected]>

* Check queue for lower blocks before processing fetched blocks

* Update logging

* get address from ds.options

* add whitelist validation

* add EthereumProcessorOptions

* add forbidNonWhitelisted validation only to filter

* dictionary query bug fix

* Example of faster dictionary queries

* Update node core, tidy up

* Remove commented code

* Add query limit for num of filter address

* fix missing check 0

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* Fix not working with polygon and other chains without POS

* Lazy load tx receipts

…#40)

* add some extra profiling and print to console

* Hook up some worker events

* Improve rpc retry logic by using built in ethers logic

* Add support for http keep alive

* Use batch provider

* Hide meta logging behind profiler flag

* Join promise.all 2 metadata store updates

* Cache function and event signatures

---------

Co-authored-by: Ian He <[email protected]>

* fix project template could undefined

* Update packages/node/src/indexer/dynamic-ds.service.ts

Co-authored-by: Scott Twiname <[email protected]>

---------

Co-authored-by: Scott Twiname <[email protected]>

* bump dependencies

* update for node

* Sync with main sdk workers branch

* Tidy up

* Update node-core

* Fix json stringify block content

* Drop http request batch size to 10

* sync with main sdk store cache 0.0.4 (#47)

* Add dns cache lookup to http agents (#46)

* Fix reindex, remove unsynced code (#48)

* Ethereum store sync (#49)

* Update node core

* Fix dictionary validation

* Update node core

* Sync with latest SDK to add support for multiple endpoints

* Modify yarn.lock

* Update node code

* update subql/node-core to storeCache-0.0.2

---------

Co-authored-by: Jay Ji <[email protected]>

* [release] major release

* update changelog

* Fix dynamic ds issues, tests

* Fix loading receipts, use non-batch provider for historical requests

* update node-core, remove count feature

---------

Co-authored-by: JQQQ <[email protected]>

* Improve error handling with 429 errors

* Throw error when failing to fetch block

* check result from json-rpc-batch request is array

* create fetchBlocks in api.service

* use fetchBlocks from apiService for workers

* use fetchBlockGeneric from node-core

* [SKIP CI] Prerelease

* fix comments issue with new package (#1380)

* fix comments issue with new package

* moved yaml package

* [SKIP CI] Prerelease

* fix logic with reindex and unfinalized height and dynamic ds (#1382)

* fix logic with reindex and unfinalized height

* fix

* include fix for #1379

* update polkadot to 9.7.1 (#1384)

* [release] 20221107

* Fix remove alter table (#1387)

* remove migrate alter table

* remove

* [SKIP CI] Prerelease

* [release] 20221108 (#1388)

* fix missing sequelize sync (#1389)

* [SKIP CI] Prerelease

* [release] 20221108 patch (#1390)

* reindex bind (#1391)

* [SKIP CI] Prerelease

* [release] 20221109 (#1393)

* Handle fetch errors, then retry (#1386)

* add retryOnFail function

* add retryOnFail

* add test, fix logic

* [SKIP CI] Prerelease

* fix (#1395)

* [SKIP CI] Prerelease

* [release] 20221109 node-core (#1394)

* Fix tests hanging (#1396)

* Fix tests hanging

* Update base docker image with newer git version

* [SKIP CI] Prerelease

* Add distinct query plugin (#1274)

* Add distinct query plugin

* Clean up log

* Fix distinct not being provided to query

* Uppercase enum to be consistent with other enums

* Update dictionary queries to try distinct argument

* [SKIP CI] Prerelease

* Add query distinct dependencies (#1398)

* fix missing update forked graphile dependencies

* tidy up

* tidy up

* [SKIP CI] Prerelease

* Break block dispatcher file up and move common code to base class (#1397)

* [SKIP CI] Prerelease

* Hot schema trigger (#1401)

* implement trigger with notification

* working prior clean up

* refactor and clean up on async and await

* clean up

* clear comments

* add filter

* fix

* fix err

* [SKIP CI] Prerelease

* [release] 20221115 (#1402)

* [release] 20221115

* [release] 20221115

* [release] 20221115

* fix hot schema (#1404)

* fix and refactor

* refactor getTriggers

* [SKIP CI] Prerelease

* [release] 20221115 (#1408)

* [release] 20221115

* [release] 20221115

* fix fetchblock for works (#1410)

* [SKIP CI] Prerelease

* fix row estimate (#1417)

* fix row estimate

* tidy up

* [SKIP CI] Prerelease

* Add support to cli/validator for flare (#1416)

* [SKIP CI] Prerelease

* missing validator missing ipfs chainTypes (#1419)

* [SKIP CI] Prerelease

* query explain include graphql query (#1426)

* [SKIP CI] Prerelease

* dictionary auth link integration (#1411)

* add required dependancies and add flag for feature

* update modules and add logic to dictionary

* update dictionary parameters, update tests, remove hardcoded values

* simplify auth dictionary handling

* add apollo link type, remove un-needed from wrapper

* remove dictionary from indexer module

* update flag name, add error check

* update yargs flag description

* Update appolo-links

* update @subql/apollo-links version, remove apollo/client version change

* fix yarn.lock

* add await init to dictionary tests

* fix fetch test dictionary provider, update yarn.lock

Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: HZ <[email protected]>

* Fix missing speChanged logic with getRuntime (#1421)

* init

* need fix worker

* fix test

* tidy up

* fix test, fix missing dictionary

* inject service to runtime

* remove unused code

* [SKIP CI] Prerelease

* Feat/multi chain indexing (#1375)

* [SKIP CI] Prerelease

* remove enums (#1427)

* [SKIP CI] Prerelease

* change to print graphql sql on request (#1428)

* [SKIP CI] Prerelease

* fix missing util-crptyo in util package (#1429)

* fix missing util-crptyo in util package

* need prerelease in query service

* [SKIP CI] Prerelease

* fix import issue in query service (#1430)

* [SKIP CI] Prerelease

* fix missing query (#1432)

* [SKIP CI] Prerelease

* change fetch module SubqueryProject => ISubqueryProject (#1433)

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* Update polkadot 9.9.1 (#1434)

* update polkadot api, also fix force-clean dependencies issue

* tidy up

* [SKIP CI] Prerelease

* [release] 20221123 (#1431)

* [release] 20221123

* Update packages/query/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/validator/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* Add a way to pass settings to the graphql playground (#1436)

* Add a way to pass settings to the graphql playground

* Add a changelog entry

* [SKIP CI] Prerelease

* Fix the changelog entry for the last PR (#1437)

* [SKIP CI] Prerelease

* fix enum name missing issue (#1441)

* fix enum name missing issue

* tidy up

* [SKIP CI] Prerelease

* hash sql queries (#1438)

* hashed sql

* add comments

* add comment for hashName function

* update function

* update hashName function

* [SKIP CI] Prerelease

* [release] 20221130 (#1442)

* [release] 20221130

* [release] 20221130

* Add --file flag for codegen (#1446)

* init

* change flag behaviour so -f can overwrite -l

* update getManifestPath method, fix typo

* update argument description

* update variable names

* update arg description

* add error message on bad --file path

Co-authored-by: Lachlan McCrae <[email protected]>

* [SKIP CI] Prerelease

* add try and catch for hsr (#1449)

* [SKIP CI] Prerelease

* Bypass blocks (#1435)

* wip

* manifest error

* manfiest passed

* added logging for bypassing blocks

* sync main, moved bypassBlocks under networks

* pass all tests

* mock dictioanry case

* fix metadata

* clean up

* allow range

* bypass working, needs more testing

* wip on bypass with setting new bufferHeight

* working

* added runtime test

* add test for latestBufferHeight (blocked)

* removed unneeded test, improve logic

* clean up

* add comments

* add comments for latestBufferHeight reasoning

* update logic

* refactor

* update logic

* refactor

* [SKIP CI] Prerelease

* [release] 20221205 (#1448)

* [release] 20221205

* [release] 20221205

* [release] 20221206

* details for bypassBlocks

* Remove unused RuntimeService from indexer module, it had missing dependencies (#1453)

* [SKIP CI] Prerelease

* Fix/subcommands (#1451)

* updated forceClean

* update reindex

* [SKIP CI] Prerelease

* Enable env vars parsing at node and query commands (#1452)

* [SKIP CI] Prerelease

* Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458)

* add flag for hot-schema reload on node, ensure query is on main thread

* remove flags

* [SKIP CI] Prerelease

* Add start height to project (#1456)

* [SKIP CI] Prerelease

* Fix poi offset is 0 (#1459)

* [SKIP CI] Prerelease

* Handle when fields got undefined with historical (#1463)

* [SKIP CI] Prerelease

* update deploy cli (#1460)

* [SKIP CI] Prerelease

* exit on fail (#1464)

* [SKIP CI] Prerelease

* fix parentSpecVersion could be undefined (#1467)

* [SKIP CI] Prerelease

* [release] 20221219 (#1466)

* [release] 20221219

* update

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/cli/CHANGELOG.md

Co-authored-by: Scott Twiname <[email protected]>

* update

Co-authored-by: Scott Twiname <[email protected]>

* Fix templates with other sdks, make -f flag work like other commands (#1470)

* Fix templates with other sdks, make -f flag work like other commands

* Fix for sdks with first class smart contracts

* [SKIP CI] Prerelease

* Tweak error message for genesis hash (#1471)

* [SKIP CI] Prerelease

* Fix trigger function (#1469)

* fix trigger and function for subscription and hot schema reload

* update

* [SKIP CI] Prerelease

* [release] 20221222 (#1474)

* validate dictionary with start height (#1473)

* validate dictionary with start height

* Update packages/node/src/indexer/fetch.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* add count for entity (#1480)

* add count for entity

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <[email protected]>

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Fixing readme doc for @subql/query (#1485)

* doc: correcting grammer

* fix: typo

* [SKIP CI] Prerelease

* update polkadot api to 9.11.1 (#1483)

* update polkadot api to 9.11.1

* update

* [SKIP CI] Prerelease

* add connection filters (#1484)

* [SKIP CI] Prerelease

* fix workers missing set last buffered height from empty dictionary (#1492)

* [SKIP CI] Prerelease

* Fix custom metadata keys not being applied (#1496)

* add block hash validation after fetch (#1494)

* add block hash validation after fetch

* add tests

* skip fetch test

* Update log

* [SKIP CI] Prerelease

* [release] 20230123 (#1497)

* Add support for near projects to the CLI (#1498)

* Add support for near projects to the CLI

* Update common near version

* Update near packages

* [SKIP CI] Prerelease

* Arranges commands in alphabetical order (#1495)

* [SKIP CI] Prerelease

* user process.env.USERPROFILE for windows platform (#1502)

* [SKIP CI] Prerelease

* add blockTime to NodeConfig (#1501)

* [SKIP CI] Prerelease

* [release] 20230131 (#1503)

* Worker use runtime service (#1491)

* draft

* update

* tidy up

* update

* working worker runtimes

* into base class

* clean up

* improve base on review

* [SKIP CI] Prerelease

* Check queue for lower blocks before processing fetched blocks (#1509)

* [SKIP CI] Prerelease

* Support array type in dictionary queries (#1510)

* [SKIP CI] Prerelease

* add flag query-limit (#1514)

* [SKIP CI] Prerelease

* Fix error with tempDsRecords being undefined (#1516)

* Fix error with tempDsRecords being undefined

* Fix schema not being used for metadata, sync dynamic-ds across workers

* Be smarter about reloading dynamic ds in workers

* [SKIP CI] Prerelease

* Update postgraphile enable table partitions (#1520)

* [SKIP CI] Prerelease

* `dictionary-optimisation` flag  (#1519)

* add flag, to orderby blockheight instead of pk

* add comment

* [SKIP CI] Prerelease

* [WIP] Feature/adding postgres ssl connection option (#1513)

* enable ssl option postgres

* Check DB_SSL Env Value , default is false

* adding ENV to pass in the Path to certificates

* adding SSL option for query pacakge.

* use args to pass the postgres ssl informations.

* load file content moved to comman package.

* Update packages/node-core/src/configure/NodeConfig.ts

---------

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Support Cockroach db (#1521)

* support cockroach db

* Update comment

* [SKIP CI] Prerelease

* ensure pgpool connection work with ssl (#1525)

* [SKIP CI] Prerelease

* Fix issue when use cockroach query service not support pgPartition (#1526)

* [SKIP CI] Prerelease

* [release] 20230221 (#1528)

* fix issue store getByField limit could potentially excess config limit (#1529)

* fix issue store getByField limit could potentially excess config limit

* Update packages/node-core/src/indexer/store.service.ts

Co-authored-by: Scott Twiname <[email protected]>

* typo

---------

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Abi codegen (#1532)

* draft

* fix abi name, add support for transaction

* Update packages/cli/src/template/abi-interface.ts.ejs

Co-authored-by: Scott Twiname <[email protected]>

* fix

* fix dependencies

---------

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Move enum under schema (#1527)

* need fix query get enum

* fix enum query, clean all enums

* add logic to handle original enum type

* fix check exclude constrain check missing schema and table (#1537)

* [SKIP CI] Prerelease

* manual prerelease node for node-core update (#1538)

* [SKIP CI] Prerelease

* resolution for sequelize and prerelease node (#1539)

* [SKIP CI] Prerelease

* fixed the version of sequelize and prerelease node (#1540)

* [SKIP CI] Prerelease

* improve force clean with deprecated enum (#1541)

* [SKIP CI] Prerelease

* Remove exclude constraint (#1543)

* deprecate exclude contraint

* remove unused

* [SKIP CI] Prerelease

* [release] 20230306 (#1544)

* [release] 20230307 (#1545)

* bump yarn.lock, update imports (#1546)

* [SKIP CI] Prerelease

* [release] 20230307 (#1547)

* increase wsProvider timeout (#1550)

* increase wsProvider timeout

* remove .mmr dir

* [SKIP CI] Prerelease

* Use performance.now instead of date for profiler. (#1549)

* add SmartBatchService and BlockSizeBuffer (#1506)

* add SmartBatchService and BlockSizeBuffer

* check memory usage before fetching a batch

* use v8 heap usage statistics

* set block to null after indexing

* extend smart batch service to workers

* make smart batch size injectible

* rebase

* bug fixes

* code cleanup

* assign workers based on free memory

* fix liniting

* cleanup code

* cleanup logs

* clear blocksizebuffer inside smartbatchservice

---------

Co-authored-by: Scott Twiname <[email protected]>

* Fix build issue with profiler

* [SKIP CI] Prerelease

* fix abi codegen runtime validation (#1552)

* [SKIP CI] Prerelease

* [release] 20230309 Cli (#1553)

* Fix enum under schema not being escaped (#1555)

* [SKIP CI] Prerelease

* rename `--sponsored-dictionary` to `--dictionary-resolver` (#1559)

* renamed flag

* add missing tpyo

* [SKIP CI] Prerelease

* handle bigint in json.stringify (#1562)

* [SKIP CI] Prerelease

* add validation for `codegen`, grapql conflicts (#1564)

* add test for codegen with reserved keys

* wip validateEntityName fn

* update test, update logc

* remove logs

* [SKIP CI] Prerelease

* Fix codegen when abi is not valid (#1570)

* Fix codegen when abi is not valid

* Update packages/cli/src/controller/codegen-controller.ts

Co-authored-by: Scott Twiname <[email protected]>

---------

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* Fix block offset been reset to 1 (#1571)

* [SKIP CI] Prerelease

* remove blocking in process queueing (#1572)

* remove blocking in process queueing

* redefine locking pattern

* fix fetchedBlocks status

* [SKIP CI] Prerelease

* improve api error handling (#1576)

* [SKIP CI] Prerelease

* generate sourcemap for projects (#1569)

* improve error handling

* use inline-source-map

* revert api error handling

* beautify minified output from webpack

* code cleanup

* move terser plugin dependency to cli

* log warnings for unresolved stack trace

* get entry file from datasource

* alter convertStack prototype

* add ES2020 to lib compiler option

* update polkadot api to 10.1.4 (#1580)

* update polkadot api to 10.1.4

* update rest of dep

* [release] 20230329 (#1587)

* Multiple endpoints (#1551)

* allow multiple endpoints

* add response time based load balancing

* handle endpoint disconnects

* move fetchBlockBatches to apiService

* cleanup code

* move ConnectionPoolService to node-core

* fix order of module injection in indexermodule

* remove manually disconnecting api

* move api creation code to ApiConnection class

* code cleanup

* handle reconnection error

* change networkEndpoints to networkEndpoint

* change to use yarn build (#1589)

* test prerelease build (#1590)

* [SKIP CI] Prerelease

* [release] 20230330 (#1592)

* Upgrade `apollo-links` (#1593)

* [SKIP CI] Prerelease

* improve blockSize function (#1602)

* improve blockSize function

* add return type to blockSize

Co-authored-by: Scott Twiname <[email protected]>

---------

Co-authored-by: Scott Twiname <[email protected]>

* [SKIP CI] Prerelease

* [release] 20230405 (#1603)

* Add evmChainId to metadata (#1607)

* [SKIP CI] Prerelease

* [release] 20230411 query (#1608)

* Update progress string (#1612)

* [SKIP CI] Prerelease

* Update the jsonField directive to have an option for disabling the index (#1613)

* Update the jsonField directive to have an option for disabling the index

* Clean up

* [SKIP CI] Prerelease

* [release] 20230414 utils (#1615)

* Fix workers fetching blocks out of order (#1616)

* Fix workers fetching blocks out of order

* Remove logging endpoints with api keys

* [SKIP CI] Prerelease

* [release] 20230417 (#1617)

* DB cache and various other improvements (#1561)

* store cache

* improve on flush cache

* draft storeCache service

* update

* minor change

* Updates

* add historical data to cache (#1557)

* add __block_range to creation

* add historical record to cache

* correct range

* Complete cached store methods (#1563)

* add __block_range to creation

add historical record to cache

correct range

cache store methods

start add tests

* fix set

* update

* WIP cache with workers support

* Fix up dynamic ds service, clean up

* Update store serialisation over workers

* Improve unfinalized blocks to work without tx

* Move all chain agnostic code from block dispatchers to node core

* Add comments about worker store conversion

* Fix rebase build issues

* Implement cache for metadata entity (#1565)

* Implement cache for metadata entity

* Clean up and exclude unfinalized blocks keys from metadata cache

* Clean up

* flush with cache stored records size (#1566)

* flush with cache stored records size

* update

* Lfu get cache (#1568)

* still need fix get/count logic

fix method for LFU getCache

* Update tests

* Fix and more tests

* Update base on review

* Fix

* Run all store operations through cache (#1567)

* Run all store operations through hash

* Log when cache flushed, flush cache on application shutdown

* Cache fixes

* Fix removeCache not being init

* POI cache fixes

* Fixes

* Fix caches being cleared after db operation and potential cache changes have happened

* Fix issues with injected store

* Clean up

* Fix tests from rebase

* Fix get cache not being used

* Rename function

* Fix reindex issues

* Fix casing of import

* Clean up, log if finalized is enabled

* parent 8b0b558649bf40a4d6b9dd882cf3a775b3bd185a (#1574)

author Scott Twiname <[email protected]> 1679452749 +1300
committer JQQQ <[email protected]> 1679461182 +1300

Clean up, log if finalized is enabled

Fix rebase build issues

parent 5e4ea5d5b6b81241377cb2394c5da204403b883b
author Scott Twiname <[email protected]> 1679013784 +1300
committer JQQQ <[email protected]> 1679460224 +1300

parent 5e4ea5d5b6b81241377cb2394c5da204403b883b
author Scott Twiname <[email protected]> 1679013784 +1300
committer JQQQ <[email protected]> 1679460210 +1300

parent 5e4ea5d5b6b81241377cb2394c5da204403b883b
author Scott Twiname <[email protected]> 1679013784 +1300
committer JQQQ <[email protected]> 1679460149 +1300

parent 5e4ea5d5b6b81241377cb2394c5da204403b883b
author Scott Twiname <[email protected]> 1679013784 +1300
committer JQQQ <[email protected]> 1679460101 +1300

parent 5e4ea5d5b6b81241377cb2394c5da204403b883b
author Scott Twiname <[email protected]> 1679013784 +1300
committer JQQQ <[email protected]> 1679460038 +1300

Run all store operations through hash

Log when cache flushed, flush cache on application shutdown

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix removeCache not being init

Fixes

Fix issues with injected store

Clean up

Fix tests from rebase

Fix get cache not being used

tidy up

Co-authored-by: Scott Twiname <[email protected]>

* Update metadata when dictionary skips large number of blocks (#1577)

* Update metadata when dictionary skips large number of blocks

* Bring back option to log sql

* Dynamic cache threshold and metrics (#1578)

* Bring back fetch service tests

* Fix build issues

* Tidy up

* Fix workers historical disabled (#1579)

* Fix imports in node core

* improve close record sql, and add flag max size for getCache (#1583)

* improve close record sql, and add flag max size for getCache

* Fix

* escape id string

* Flush data at specific block heights (#1582)

* Flush data at specific block heights

* Add test

* Fix value fromBelowHeight being empty

* Fix mergedRecords potentially being empty

* Fix flushableRecordCounter

* Change log to debug

* Fix dictionary validation

* fix reindex storeCache missing EventEmitterModule (#1585)

* Make flushing cache configurable (#1586)

* Add an index to Id field on entities with historical (#1588)

* Fix tests

* Fix possible infinite loop with cache and fix test (#1591)

* Fix possible infinite loop with cache and fix test

* Add some comments

* Fix more tests

---------

Co-authored-by: JQQQ <[email protected]>

* Connect historical option up to cache model option (#1596)

* Only run delete query if there is data to delete

* Add support for non-historical indexing with cache (#1597)

* Add support for non-historical indexing with cache

* Fix api test

* remove constraintDeferrableQuery from hasOne

* fix createUniqueIndexQuery

---------

Co-authored-by: JQQQ <[email protected]>

* Cache and worker related fixes (#1598)

* fix index name too long issue (#1599)

* hash the indexes, except pk indexes

* handle existed indexes

* update

* Remove store count function (#1600)

* support cockroach db with non-historical (#1604)

* support cockroach db with non-historical

* Add more comments

* move fetchBlocks to node-core (#1611)

* move fetchBlocks to node-core

* code cleanup

* code cleanup

* remove extra args from fetchBlock

* Fix making data serializable for store bulkCreate and bulkUpdate (#1622)

* Improve big int error in workers (#1619)

* improve error handling in workers

* Make bigint stringifiable in worker and main threads

---------

Co-authored-by: Scott Twiname <[email protected]>

* fix type error (#1623)

* fix type error

* cast type

* Flush store cache in operational order (#1614)

* Init

* better log message

* tidy up

* Tidy up and test

* Update packages/node/src/yargs.ts

Co-authored-by: Scott Twiname <[email protected]>

* Update packages/node-core/src/indexer/storeCache/storeCache.service.ts

Co-authored-by: Scott Twiname <[email protected]>

* add benchmarking

* update benchmarking

* Tidy up

* test

* remove flag, use last flushed index, more tidy up

* make operationIndex required

* fix test

---------

Co-authored-by: Scott Twiname <[email protected]>

* update default values

* fix missing indexes on cockroach db (#1624)

* fix getLatest() could be undefined

* Fix StoreOperation not pushing enums

---------

Co-authored-by: JQQQ <[email protected]>
Co-authored-by: Naveen V <[email protected]>

* [SKIP CI] Prerelease

* Testing Framework (#1584)

* implement runTest

* check handler kind for running test

* implement testing service

* run tests in sandbox

* build test files through subql build

* run handlers in sandbox

* cleanup sandbox script and improve logging

* index data with `IndexerManager.indexData`

* use indexBlock() instead of indexData() for tests

* add getter _name to entities during codegen

* code cleanup

* improve logging

* improve logging

* bug fixes

* code cleanup

* improve logging

* indexBlock: get datasource from arguments

* move testingService to node-core

* code cleanup

* update ci

* code cleanup

* cleanup

* cleanup

* cleanup

* handle 0 test cases

* prevent calling getGqlType for 'contains' matcher (#1626)

* prevent calling getGqlType for 'contains' matcher

* get gqlType with ternary condition

* remove indexer module from testing module (#1628)

* remove indexer module from testing module

* cleanup

* cleanup

* Enable strict ts setting for node core (#1625)

* WIP enable strict ts setting for node core

* fix undefined in historicalMarkPreviousHeightRecordsBatch

* typo

* Update typescript, fix most strict related errors

* Strict changes for testing framework

* Runtime fixes

* change log level on cache poi

---------

Co-authored-by: JQQQ <[email protected]>

* fix prerelease (#1629)

* [SKIP CI] Prerelease

* [SKIPCI] fix testing publish order (#1630)

* fix missing @subql/testing in node-core (#1631)

* [SKIP CI] Prerelease

* manual prerelease for node-core fix (#1632)

* [SKIP CI] Prerelease

* Improve yargs descriptions (#1633)

* [SKIP CI] Prerelease

* Support tests from ./test and ./tests dirs, fix store blockheight issue

* Remove log

* [SKIP CI] Prerelease

* [release] 2.0.0 (#1627)

* release-2.0.0

* update

* update node changelog

* remove typescript from query dependency

* remove typescript from node dependency

* [SKIPCI] fix release

* [release] 2.0.0 republish (#1635)

* Update common, tidy up

---------

Co-authored-by: Filippo <[email protected]>
Co-authored-by: Ben <[email protected]>
Co-authored-by: Jay Ji <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: Lachlan McCrae <[email protected]>
Co-authored-by: HZ <[email protected]>
Co-authored-by: MOZGIII <[email protected]>
Co-authored-by: James Bayly <[email protected]>
Co-authored-by: Mingyang Li <[email protected]>
Co-authored-by: seandotau <[email protected]>
Co-authored-by: Naveen V <[email protected]>
Co-authored-by: Darwin Subramaniam <[email protected]>

* remove polkadot/util from common dependency

* update yarn.lock

* Fix api not being defined, fix not using filter on datasources (#1647)

* Force flush cache with reindex command (#1645)

* deprecate local mode (#1648)

* Update node core

---------

Co-authored-by: Jay Ji <[email protected]>

* fix fetch dictionary meta log

* fix order