Milestone 1 Support #1

guplersaxanoid · 2023-07-14T08:56:09Z

starter project for testing: https://github.com/DeveloperInProgress/soroban-subql-starter

* update dictionary validation to validate chainID * bug fixes * get evmChainId only one * Bug fix Co-authored-by: Scott Twiname <[email protected]> * Fix unwrapping dictionary response Co-authored-by: Scott Twiname <[email protected]>

* add header to manifest network * Improve validation of runner name * add all search params to connection headers Co-authored-by: Scott Twiname <[email protected]>

* add unfinalizedBlock service * add unit tests for unfinalized blocks * modify log levels of best and finalised blocks

* fix: incomplete ds data in same block (#1370) Co-authored-by: Orion <[email protected]> * [SKIP CI] Prerelease * Best Block (#1308) * Draft * update changes * rebase changes * fix * fix (#1329) * disable best blocks for workers until we support it * Rename bestBlock to unfinalizedBlocks, other clean up * Clean up * Further clean up * Dedupe reindex function * Fix tests * Clean up * Update checking finalization to use parent hash * Rename logger * Refactor unfinalized blocks * Use header rather than full block, improve detecting forks * Verify unfinalized blocks when disabled. Use sorted array for storing unfinalized blocks * Clean up logs * Fix not indexing unfinalized blocks right away, exit if historical not enabled Co-authored-by: Scott Twiname <[email protected]> * [SKIP CI] Prerelease * Fix bugs with unfinalized blocks (#1374) * Fix bugs with unfinalized blocks - Fix using wrong hash for unfinalized blocks - Fix not removing unfinalized blocks when the latest unfinalized block < finalized * Fix tests getting stuck * Fix issue finding where fork occurred * [SKIP CI] Prerelease * [release] 20221028 (#1372) * hot fix tests (#1360) * update tests: api.service.spec, init-controller.test, publish-controller.spec * update tests following comments * [SKIP CI] Prerelease * add ethereum to CLI and Validator (#1378) * Imporve dictionary query (#1371) * wip * wip * wip * broken wip * wip refactoring * seperate dictionaryQueryEntries * fix logic for setDicitonaryQueryEntries * cleaning up * move dictionaryEntry.ts to node-core * remove comments * create class for dictionaryqueryentries * update tests * move dictionaryQueryService into dicitonaryService * refactor * update logic for useDicitonary * relocated dictionaryQuery funcs * add test for dictquerymap * tests failing on windows? * test getDictionaryQueryEntries * test fixed for getDictioanryQueryEntries * added test for sorting * conflict fixed ? * fix conflict_2 * fix conflict_3 * add delete temp ds records back * clean up with new logic * clean up, add comments for test * fix * update logic * add generic type * [SKIP CI] Prerelease * Enable for better inheritance of generated entity modeld (#1377) * refactor: enable for entity inheritance * refactor: include also filed getters * [SKIP CI] Prerelease * fix comments issue with new package (#1380) * fix comments issue with new package * moved yaml package * [SKIP CI] Prerelease * fix logic with reindex and unfinalized height and dynamic ds (#1382) * fix logic with reindex and unfinalized height * fix * include fix for #1379 * update polkadot to 9.7.1 (#1384) * [release] 20221107 * Fix remove alter table (#1387) * remove migrate alter table * remove * [SKIP CI] Prerelease * [release] 20221108 (#1388) * fix missing sequelize sync (#1389) * [SKIP CI] Prerelease * [release] 20221108 patch (#1390) * reindex bind (#1391) * [SKIP CI] Prerelease * [release] 20221109 (#1393) * Handle fetch errors, then retry (#1386) * add retryOnFail function * add retryOnFail * add test, fix logic * [SKIP CI] Prerelease * fix (#1395) * [SKIP CI] Prerelease * [release] 20221109 node-core (#1394) * Fix tests hanging (#1396) * Fix tests hanging * Update base docker image with newer git version * [SKIP CI] Prerelease * Add distinct query plugin (#1274) * Add distinct query plugin * Clean up log * Fix distinct not being provided to query * Uppercase enum to be consistent with other enums * Update dictionary queries to try distinct argument * [SKIP CI] Prerelease * Add query distinct dependencies (#1398) * fix missing update forked graphile dependencies * tidy up * tidy up * [SKIP CI] Prerelease * Break block dispatcher file up and move common code to base class (#1397) * [SKIP CI] Prerelease * Hot schema trigger (#1401) * implement trigger with notification * working prior clean up * refactor and clean up on async and await * clean up * clear comments * add filter * fix * fix err * [SKIP CI] Prerelease * [release] 20221115 (#1402) * [release] 20221115 * [release] 20221115 * [release] 20221115 * fix hot schema (#1404) * fix and refactor * refactor getTriggers * [SKIP CI] Prerelease * [release] 20221115 (#1408) * [release] 20221115 * [release] 20221115 * fix fetchblock for works (#1410) * [SKIP CI] Prerelease * fix row estimate (#1417) * fix row estimate * tidy up * [SKIP CI] Prerelease * Add support to cli/validator for flare (#1416) * [SKIP CI] Prerelease * missing validator missing ipfs chainTypes (#1419) * [SKIP CI] Prerelease * query explain include graphql query (#1426) * [SKIP CI] Prerelease * dictionary auth link integration (#1411) * add required dependancies and add flag for feature * update modules and add logic to dictionary * update dictionary parameters, update tests, remove hardcoded values * simplify auth dictionary handling * add apollo link type, remove un-needed from wrapper * remove dictionary from indexer module * update flag name, add error check * update yargs flag description * Update appolo-links * update @subql/apollo-links version, remove apollo/client version change * fix yarn.lock * add await init to dictionary tests * fix fetch test dictionary provider, update yarn.lock Co-authored-by: Lachlan McCrae <[email protected]> Co-authored-by: HZ <[email protected]> * Fix missing speChanged logic with getRuntime (#1421) * init * need fix worker * fix test * tidy up * fix test, fix missing dictionary * inject service to runtime * remove unused code * [SKIP CI] Prerelease * Feat/multi chain indexing (#1375) * [SKIP CI] Prerelease * remove enums (#1427) * [SKIP CI] Prerelease * change to print graphql sql on request (#1428) * [SKIP CI] Prerelease * fix missing util-crptyo in util package (#1429) * fix missing util-crptyo in util package * need prerelease in query service * [SKIP CI] Prerelease * fix import issue in query service (#1430) * [SKIP CI] Prerelease * fix missing query (#1432) * [SKIP CI] Prerelease * change fetch module SubqueryProject => ISubqueryProject (#1433) Co-authored-by: Lachlan McCrae <[email protected]> * [SKIP CI] Prerelease * Update polkadot 9.9.1 (#1434) * update polkadot api, also fix force-clean dependencies issue * tidy up * [SKIP CI] Prerelease * [release] 20221123 (#1431) * [release] 20221123 * Update packages/query/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/validator/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> Co-authored-by: Scott Twiname <[email protected]> * Add a way to pass settings to the graphql playground (#1436) * Add a way to pass settings to the graphql playground * Add a changelog entry * [SKIP CI] Prerelease * Fix the changelog entry for the last PR (#1437) * [SKIP CI] Prerelease * fix enum name missing issue (#1441) * fix enum name missing issue * tidy up * [SKIP CI] Prerelease * hash sql queries (#1438) * hashed sql * add comments * add comment for hashName function * update function * update hashName function * [SKIP CI] Prerelease * [release] 20221130 (#1442) * [release] 20221130 * [release] 20221130 * Add --file flag for codegen (#1446) * init * change flag behaviour so -f can overwrite -l * update getManifestPath method, fix typo * update argument description * update variable names * update arg description * add error message on bad --file path Co-authored-by: Lachlan McCrae <[email protected]> * [SKIP CI] Prerelease * add try and catch for hsr (#1449) * [SKIP CI] Prerelease * Bypass blocks (#1435) * wip * manifest error * manfiest passed * added logging for bypassing blocks * sync main, moved bypassBlocks under networks * pass all tests * mock dictioanry case * fix metadata * clean up * allow range * bypass working, needs more testing * wip on bypass with setting new bufferHeight * working * added runtime test * add test for latestBufferHeight (blocked) * removed unneeded test, improve logic * clean up * add comments * add comments for latestBufferHeight reasoning * update logic * refactor * update logic * refactor * [SKIP CI] Prerelease * [release] 20221205 (#1448) * [release] 20221205 * [release] 20221205 * [release] 20221206 * details for bypassBlocks * Remove unused RuntimeService from indexer module, it had missing dependencies (#1453) * [SKIP CI] Prerelease * Fix/subcommands (#1451) * updated forceClean * update reindex * [SKIP CI] Prerelease * Enable env vars parsing at node and query commands (#1452) * [SKIP CI] Prerelease * Workers: Fix SequelizeDatabaseError - tuple concurrently updated (#1458) * add flag for hot-schema reload on node, ensure query is on main thread * remove flags * [SKIP CI] Prerelease * Add start height to project (#1456) * [SKIP CI] Prerelease * Fix poi offset is 0 (#1459) * [SKIP CI] Prerelease * Handle when fields got undefined with historical (#1463) * [SKIP CI] Prerelease * update deploy cli (#1460) * [SKIP CI] Prerelease * exit on fail (#1464) * [SKIP CI] Prerelease * fix parentSpecVersion could be undefined (#1467) * [SKIP CI] Prerelease * [release] 20221219 (#1466) * [release] 20221219 * update * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/node/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/node-core/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * Update packages/cli/CHANGELOG.md Co-authored-by: Scott Twiname <[email protected]> * update Co-authored-by: Scott Twiname <[email protected]> * Fix templates with other sdks, make -f flag work like other commands (#1470) * Fix templates with other sdks, make -f flag work like other commands * Fix for sdks with first class smart contracts * [SKIP CI] Prerelease * Tweak error message for genesis hash (#1471) * [SKIP CI] Prerelease * Fix trigger function (#1469) * fix trigger and function for subscription and hot schema reload * update * [SKIP CI] Prerelease * [release] 20221222 (#1474) * validate dictionary with start height (#1473) * validate dictionary with start height * Update packages/node/src/indexer/fetch.service.ts Co-authored-by: Scott Twiname <[email protected]> Co-authored-by: Scott Twiname <[email protected]> * [SKIP CI] Prerelease * add count for entity (#1480) * add count for entity * Update packages/node-core/src/indexer/store.service.ts Co-authored-by: Scott Twiname <[email protected]> Co-authored-by: Scott Twiname <[email protected]> * [SKIP CI] Prerelease * Fixing readme doc for @subql/query (#1485) * doc: correcting grammer * fix: typo * [SKIP CI] Prerelease * update polkadot api to 9.11.1 (#1483) * update polkadot api to 9.11.1 * update * [SKIP CI] Prerelease * add connection filters (#1484) * [SKIP CI] Prerelease * update fetch.module dictionary service * add bypassBlocks Co-authored-by: hariu-starfish <[email protected]> Co-authored-by: Orion <[email protected]> Co-authored-by: Jay Ji <[email protected]> Co-authored-by: Scott Twiname <[email protected]> Co-authored-by: Naveen V <[email protected]> Co-authored-by: Filippo <[email protected]> Co-authored-by: Lachlan McCrae <[email protected]> Co-authored-by: Lachlan McCrae <[email protected]> Co-authored-by: HZ <[email protected]> Co-authored-by: MOZGIII <[email protected]> Co-authored-by: James Bayly <[email protected]> Co-authored-by: Mingyang Li <[email protected]>

* Ensure addresses are checksum addresses * Add a test for api * Fixes --------- Co-authored-by: JQQQ <[email protected]>

jamesbayly · 2023-07-24T01:24:34Z

Minimal Indexer that allows customer to index only events on Soroban
Define new manifest structure and update SubQuery CLI
Create basic documentation for Beta support

stwiname

I've read through the code. I still have some testing to do though

packages/common-soroban/package.json

packages/common-soroban/src/project/models.ts

packages/common-soroban/src/project/project.spec.ts

packages/common-soroban/src/project/versioned/v1_0_0/model.ts

packages/node/src/indexer/fetch.module.ts

stwiname · 2023-07-24T22:51:49Z

packages/node/src/soroban/block.soroban.ts

+        //decode gives control chars around the string, remove them
+        /* eslint-disable no-control-regex */
+        const decodedTopic = Buffer.from(event.topic[i], 'base64')
+          .toString()
+          .replace(/[\u0000-\u001F\u007F-\u009F]/g, '');


Do we know why these exist? I can get you in contact with someone from Soroban if you need to understand why

it's because events contain XDR representation of data

I added a new commit that uses their xdr library to decode the topics and values

stwiname · 2023-07-24T22:52:18Z

packages/node/src/soroban/safe-api.ts

+
+const logger = getLogger('safe.api.soroban');
+
+export default class SafeSorobanProvider extends SorobanServer {


We shouldn't inject the safe api into the sandbox then as it's no use.

We can leave this code here in case at a later stage this becomes possible

stwiname · 2023-07-24T22:56:44Z

packages/node/src/subcommands/testing.service.ts

+    const apiService = testContext.get(ApiService);
+
+    // Initialise async services, we do this here rather than in factories, so we can capture one off events
+    await (apiService as SorobanApiService).init();


Why is there a cast here? Does BaseTestingService not type this properly?

we provide the base ApiService with a factory that returns SorobanApiService

It already seems to be init here https://github.com/subquery/subql-soroban/pull/1/files#diff-bcab2c6418e84f215507df5daf3213a159fc422d9cf05b1cbe1fb91d4308ef54R57

packages/types/package.json

stwiname · 2023-07-25T01:30:51Z

From testing:

It would be nice to have a better error message when failing to fetch blocks. Especially if they are using an endpoint if there is only 24h of history
Workers are broken. I think there's a dependency issue, check the main sdk for fixes
The batch size seems to be 1-2. I had it go to 100 at one point but then it returned back to 1-2
ERROR Having a problem when getting best block AxiosError: Request failed with status code 502 when fully synced. Not sure theres anythign we can do about this.
Metadata genesis hash is the chain id not the genesis
With the starter i get no data indexed. Can we have something more general to see there's data?

guplersaxanoid · 2023-07-25T10:29:32Z

With the starter i get no data indexed. Can we have something more general to see there's data?

I update starter to index transfer events

stwiname · 2023-07-31T02:54:13Z

package.json

+  },
+  "resolutions": {
+    "node-fetch": "2.6.7",
+    "@subql/common": "2.3.1-1"


I think this needs to be removed

stwiname · 2023-07-31T03:03:14Z

packages/node/src/soroban/soroban.server.ts

+const logger = getLogger('soroban-server');
+const DEFAULT_PAGE_SIZE = 100;
+
+export class SorobanServer extends Server {


This is going to lead to a huge request and being very inefficient. We should fetch more pages until we move on to a following block.

stwiname · 2023-07-31T03:05:27Z

packages/node/src/soroban/soroban.server.ts

+        );
+      }
+
+      delete this.eventsCache[maxEventHeight];


If maxEventHeight equals request.startLedger were going to get no results

stwiname · 2023-07-31T03:06:02Z

packages/node/src/soroban/utils.soroban.ts

+import { ApiWrapper } from '@subql/types-soroban';
+
+export function calcInterval(api: ApiWrapper): number {
+  // TODO find a way to get this from the blockchain


Lets remove the comment then

stwiname · 2023-07-31T03:07:38Z

packages/node/src/subcommands/testing.service.ts

+    const apiService = testContext.get(ApiService);
+
+    // Initialise async services, we do this here rather than in factories, so we can capture one off events
+    await (apiService as SorobanApiService).init();


It already seems to be init here https://github.com/subquery/subql-soroban/pull/1/files#diff-bcab2c6418e84f215507df5daf3213a159fc422d9cf05b1cbe1fb91d4308ef54R57

stwiname · 2023-07-31T03:10:00Z

packages/node/src/subcommands/testing.service.ts

+    return testContext.get(TestRunner);
+  }
+
+  async indexBlock(


Other than substrate these all have the same implementation. We should move this to the base service and only substrate sdk can override this method

packages/types/src/global.ts

stwiname · 2023-07-31T03:13:18Z

packages/node/src/soroban/block.soroban.ts

-      case xdr.ScValType.scvSymbol(): {
-        return value.sym().toString();
-      }
+  static decodeScVals(scVal: xdr.ScVal): any {


Does soroban not provide this functionality?

their scValToNative had issues for some datatypes, such as ScvString.

Is it worth contributing back to soroban sdk?

The issue with ScvString was we get a Uint8Array. From the docs, it looks it was intended to be that way:

string -> string IF the underlying buffer can be decoded as ascii/utf8, Uint8Array of the raw contents in any error case

so, we just extract the Uint8Array and decode it from base64 to utf8

stwiname · 2023-08-01T22:00:10Z

packages/node/src/soroban/soroban.server.ts

+    const startLedgerEvents = events.filter(
+      (event) => parseInt(event.ledger) === sequence,
+    );
+    const otherEvents = events.filter(
+      (event) => parseInt(event.ledger) !== sequence,
+    );


https://lodash.com/docs/4.17.15#groupBy

stwiname · 2023-08-01T22:00:13Z

packages/node/src/soroban/soroban.server.ts

+  }
+
+  //fetch events of a ledger from subsequent pages
+  private async fetchAdditionalEvents(


This and getEvents seem way over complicated.

I suggest having a recursive function to get all the events for a block.

Then another that adds a caching layer over the top of it

stwiname · 2023-08-01T22:02:02Z

Tests are failing now. I think there might need to be some changes ported over from the testing framework and the connection pool state

stwiname and others added 30 commits November 17, 2022 15:02

[release] 20221117 (#15)

eecb136

Fix docker publish script

5d869a8

Improve scripts

d7bc683

Debug node docker workflow

76c2708

fix flare workflow step ids

9ab7aaf

[release] 20221118

0540bc9

[release] 20221118

3ac68af

[SKIP CI] Create new dockerfile for flare

225076d

[release] 20221117

aa4bd14

fix filter calls and block txs

d0028a5

[SKIP CI] Prerelease

1da12bc

[release] 20221122

ce571a3

Validate chainIds of dictionary (#17)

12fa1d4

* update dictionary validation to validate chainID * bug fixes * get evmChainId only one * Bug fix Co-authored-by: Scott Twiname <[email protected]> * Fix unwrapping dictionary response Co-authored-by: Scott Twiname <[email protected]>

[SKIP CI] Prerelease

d6e78b1

Fix bin name

bf171ac

[SKIP CI] Prerelease

1f533fd

add header to manifest network (#16)

0212e45

* add header to manifest network * Improve validation of runner name * add all search params to connection headers Co-authored-by: Scott Twiname <[email protected]>

[SKIP CI] Prerelease

ed74ef6

[release]-20221124 (#18)

c89cb0e

Add Unfinalized Block Service (#19)

fc1a805

* add unfinalizedBlock service * add unit tests for unfinalized blocks * modify log levels of best and finalised blocks

[SKIP CI] Prerelease

1398dea

[SKIP CI] Prerelease

2f5fcf6

fix publish missing templates name (#22)

b70c8ec

[SKIP CI] Prerelease

d977a14

Use much more efficent rpc method for getting block receipts (#23)

a803724

[SKIP CI] Prerelease

ad3be07

[release] 20230203 (#24)

52ba931

Ensure addresses are checksum addresses (#25)

695f57c

* Ensure addresses are checksum addresses * Add a test for api * Fixes --------- Co-authored-by: JQQQ <[email protected]>

[SKIP CI] Prerelease

00d0022

jamesbayly changed the title ~~Create soroban indexer to process events~~ Milestone 1 Support Jul 24, 2023

jamesbayly mentioned this pull request Jul 24, 2023

Stellar Soroban - Milestone 1 Support #2

Closed

guplersaxanoid added 5 commits July 24, 2023 04:22

update node-core

9f8a32f

code cleanup

f0fb91e

fix tests

cd8b2ae

remove ws support

df43e15

code cleanup and update license

3c2ba98

stwiname reviewed Jul 24, 2023

View reviewed changes

guplersaxanoid added 5 commits July 25, 2023 04:39

code-cleanup and bug fixes

4f47746

udpate common and remove safeApi

eaef704

improve error message for pruned node

0b2ab64

xdr decode topics and values

7141933

fix tests for filter

7ae5d20

guplersaxanoid added 3 commits July 26, 2023 03:46

update decodeScVal with more ScValType

17daff5

cleanup

61aa6ad

update decodeScVal to return native value by default

9f8ef6b

stwiname reviewed Jul 31, 2023

View reviewed changes

guplersaxanoid added 4 commits July 31, 2023 04:24

code cleanup

37d1ad5

update getEvents to handle pagination

a706301

update node-core

4d552e4

remove indexBlock from testing service

075422d

stwiname reviewed Aug 1, 2023

View reviewed changes

guplersaxanoid added 2 commits August 2, 2023 05:24

make fetching events recursive

c178dae

rebase

3e4f987

stwiname merged commit d4d60e3 into main Aug 3, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Milestone 1 Support #1

Milestone 1 Support #1

guplersaxanoid commented Jul 14, 2023 •

edited

Loading

jamesbayly commented Jul 24, 2023

stwiname left a comment

stwiname Jul 24, 2023

guplersaxanoid Jul 25, 2023

guplersaxanoid Jul 25, 2023

stwiname Jul 24, 2023

stwiname Jul 24, 2023

guplersaxanoid Jul 25, 2023

stwiname Jul 31, 2023

stwiname commented Jul 25, 2023

guplersaxanoid commented Jul 25, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

stwiname Jul 31, 2023

guplersaxanoid Jul 31, 2023

stwiname Aug 1, 2023

guplersaxanoid Aug 1, 2023

stwiname Aug 1, 2023

stwiname Aug 1, 2023

stwiname commented Aug 1, 2023


		const logger = getLogger('safe.api.soroban');

		export default class SafeSorobanProvider extends SorobanServer {

Milestone 1 Support #1

Milestone 1 Support #1

Conversation

guplersaxanoid commented Jul 14, 2023 • edited Loading

jamesbayly commented Jul 24, 2023

stwiname left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stwiname commented Jul 25, 2023

guplersaxanoid commented Jul 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stwiname commented Aug 1, 2023

guplersaxanoid commented Jul 14, 2023 •

edited

Loading