Add nep11 tracker #651

Ashuaidehao · 2021-10-09T08:15:16Z

Close #542
Add nep11 tracker.

src/RpcServer/RpcServer/config.json

roman-khimov

It technically closes #542, so it should be mentioned somewhere.

src/RpcNep11Tracker/Storage/Nep11Balance.cs

roman-khimov · 2021-10-12T14:19:07Z

src/RpcNep11Tracker/RpcNep11Tracker.cs

+        }
+
+        [RpcMethod]
+        public JObject GetNep11Transfers(JArray _params)


Just FYI, take a look at NeoGo extensions for getnep17transfers (limits and paging): https://github.com/nspcc-dev/neo-go/blob/master/docs/rpc.md#limits-and-paging-for-getnep17transfers

We'll make them available for NEP-11 tracking also, but maybe C# node can make use of them too.

src/RpcNep11Tracker/RpcNep11Tracker.cs

roman-khimov · 2021-10-12T14:39:40Z

src/RpcNep11Tracker/RpcNep11Tracker.cs

+
+            ResetBatch();
+
+            ushort transferIndex = 0;


Why do we have it block-level instead of execution context (transaction, basically) level?

Nep11TransferKey will use it to distinguish different transfers with same parameters.

I see. We've changed NEP-17 tracker storage scheme long time ago and it's no longer needed for this. This data at the same time just seems a little awkward, it's returned in transfernotifyindex and I have no idea what user can do with it. Maybe we can keep it internal then, not return in JSON? To me, more valuable thing to have here would be transaction-level index (so that we could getapplicationlog and see the original notification event for this transfer).

And BTW, if it's block-level now then ushort might be a bit too short, technically the protocol allows for 64K transactions in a block (and maybe even more after neo-project/neo#2300, 64K was a hard limit around neo-project/neo#1937) and every transaction can emit a number of events.

src/RpcNep11Tracker/RpcNep11Tracker.cs

superboyiii · 2021-10-19T02:59:25Z

Test Pass
From my side, it solved current requirements of showing Nep11 tx history (sorted by timestamp) and current NFTs in specific address with group by contract hash. But still need @roman-khimov and @shargon review again.

…into add-nep11

shargon · 2021-10-19T12:09:00Z

src/RpcNep11Tracker/Storage/Nep11BalanceKey.cs

+    {
+        public readonly UInt160 UserScriptHash;
+        public readonly UInt160 AssetScriptHash;
+        public ByteString Token;


Is not better to use Nep17BalanceKey or join both in NepBalanceKey with extras, instead of having two plugins ?

Do you mean merge nep17 and nep11 tracker?

I think that it could me done easily, but what do you think?

It will be a heavy plugin, @superboyiii what do you think?

I think no need to merge. Nep11 Tracker and Nep17 Tracker are different requirements. As I know many exchanges only support Nep17, no requirements for Nep11 as well as some wallets.

src/RpcNep11Tracker/Storage/Nep11Balance.cs

src/RpcNep11Tracker/RpcNep11Tracker.cs

src/RpcNep11Tracker/Storage/Nep11BalanceKey.cs

src/RpcNep11Tracker/Storage/Nep11Transfer.cs

src/RpcNep11Tracker/RpcNep11Tracker.cs

…into add-nep11

src/RpcNep11Tracker/RpcNep11Tracker.csproj

superboyiii · 2021-10-22T08:41:34Z

Test Pass:
neoSystem.LoadStore works well after fixed.

…into add-nep11

superboyiii · 2021-10-28T08:20:56Z

Test PASS
[SupportedStandards("NEP-11")], [SupportedStandards("nep-11")], [SupportedStandards("Nep-11")].... these could successfully be tracked. Others like [SupportedStandards("nep11")] will fail.

roman-khimov · 2021-10-28T08:36:02Z

I'd actually only allow NEP-11, see neo-project/proposals#139, we should have exactly one proper way of writing this for contracts to follow. Otherwise we'll inevitably have a lot of variations among different contracts and everyone using this data will have to implement support for all of them.

superboyiii · 2021-10-28T09:21:08Z

I'd actually only allow NEP-11, see neo-project/proposals#139, we should have exactly one proper way of writing this for contracts to follow. Otherwise we'll inevitably have a lot of variations among different contracts and everyone using this data will have to implement support for all of them.

neo-project/proposals#142

vncoelho · 2021-10-28T15:24:12Z

@Ashuaidehao, the consensus and nodes stopped syncing when testing this PR.

superboyiii · 2021-11-01T03:10:50Z

@Ashuaidehao, the consensus and nodes stopped syncing when testing this PR.

Could you tell us more details? I can't replay your issue.

vncoelho · 2021-11-01T12:19:27Z

@superboyiii,

testing the following config:

# https://github.com/neo-project/neo-node/commits/master
NEO_CLI_BLOCKCHAIN_URL=https://github.com/neo-project/neo-node.git
NEO_CLI_BRANCH=master
NEO_CLI_COMMIT=8153553dcd8d68c752b9988ce7d7b9e360529b65

# https://github.com/neo-project/neo/commits/master
NEO_BLOCKCHAIN_URL=https://github.com/Qiao-Jin/neo.git
NEO_BLOCKCHAIN_BRANCH=taskmanager_optimization
NEO_BLOCKCHAIN_COMMIT=456fe0242001eec2ce92b44fbba84c5d5e9393b1

# https://github.com/neo-project/neo-modules/commits/master
NEO_PLUGINS_URL=https://github.com/neo-project/neo-modules.git
NEO_PLUGINS_BRANCH=master
NEO_PLUGINS_COMMIT=d0f143ac43fe6d4496430c01f737f3dd9f232904

Worked.
Now, when changing to:

# https://github.com/neo-project/neo-modules/commits/master
NEO_PLUGINS_URL=https://github.com/Ashuaidehao/neo-modules.git
NEO_PLUGINS_BRANCH=add-nep11
NEO_PLUGINS_COMMIT=7556a43

It stopped synced and stuck on changeview.

In fact, it looks like it is because the *.dll that have dependency on RpcServer can not be on the folder if the node does not have RpcServer enabled.
Since two out of four of our CNs do not have RpcServer, I noticed that I had to also delete it before starting the node:

if [[ ${IS_RPC_SERVER} = "0" ]]; then
	echo "Deleting RPCServer related plugins"
	rm /opt/node/neo-cli/Plugins/RpcServer.dll
	rm /opt/node/neo-cli/Plugins/RpcNep17Tracker.dll
	rm /opt/node/neo-cli/Plugins/RpcNep11Tracker.dll
	rm /opt/node/neo-cli/Plugins/ApplicationLogs.dll
	rm /opt/node/neo-cli/Plugins/StateService.dll	
	sleep 1
fi

In this sense, the problem is more generic than this PR,it is a result of not having the RpcServer.
Perhaps the *.dll could be in the folder and we could verify if the config file is created or node, something like that.
Anyway, the problem was on our side here by not deleting the * .dll for some nodes.

superboyiii · 2021-11-03T09:49:40Z

In this sense, the problem is more generic than this PR,it is a result of not having the RpcServer. Perhaps the *.dll could be in the folder and we could verify if the config file is created or node, something like that. Anyway, the problem was on our side here by not deleting the * .dll for some nodes.

It's an issue about depedency check on neo-core. Since all RpcServer required plugins have this issue, we could make it into another PR to check dependency before onload. Currently, for this PR I think it's OK. What do you think?

shargon · 2021-11-03T11:46:03Z

src/RpcNep11Tracker/RpcNep11Tracker.cs

+                Console.WriteLine($"Fault:{record.from},{record.to},{record.tokenId.GetSpan().ToHexString()}");
+                return;
+            }
+            Put(Nep11BalancePrefix, new Nep11BalanceKey(record.to, record.asset, record.tokenId), new Nep11Balance() { Balance = engine.ResultStack.Pop().GetInteger(), LastUpdatedBlock = _currentHeight });


Is not easy to deny the service if the contract is prepared to return a wrong result? it will deny the service to any node that will have this plugin, I think that it should log the error but not deny the service, imagine if a consensus node has this plugin.

https://github.com/neo-project/neo-vm/blob/b48113f8dc787d059324c2d34c855f77b0e3645d/src/neo-vm/Types/StackItem.cs#L116

Please check #671

superboyiii · 2021-11-26T03:21:08Z

I will close this since it's replaced by #671

Ashuaidehao added 3 commits September 15, 2021 01:06

init

94c9304

nep11

c7e97ff

Merge https://github.com/neo-project/neo-modules into add-nep11

18c0443

superboyiii reviewed Oct 12, 2021

View reviewed changes

src/RpcServer/RpcServer/config.json Outdated Show resolved Hide resolved

mainnet

0fd052a

roman-khimov reviewed Oct 12, 2021

View reviewed changes

Ashuaidehao and others added 7 commits October 13, 2021 10:55

fix

588d7bd

is divisible

d5440c0

Merge branch 'master' into add-nep11

85e1506

rename and sort

5f6e2d8

group balance

2715cef

rename

db0861b

Merge branch 'master' into add-nep11

2ebd447

superboyiii previously approved these changes Oct 19, 2021

View reviewed changes

superboyiii requested a review from shargon October 19, 2021 03:01

Ashuaidehao added 2 commits October 19, 2021 14:17

copyright

b57db78

Merge branch 'add-nep11' of https://github.com/Ashuaidehao/neo-modules …

66b5657

…into add-nep11

Ashuaidehao dismissed superboyiii’s stale review via 66b5657 October 19, 2021 06:18

format

501fce9

shargon reviewed Oct 19, 2021

View reviewed changes

Clean file

db6fa0d

shargon reviewed Oct 19, 2021

View reviewed changes

roman-khimov reviewed Oct 19, 2021

View reviewed changes

src/RpcNep11Tracker/RpcNep11Tracker.cs Outdated Show resolved Hide resolved

Ashuaidehao added 3 commits October 20, 2021 11:04

update limit

f5394dc

Merge branch 'add-nep11' of https://github.com/Ashuaidehao/neo-modules …

f5c63ea

…into add-nep11

fix network

9fe7098

steven1227 mentioned this pull request Oct 20, 2021

NEP-11 properties method inconsistency neo-project/proposals#141

Open

devhawk reviewed Oct 20, 2021

View reviewed changes

src/RpcNep11Tracker/RpcNep11Tracker.csproj Outdated Show resolved Hide resolved

remove leveldb

51dad29

Merge branch 'master' into add-nep11

025392c

superboyiii and others added 3 commits October 25, 2021 10:20

Merge branch 'master' into add-nep11

7cc8de4

format

d28912c

Merge branch 'add-nep11' of https://github.com/Ashuaidehao/neo-modules …

2a3e1a5

…into add-nep11

superboyiii previously approved these changes Oct 25, 2021

View reviewed changes

check SupportedStandards

8c527f5

Ashuaidehao dismissed superboyiii’s stale review via 8c527f5 October 27, 2021 07:18

superboyiii and others added 5 commits October 27, 2021 16:39

Merge branch 'master' into add-nep11

014b4f9

check nep11

e9285fa

Merge branch 'add-nep11' of https://github.com/Ashuaidehao/neo-modules …

72697d2

…into add-nep11

ignore case

219df5a

Merge branch 'master' into add-nep11

f2cdfcd

update

7556a43

superboyiii previously approved these changes Nov 3, 2021

View reviewed changes

superboyiii mentioned this pull request Nov 3, 2021

Neo v3.1.0 Checklist neo-project/neo#2620

Closed

13 tasks

Clean files

e74622b

shargon dismissed superboyiii’s stale review via e74622b November 3, 2021 11:42

shargon requested changes Nov 3, 2021

View reviewed changes

Ashuaidehao mentioned this pull request Nov 4, 2021

add tokens tracker #671

Merged

superboyiii closed this Nov 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nep11 tracker #651

Add nep11 tracker #651

Ashuaidehao commented Oct 9, 2021 •

edited by superboyiii

Loading

roman-khimov left a comment

roman-khimov Oct 12, 2021

roman-khimov Oct 12, 2021

Ashuaidehao Oct 18, 2021

roman-khimov Oct 19, 2021

roman-khimov Oct 20, 2021

superboyiii commented Oct 19, 2021 •

edited

Loading

shargon Oct 19, 2021

Ashuaidehao Oct 20, 2021

shargon Oct 20, 2021

Ashuaidehao Oct 20, 2021

superboyiii Oct 21, 2021

superboyiii commented Oct 22, 2021

superboyiii commented Oct 28, 2021 •

edited

Loading

roman-khimov commented Oct 28, 2021

superboyiii commented Oct 28, 2021

vncoelho commented Oct 28, 2021

superboyiii commented Nov 1, 2021

vncoelho commented Nov 1, 2021 •

edited

Loading

superboyiii commented Nov 3, 2021 •

edited

Loading

shargon Nov 3, 2021

Ashuaidehao Nov 4, 2021

superboyiii commented Nov 26, 2021

Add nep11 tracker #651

Add nep11 tracker #651

Conversation

Ashuaidehao commented Oct 9, 2021 • edited by superboyiii Loading

roman-khimov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superboyiii commented Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superboyiii commented Oct 22, 2021

superboyiii commented Oct 28, 2021 • edited Loading

roman-khimov commented Oct 28, 2021

superboyiii commented Oct 28, 2021

vncoelho commented Oct 28, 2021

superboyiii commented Nov 1, 2021

vncoelho commented Nov 1, 2021 • edited Loading

superboyiii commented Nov 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

superboyiii commented Nov 26, 2021

Ashuaidehao commented Oct 9, 2021 •

edited by superboyiii

Loading

superboyiii commented Oct 19, 2021 •

edited

Loading

superboyiii commented Oct 28, 2021 •

edited

Loading

vncoelho commented Nov 1, 2021 •

edited

Loading

superboyiii commented Nov 3, 2021 •

edited

Loading