Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mx 16271 indexing incoming tokens #325

Merged
merged 20 commits into from
Feb 12, 2025

Conversation

axenteoctavian
Copy link

@axenteoctavian axenteoctavian commented Jan 27, 2025

Proposed changes

  • defined a main chain elastic config
  • option to enable/disable this indexation
  • read tokens from the SCR generated by incoming tx processor, if the tokens are not in ES will read them from main chain elastic (another optimization is looking at the token prefix)
  • new run type component IndexTokensHandler

Testing procedure

  • integration tests
  • sovereign local setup

@axenteoctavian axenteoctavian self-assigned this Jan 27, 2025
@axenteoctavian axenteoctavian marked this pull request as ready for review January 28, 2025 14:24
# Conflicts:
#	integrationtests/valuesIndex_test.go
@mariusmihaic mariusmihaic self-requested a review February 3, 2025 09:47
cmd/elasticindexer/config/prefs.toml Outdated Show resolved Hide resolved
data/tokens.go Outdated Show resolved Hide resolved
data/tokens.go Show resolved Hide resolved
factory/runType/runTypeComponentsHandler.go Outdated Show resolved Hide resolved
process/elasticproc/check.go Show resolved Hide resolved
process/elasticproc/elasticProcessor.go Show resolved Hide resolved
@@ -1,4 +1,5 @@
IMAGE_NAME=elastic-container
IMAGE_NAME_2=elastic-container-2

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps another name?

scripts/script.sh Outdated Show resolved Hide resolved
docker-compose.yml Outdated Show resolved Hide resolved
integrationtests/consts.go Outdated Show resolved Hide resolved
integrationtests/incomingSCR_test.go Outdated Show resolved Hide resolved
scripts/script.sh Outdated Show resolved Hide resolved
factory/runType/sovereignRunTypeComponentsFactory.go Outdated Show resolved Hide resolved
docker-compose.yml Outdated Show resolved Hide resolved
process/elasticproc/tokens/sovereignIndexTokensHandler.go Outdated Show resolved Hide resolved
process/elasticproc/tokens/sovereignIndexTokensHandler.go Outdated Show resolved Hide resolved
@@ -23,3 +23,9 @@
username = ""
password = ""
bulk-request-max-size-in-bytes = 4194304 # 4MB

[config.main-chain-elastic-cluster]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you can add a comment here to describe for what is used this section of config

@@ -75,6 +79,9 @@ func (mrtc *managedRunTypeComponents) CheckSubcomponents() error {
if check.IfNil(mrtc.rewardTxData) {
return transactions.ErrNilRewardTxDataHandler
}
if check.IfNil(mrtc.indexTokensHandler) {
return transactions.ErrNilIndexTokensHandler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you should move transactions.ErrNilIndexTokensHandler in another package, because it is not relayed with the transactions package

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to dataIndexer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you should rename the file in disalbedIndexTokensHandler or similar

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed

return ""
}

func (sit *sovereignIndexTokensHandler) indexNewTokens(responseTokensInfo []data.ResponseTokenInfoDB, buffSlice *data.BufferSlice) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe you can rename this function in serializeNewTokens

@@ -33,6 +31,8 @@ type ArgsIndexerFactory struct {
UseKibana bool
ImportDB bool
Sovereign bool
ESDTPrefix string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from where do you read the value of this parameter ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added parameter in config.toml

process/elasticproc/interface.go Show resolved Hide resolved
client/mainChainElasticClient.go Outdated Show resolved Hide resolved
.github/workflows/deploy-docker.yml Show resolved Hide resolved
return nil, err
}

func NewMainChainElasticClient(esClient elasticproc.DatabaseClientHandler, indexingEnabled bool) (*mainChainElasticClient, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add check.ifNil for esClient and return error

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

@axenteoctavian axenteoctavian merged commit 7268376 into feat/sovereign Feb 12, 2025
9 checks passed
@axenteoctavian axenteoctavian deleted the MX-16271-indexing-incoming-tokens branch February 12, 2025 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants