Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize safe events indexer #2194

Merged
merged 24 commits into from
Sep 13, 2024
Merged

Optimize safe events indexer #2194

merged 24 commits into from
Sep 13, 2024

Conversation

moisses89
Copy link
Member

@moisses89 moisses89 commented Aug 19, 2024

Closes #2181

Description

This PR reduce the number of queries used to process the SafeSetup and ProxyCreation events, taking the advantage of commonly this two events are together in the same transaction or close in a batch of blocks.
Is important to highlight if the first event is SafeSetup will be always together with a ProxyCreation event because is a result of call DeployProxy with initializer.

Benchmark

The following sctipt was used to compare the results between the previous implementation and the implementation of this PR.

        SafeMasterCopy("0x29fcB43b46531BcA003ddC8FCB67FFE91900C762").save()
        SafeMasterCopy("0x3E5c63644E683549055b9Be8653de26E0B4CD36E").save()
        addresses = set(SafeMasterCopy.objects.all().values_list("address", flat=True))
        indexer = SafeEventsIndexerProvider()

        start_block = 1799327
        relevant = indexer.find_relevant_elements(
            addresses, start_block, start_block + 100
        )
        start = time.time()
        indexer.process_elements(relevant)
        print(f"Total time for {len(relevant)} {time.time() - start}")

Also used the same chain and RPC in both chains.

Results

Previous implementation:

2024-08-23 08:51:03,575 [INFO] [MainProcess] SafeEventsIndexer: Found 9288 events from block-number=1799327 to block-number=1799427 for 2 addresses
2024-08-23 08:51:11,350 [INFO] [MainProcess] Opening connection to Redis
Total time for process decoded elements 47.67031955718994
Total time for 9288 55.48148560523987

Current implementation:

2024-08-23 08:47:00,677 [INFO] [MainProcess] SafeEventsIndexer: Found 9288 events from block-number=1799327 to block-number=1799427 for 2 addresses
2024-08-23 08:47:10,637 [INFO] [MainProcess] Opening connection to Redis
2024-08-23 08:47:11,468 [INFO] [MainProcess] Inserted 9288 internal_txs 
2024-08-23 08:47:11,468 [INFO] [MainProcess] Inserted 4644 internal_decoded_txs
Total time for process decoded elements 4.111443758010864
Total time for 9288 11.770971059799194

@moisses89 moisses89 changed the title Process safe creation first Optimize safe events indexer Aug 20, 2024
# ProxyCreation first and SafeSetup later
proxy_creation_event = events[1]
else:
# Proxy was created in previous blocks.
Copy link
Member Author

@moisses89 moisses89 Aug 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous implementation always tried to remove the InternalTx generated from ProxyCreation just in case the ProxyCreation came first. https://github.com/safe-global/safe-transaction-service/blob/main/safe_transaction_service/history/indexers/safe_events_indexer.py#L366
In such case, because the ProxyCreation never comes later, the InternalTx generated by SafeSetup leaves without singleton (to) forever.
Maybe would be better try to get the Proxy InternalTx from database to insert the correct parameters of SafeSetup what do you think? @safe-global/core-api
In the current implementation the ProxyCreation is storing the singleton address in the to field of InternalTx, not sure if is completely right.

@moisses89 moisses89 marked this pull request as ready for review August 23, 2024 09:06
@moisses89 moisses89 requested a review from a team as a code owner August 23, 2024 09:06
Comment on lines 591 to 592
InternalTx.objects.bulk_create(internal_txs)
InternalTxDecoded.objects.bulk_create(internal_decoded_txs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if they are already inserted?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raise an exception.
Ignoring conflicts in such case would fix the issue, 5eeabd4

Copy link
Member Author

@moisses89 moisses89 Sep 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Uxio0 ignore_conflicts fail because InternalDecodedTx have relation with internal_tx, basically the ORM try to prevent lose data.
We can use instead of that update_conflicts, but I'm concerned because I just discovered that bulk_create does not call save, and in that case post_save will not be fired.
Will this affect our signals handle?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bulk_create does in case we inherit our BulkCreateMixin

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a workaround in case that would be a collision inserting the same instance, shouldn't be any common because the bulk create is in atomic section and we are checking if the contract were indexed, basically if the setup and proxyevent were correctly inserted, shouldn't reach the insert section if would be indexed again.

@moisses89 moisses89 merged commit 8f96b0b into main Sep 13, 2024
6 checks passed
@moisses89 moisses89 deleted the optimize_process_elements branch September 13, 2024 10:31
@github-actions github-actions bot locked and limited conversation to collaborators Sep 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize ´_process_decoded_element´
2 participants