fix(audit): Remove Task data from aggregator after a response has been responded or expires. #1004

PatStiles · 2024-09-18T20:52:52Z

This PR:

Removes Task data from aggregator response maps after a response has been responded to or expired.

closes #977

#To Test:
Setup local dev net as per the guide

make anvil_start_with_block_time

 make aggregator_start

make batcher_start_local

(optional:)

make operator_register_and_start

And leave a task sender running until the end of the test

make batcher_send_burst_groth16

Everything should work as normally.

Also, every 120 seconds, you should see the log: Cleaning finalized tasks from maps , with some extra information about which tasks are being removed from the Aggregator. These are the tasks being removed, all older than 10 blocks of age.

In prod this variables change to bigger numbers, but the flow is the same.

Note: in dev, first iteration may be small because of the constraints, but the second iteration should include more tasks.

…respond

MauroToscano

What happens if an answer comes after the batch is closed ? It should probably be tested with more operators

PatStiles · 2024-09-23T05:27:53Z

I ran a local testnet with 3 operators produced. The aggregator did not experience any errors and and successfully removed added task information from the aggregator after successfully responding.

aggregator/internal/pkg/aggregator.go

entropidelic · 2024-09-23T19:43:39Z

Changed this PR to draft since we should prioritize other issues first

MarcosNicolau · 2024-09-27T14:59:38Z

What happens if an answer comes after the batch is closed? It should probably be tested with more operators

The aggregator when receiving the response checks that the batch indeed exists here. Anyway, I tested it on my machine doing:

Registered four operators
Killed one
Sent a bunch of proofs
Wait for the aggregator to respond to the task
Woke the killed operator and when it got the latest task, it sent his response and the aggregator ignored it, logging the following message:

[2024-09-27 11:54:10.904 -03] INFO (pkg/server.go:63) - Locked Resources: Starting processing of Response
[2024-09-27 11:54:10.904 -03] INFO (pkg/server.go:67) - Unlocked Resources: Task not found in the internal map

MarcosNicolau

Everything worked as expected on my machine!

JuArce · 2024-09-27T18:46:02Z

Imo the deletion of the task in the maps should be delayed, so it will accept responses from operators after reaching the quorum

It can be done with a go routine like

go func() {
		time.Sleep(10 * time.Second)
		// Deletion logic
	}()

The deletion is done once the task was verified in Ethereum (so there is a time between the quorum is reached and the transaction is accepted in Ethereum). So I'm not sure if we should add an extra delay in the deletion

aggregator/internal/pkg/aggregator.go

…denial-of-service

aggregator/internal/pkg/aggregator.go

Oppen · 2024-10-03T16:41:58Z

The deletion is done once the task was verified in Ethereum (so there is a time between the quorum is reached and the transaction is accepted in Ethereum). So I'm not sure if we should add an extra delay in the deletion

That sounds to me like what we really want is a kind of queue (jargon for "an array") and a periodic task that checks if we're done with stuff.

EDIT: actually, I think it's simpler than that. We already have indices for each batch. Every once in a while (the appropriate time is probably around the time Ethereum takes to confirm our transactions) we check the oldest (lowest) index, find its Merkle root, and use that and the index to clean the dictionaries. This could be a long-lived goroutine started by main.

…ory-leak-could-lead-to-denial-of-service

aggregator/internal/pkg/aggregator.go

MauroToscano

There is no docs on why a garbage collector is needed. This seems overengineered.

…denial-of-service

Oppen · 2024-10-14T17:21:34Z

aggregator/internal/pkg/aggregator.go

@@ -178,6 +178,8 @@ func NewAggregator(aggregatorConfig config.AggregatorConfig) (*Aggregator, error
 		telemetry:             aggregatorTelemetry,
 	}

+	go aggregator.clearTasksFromMaps(garbageCollectorPeriod, garbageCollectorTasksAge)


This doesn't look like the right place to instantiate the task. I would do it at the call-site. Here we're just asking for a struct to be initialized and returned.

done, with supervisor

uri-99

biased self-approve

JuArce · 2024-10-16T02:39:11Z

aggregator/cmd/main.go

@@ -38,6 +39,9 @@ func main() {
 	}
 }

+const garbageCollectorPeriod = time.Second * 150 //TODO change to time.Day * 1


When should we change it? If we have to change it, it may be a config variable, so we do not need to change code

JuArce · 2024-10-16T02:39:17Z

aggregator/cmd/main.go

@@ -38,6 +39,9 @@ func main() {
 	}
 }

+const garbageCollectorPeriod = time.Second * 150 //TODO change to time.Day * 1
+const garbageCollectorTasksAge = uint64(10)     //TODO change to 2592000, 1 month of blocks


When should we change it? If we have to change it, it may be a config variable, so we do not need to change code

JuArce · 2024-10-16T02:40:24Z

aggregator/cmd/main.go

@@ -38,6 +39,9 @@ func main() {
 	}
 }

+const garbageCollectorPeriod = time.Second * 150 //TODO change to time.Day * 1
+const garbageCollectorTasksAge = uint64(10)     //TODO change to 2592000, 1 month of blocks


Also Age does not tell much more. What does it mean?

JuArce · 2024-10-16T02:41:38Z

aggregator/internal/pkg/aggregator.go

+
+// long-lived gorouting that periodically checks and removes old Tasks from stored Maps
+func (agg *Aggregator) ClearTasksFromMaps(period time.Duration, blocksOld uint64) {
+	defer func() {


Would be nice to document it. At least a brief explanation of how it works and its motivation

JuArce · 2024-10-16T02:43:04Z

aggregator/internal/pkg/aggregator.go

+	lastIdxDeleted := uint32(0)
+
+	for {
+		time.Sleep(period)


Normally we sleep the function after execute the main task, this way we can know easily if the task is being executed correctly (even if it is not deleted anything yet, at least we now it is being executed)

JuArce · 2024-10-16T02:55:30Z

core/chainio/avs_reader.go

+
+// This function is a helper to get a task hash of aproximately nBlocksOld blocks ago
+func (r *AvsReader) GetOldTaskHash(nBlocksOld uint64) (*[32]byte, error) {
+	// r.ChainReader.ethClient.CallContext(context.Background(), &blockNumber, "eth_blockNumber")


Maybe delete this comment?

PatStiles added 2 commits September 18, 2024 17:47

remove responded entries after they have been responded or failed to …

1e56af6

…respond

comment

2514d25

PatStiles self-assigned this Sep 19, 2024

MauroToscano reviewed Sep 19, 2024

View reviewed changes

Oppen reviewed Sep 23, 2024

View reviewed changes

aggregator/internal/pkg/aggregator.go Outdated Show resolved Hide resolved

entropidelic marked this pull request as draft September 23, 2024 19:43

PatStiles added 2 commits September 23, 2024 16:59

move logs outside of mutex

2c92e79

move logs in failure to respond

ea592f4

PatStiles added the audit label Sep 27, 2024

PatStiles marked this pull request as ready for review September 27, 2024 14:18

MarcosNicolau approved these changes Sep 27, 2024

View reviewed changes

wrap deletion logic in go func

3bf3fc0

PatStiles requested review from MauroToscano and Oppen September 30, 2024 13:50

add sleep to go routine

5e23085

uri-99 reviewed Oct 3, 2024

View reviewed changes

aggregator/internal/pkg/aggregator.go Outdated Show resolved Hide resolved

uri-99 and others added 2 commits October 3, 2024 11:34

Update aggregator/internal/pkg/aggregator.go

a32492c

Merge branch 'staging' into 977-aggregator-memory-leak-could-lead-to-…

c17a5ef

…denial-of-service

Oppen reviewed Oct 3, 2024

View reviewed changes

aggregator/internal/pkg/aggregator.go Outdated Show resolved Hide resolved

Oppen reviewed Oct 3, 2024

View reviewed changes

aggregator/internal/pkg/aggregator.go Outdated Show resolved Hide resolved

uri-99 added 5 commits October 3, 2024 15:10

refactor: clearTaskFromMaps function

1c51c9c

refactor: clearTaskFromMaps is sequential and no longer a goroutine

2a031ce

feat: refactor now using map to have a garbage collector

91fad8e

chore: comments

da173bd

chore: cargo.lock

34841b4

uri-99 added 4 commits October 8, 2024 12:52

feat: chan usage

56f9b70

Merge remote-tracking branch 'origin/staging' into 977-aggregator-mem…

b50facc

…ory-leak-could-lead-to-denial-of-service

chore: cargo

ce96d69

fix: agg retasking expired tasks

36cb8df

MauroToscano reviewed Oct 8, 2024

View reviewed changes

aggregator/internal/pkg/aggregator.go Outdated Show resolved Hide resolved

MauroToscano requested changes Oct 8, 2024

View reviewed changes

uri-99 marked this pull request as draft October 8, 2024 20:39

uri-99 added 4 commits October 9, 2024 18:24

refactor: now delete old tasks

56132e8

fix: wip

9d8faeb

Merge branch 'staging' into 977-aggregator-memory-leak-could-lead-to-…

6c709b7

…denial-of-service

fix: cleaner

7391495

uri-99 marked this pull request as ready for review October 14, 2024 15:41

Oppen reviewed Oct 14, 2024

View reviewed changes

uri-99 approved these changes Oct 14, 2024

View reviewed changes

MauroToscano approved these changes Oct 15, 2024

View reviewed changes

refactor: GC has supervisor

ef8e5d6

JuArce requested changes Oct 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(audit): Remove Task data from aggregator after a response has been responded or expires. #1004

fix(audit): Remove Task data from aggregator after a response has been responded or expires. #1004

PatStiles commented Sep 18, 2024 •

edited by uri-99

Loading

MauroToscano left a comment •

edited

Loading

PatStiles commented Sep 23, 2024

entropidelic commented Sep 23, 2024

MarcosNicolau commented Sep 27, 2024 •

edited

Loading

MarcosNicolau left a comment

JuArce commented Sep 27, 2024

Oppen commented Oct 3, 2024 •

edited

Loading

MauroToscano left a comment

Oppen Oct 14, 2024

uri-99 Oct 15, 2024 •

edited

Loading

uri-99 left a comment

JuArce Oct 16, 2024

JuArce Oct 16, 2024

JuArce Oct 16, 2024

JuArce Oct 16, 2024

JuArce Oct 16, 2024

JuArce Oct 16, 2024

fix(audit): Remove Task data from aggregator after a response has been responded or expires. #1004

Are you sure you want to change the base?

fix(audit): Remove Task data from aggregator after a response has been responded or expires. #1004

Conversation

PatStiles commented Sep 18, 2024 • edited by uri-99 Loading

This PR:

MauroToscano left a comment • edited Loading

Choose a reason for hiding this comment

PatStiles commented Sep 23, 2024

entropidelic commented Sep 23, 2024

MarcosNicolau commented Sep 27, 2024 • edited Loading

MarcosNicolau left a comment

Choose a reason for hiding this comment

JuArce commented Sep 27, 2024

Oppen commented Oct 3, 2024 • edited Loading

MauroToscano left a comment

Choose a reason for hiding this comment

Oppen Oct 14, 2024

Choose a reason for hiding this comment

uri-99 Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

uri-99 left a comment

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

JuArce Oct 16, 2024

Choose a reason for hiding this comment

PatStiles commented Sep 18, 2024 •

edited by uri-99

Loading

MauroToscano left a comment •

edited

Loading

MarcosNicolau commented Sep 27, 2024 •

edited

Loading

Oppen commented Oct 3, 2024 •

edited

Loading

uri-99 Oct 15, 2024 •

edited

Loading