Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

efficient receipts transfer #339

Merged
merged 76 commits into from
Jan 13, 2025
Merged

efficient receipts transfer #339

merged 76 commits into from
Jan 13, 2025

Conversation

kkowalski-reef
Copy link
Collaborator

@kkowalski-reef kkowalski-reef commented Dec 4, 2024 •
edited
Loading

General ideas:

  • Miners append serialized receipts to jsonl files, a file is equivalent to a "page" of receipts.
  • Page files are append-only.
  • Receipts are always appended to the "current" page.
  • Current page number is determined by global clock so that we don't need to add additional communication about the pages themselves.
  • Current page changes every 5 minutes, so as not to make the pages too big. (this may be tweaked or could be moved to a dynamic config)
  • Validators keep up with receipts by continuously requesting 2 most recent pages - that's 2 to allow for slight desync between their clocks.
  • Validators use Range request header to only retrieve receipts added since their last request - or in most cases get an empty response indicating there's nothing new to serve.
  • When starting up, validators will request all pages for the latest 5 hours - so that they have 2 full cycles worth of data to work with.

@kkowalski-reef
manage.py command for generating random receipts
5d694e8
@kkowalski-reef
debug receipt generator: support random receipt timestamp
5c0c55d
@kkowalski-reef
paged receipts store
b7d739b
@kkowalski-reef
move receipts store code to library
cdb6ca9
@kkowalski-reef
fix jsonl writing
7021b70
@kkowalski-reef
receipt transfer checkpointing
09feff5
@kkowalski-reef
old page compression function + manage.py command
67f61cb
@kkowalski-reef
basic script for single-shot receipt fetching
a5bdb57
@kkowalski-reef
simple multi-miner receipts pull
201a0de
@kkowalski-reef
cleanup
1a1f923
@kkowalski-reef
dev celery start scripts
7a60c62
@kkowalski-reef
enable celery schedule for receipts
aa41557
@kkowalski-reef
rewrite to aiohttp
5a92059
@kkowalski-reef
move more code to library
311f989
@kkowalski-reef
cleanup
8802e86
@kkowalski-reef
reuse http connections; dont forget checkpoints too fast; cleanup & s…
80349b8 
…treamline code
@kkowalski-reef
increase batch size for bulk insert
fa2c779
@kkowalski-reef
another cleanup
ddbccdb
@kkowalski-reef
moved manage.py command to validator; reenable rejecting badly signed…
6471a61 
… receipts
@kkowalski-reef
Merge branch 'master' into COM-253-efficient-receipts-transfer
5ca957e 
# Conflicts:
#	compute_horde/uv.lock
#	validator/app/src/compute_horde_validator/validator/tasks.py
@kkowalski-reef
miner receipts: evict old receipt pages
28a4cb1
@kkowalski-reef
validator runner: add receipt transfer daemon
841ddfc
@kkowalski-reef
receipt transfer: add --interval option
2ea54a6
@kkowalski-reef
receipts transfer: use env var for debug miners
1a741aa
@kkowalski-reef
receipts transfer: ahdn off receipt parsing/validation/signature chec…
9f311fe 
…ks to sync thread
@kkowalski-reef
cleanup
5a7bae0
@kkowalski-reef
remove old receipts queue and tasks
8c692d4
@kkowalski-reef
specify what type of receipt was skipped
9bccba0
@kkowalski-reef
fix generating random receipts + signatures; cleanups
7a7d1a5
@kkowalski-reef
cleanup and format
384d14d
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch from 6d08b42 to bfe0c72 Compare December 12, 2024 20:30
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch 2 times, most recently from f5dd22d to e36ed43 Compare December 12, 2024 20:48
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch from e36ed43 to bc034a3 Compare December 12, 2024 20:51
kkowalski-reef and others added 10 commits December 17, 2024 17:07
@kkowalski-reef
feature flag for receipts transfer
801d836
@kkowalski-reef
use existing aget_config to get dynamic config; add missing support f…
bcd5ec4 
…or disabled transfer feature flag in miner
@kkowalski-reef
add evict as base receipt store method
474e730
@kkowalski-reef
Merge pull request #351 from backend-developers-ltd/COM-253-efficient…
04a0f0e 
…-receipts-transfer-featureflag

feature flag for receipts transfer
@kkowalski-reef
Merge branch 'master' into COM-253-efficient-receipts-transfer
ee1ca86
@kkowalski-reef
Merge branch 'redis-pubsub-channel-layer' into COM-253-efficient-rece…
cc60219 
…ipts-transfer
@kkowalski-reef
fix sync/async thread hopping
e632569
@kkowalski-reef
Merge branch 'master' into COM-253-efficient-receipts-transfer
813177c
@kkowalski-reef
fix missing await
0419358
@kkowalski-reef
Merge branch 'master' into COM-253-efficient-receipts-transfer
3e45c7a 
# Conflicts:
#	compute_horde/pyproject.toml
#	compute_horde/tests/settings.py
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch from e3f0d65 to 3e45c7a Compare January 6, 2025 07:57
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch from 0b03021 to 8d2da9b Compare January 8, 2025 17:00
@kkowalski-reef
metric for successful transfers; comments
745e2d8
@kkowalski-reef
Merge branch 'master' into COM-253-efficient-receipts-transfer
6ed7daa 
# Conflicts:
#	compute_horde/pyproject.toml
#	miner/app/src/compute_horde_miner/settings.py
#	validator/app/src/compute_horde_validator/settings.py
#	validator/envs/runner/data/docker-compose.yml
@kkowalski-reef kkowalski-reef force-pushed the COM-253-efficient-receipts-transfer branch from 25bcb52 to 6ed7daa Compare January 13, 2025 14:04
@kkowalski-reef kkowalski-reef merged commit 9b894f2 into master Jan 13, 2025
13 of 15 checks passed
@kkowalski-reef kkowalski-reef deleted the COM-253-efficient-receipts-transfer branch January 13, 2025 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mpnowacki-reef mpnowacki-reef mpnowacki-reef left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kkowalski-reef @mpnowacki-reef