services/horizon: Remove --parallel-job-size config parameter used for reingestion. #5484
Conversation
urvisavla
force-pushed
the
5468/remove-paralleljobsize
branch
from
70cd0ae
to
6c9486e
Compare
urvisavla
force-pushed
the
5468/remove-paralleljobsize
branch
from
6c9486e
to
be2f5b4
Compare
| assert.Eventually(t, func() bool { return len(ledgerBuffer.ledgerQueue) == 15 }, time.Second*1, time.Millisecond*50) | ||
| assert.NoError(t, err) | ||
|
|
||
| for i := uint32(0); i < endLedger; i++ { |
There was a problem hiding this comment.
why does i start at 0 instead of startLedger?
| func calculateParallelLedgerBatchSize(rangeSize uint32, workerCount uint) uint32 { | ||
| // let's try to make use of all the workers | ||
| batchSize := rangeSize / uint32(workerCount) | ||
|
|
||
| // Use a minimum batch size to make it worth it in terms of overhead |
There was a problem hiding this comment.
should there be a maximum batch size as well? I remember you mentioned that it could be helpful to have a maximum batch size because the ledger ingestion time varies based on whether the ledger is from recent history vs very old history. So, in the scenario where you want to reingest full history with, for example, 4 workers, the workers which handle the first half of history will finish first and be idle for a long time.
There was a problem hiding this comment.
it seems like adding max batch size here would be trying to address a larger concern of maximizing worker pool throughput(minimizing idle workers) which seems like more scope than this function is meant for, it's a good point on performance, wondering if it warrants a separate feature ticket to investigate how constant worker throughput could be accomplished?
one wild thought, workers could be interruptable, so that an idle worker could interrupt one that is running, and take some of it's upper range, the worker would stop it's captive core when it receives lcm for it's adjusted 'to' range in any case.
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
CHANGELOG.mdwithin the component folder structure. For example, if I changed horizon, then I updated (services/horizon/CHANGELOG.md. I add a new line item describing the change and reference to this PR. If I don't update a CHANGELOG, I acknowledge this PR's change may not be mentioned in future release notes.semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
--parallel-job-sizeconfig parameter.Why
Fixes #5468
Known limitations
This could potentially disrupt any automation scripts using
--parallel-job-sizeparameter, although it's unlikely since reingestion is generally run as a batch job on as needed basis.