Index updates can be significantly sped up (on some coins?)

[SharkWipf]

I just set up Iquidus git for a small coin, and noticed the indexing process was going slower and slower.
I also noticed it was fetching the same raw transactions many times per block on the node process (see image).
It did this every ~5 or so blocks and this significantly slowed down the syncing, from several blocks per second to a block every 5-or-so seconds.
I checked, and the data it fetched was the same every time.

I messed around with it a bit and added an Nginx reverse proxy caching server inbetween, and this sped up the syncing process enormously for these entries, back up to around ~10-20 blocks per second.

If the code were to be rewritten to re-use the raw txid if it's used multiple times, this would significantly speed up the index process and would possibly even resolve/lessen issues like #104, #240 and #247.

(Edit: forgot to attach the image)

[uaktags]

can you provide what coin you experienced this with and any specific config you used with nginx?

For the most part, atleast in terms of the aforementioned issues, reusing the txid wouldn't help at all as their complaints are just that there's alot of blocks, and with high number of txes per block, it can take a long time. Only real way to speed this up would be use more workers (there's only 1 worker used in sync.js) or have another way of reading the data (perhaps from a bootstrap parsing?)

I'm not sure why it would be grabbing the same txid though multiple times in your case above. The logic of sync is "getblock", read the txes from the block, "getrawtransaction" per tx in the block. It shouldn't ever run into the same txid.

[SharkWipf]

Apologies, I completely forgot about this issue.

can you provide what coin you experienced this with

ORL, Orwell, a coin that has most likely exit scammed and hardly had any presence to begin with.

and any specific config you used with nginx?

All I did was enable caching of requests to /api/ so subsequent identical(!) requests would get fetched from the Nginx cache instead of from the Iquidus service.
I also played around with the Iquidus code lightly in order to make the sync script connect to the Nginx server instead of Iquidus directly.
I only did this after I had ran into the issue with the many slow duplicate requests, so Nginx is not the cause.

I haven't tested Iquidus with other coins yet, I only set it up for ORL because the official explorer was gone and I needed to dive into something, using an interactive explorer is much easier than diving into RPC of course.
I don't know if these calls to identical txids only happen for ORL or if they happen for other coins as well, but I imagine something's probably wrong somewhere if they happen like this at all.

[Bruttagente]

Hi, can you please provide me how exactly you changed the Iquidus code to connect to Nginx? I'm facing the same slowness issue and I would like to try this solution to speed up the update, thank you

[SharkWipf]

Sadly I don't have the modified Iquidus files anymore, so I can't tell you exactly what I did, but basically all I changed was the way Iquidus used its listening port to allow Nginx to run inbetween.
That said, if you're not in an immediate rush, check out #257, if all goes well there should be big performance improvements coming to Iquidus soon.

[TheHolyRoger]

I think this is already much improved in #319, if you can test it would be great

[TheHolyRoger]

Merged in 8d44d4a