Lens update - milestone 2

[djfnd]

Creating this thread as a staging point for the next neume update blog, and to coordinate updates for the next Lens update.

[djfnd]

Second tranche milestones

1. Get the Lens crawl up and running
2. Onboard front ends in the Lens community to neume
3. Refine and further develop the specification of neuIP-5

Update

1. Get the Lens crawl up and running

@il3ven ($3k)

Research

How to sync nodes and distribute crawled data
The major research was done on how to sync nodes and distribute crawled data. In the discussion over at GitHub, we discussed around 4 possible ways. Each method has its own drawbacks and at the end we weren’t able to find the one perfect method but chose the one with the least cons and practical dev efforts.

In summary, we faced the following trade-off. Either implement a perfect decentralized set of nodes that can crawl the web3 music space through peer-to-peer technique or be practical and allow clients to easily use neume in their tech stack.

Dealing with the scale of Lens
Another research effort was done to estimate the scale of Lens and ways to deal with it. Before Lens, neume has only dealt with platforms a 1/100th in scale.

We had to make sure to avoid the following possible pits:

• crawler takes a long amount of time to get up to date from scratch
• our database crashes, slows down or takes up huge space
• price incured due to network requests to arweave and ethereum gateways
• We did think around crawling a subset of the Lens ecosystem. For example, selected accounts but after estimating the scale we decided to crawl all music on Lens.

Development

The development part consists of actually writing the code, thoroughly testing it out and hosting it on the cloud. And after fixing any edge cases we find along the way.

Here are a few snapshots of the journey.

• We converted all the existing strategies to the new interface that we decided upon in the first tranche. This new interface makes the strategies more generic.
• Due to the changes in architecture such as syncing of nodes, new chains and new database. We had to re-write portion of the code.
• After setting up the crawler and running it on the cloud we faced new challenges - choking of database on high load, crashing of the crawler due to unhandled cases or malformed metadata.
• Since we were dealing with high load for the first time, we found a bug in our old code. The network retries were not working predictably. - Network retries along with timeouts are very important for crawler’s stability

We have also built a simple front end to visualise the data coming through neume, and have allocated gitcoin grant money towards further development of a front end.

2. Onboard front ends in the Lens community to neume

@estmcmxci ($1k)

We have engaged with the Lens community management, and started a project into building out the "neume lore" to support with onboarding Lens front ends

3. Refine and further develop the specification of neuIP-5

@neatonk ($1k)

We have progressed the "neume as a network" branch of development towards something that is ready to become the focus of the next tranche of work, specifically, creating distribution of neume via arweave, which will lead to towards a neume SDK. See this page for full details.

Third tranche milestones

1. neume distribution via arweave leading to neume SDK - scope @neatonk ($4k)
2. onboard front ends - scope @estmcmxci ($1k)

[djfnd]

Thanks for the input everyone, I have now submitted the update to Lens
@il3ven @neatonk @estmcmxci