Distributing the registry storage via p2p file sharing

[zkochan]

From v0.58.0 pnpm caches of all the tarballs in a "registry mirror" hosted in the local filesystem. The local registry uses the same structure that verdaccio (former sinopia) uses for its storage.

As a consequence, the tarballs storage can be a torrent seeded by the verdaccio server.

Clients would have all the tarballs hosted locally even before installation started.

I'll probably create an issue in the verdaccio backlog, but there might be even other ways to distribute the registry.

[iamstarkov]

dat project?

[zkochan]

TBH, I have never heard about this project. But it seems like it is only for archives, not a npm-compatible registry

[zkochan]

Created an issue at verdaccio, maybe someone will be interested in this concept.

verdaccio/verdaccio#128

[iamstarkov]

@zkochan what is your vision on how it will work?

[zkochan]

My vision is some sort of a smart cache that will contain the tarballs you need before you even executed pnpm install.

[zkochan]

About the way it will work. I see these solutions:

• the smart cache can contain the whole registry. But that might consume a lot of disk space, I don't know.
• it can have a fresh cache only for the packages that were ever downloaded on the machine. So if you do a pnpm update, it will be really fast
• it can have all the packages from npm that have more than some number of downloads per day.

[zkochan]

If you look into this package https://github.com/nice-registry/download-counts you'll see that only a small percentage of packages in the npm registry have more than 100 downloads a day. So I think the 3rd solution might be the best one.

[zkochan]

Also similar idea: https://github.com/mixu/npm_lazy

[zkochan]

We might be able to do this if we make pnpm run with the store server by default. The issue with the store server is currently that it is pretty slow. Installations are up to 20% slower with the store server than w/o it.

[staltz]

I really would like to see this happen with Dat. Is the store server installation speed the only hurdle? Could we build it anyway? I would prefer slow decentralized install over fast centralized installs.

I discussed with mafintosh some months ago that the approx. best way of doing this with Dat is one Dat Archive per author, so essentially installing package foo would be the address dat://7e7432..a4109b/foo and we could also create local aliases for authors, so you could do pnpm dat-alias staltz 7e7432..a4109b. Storage shouldn't be a problem, in fact authors would be incentivized to keep their Dat archive small, because the smaller the archive, the more likely people would easily seed it.

If you want guidance/help how to do this, you can count on me to get started.

[zkochan]

@staltz I can start working on it not soon. Currently, I am invested into making pnpm the best package manager for multi-package repositories. This can be the next big thing though.

We can start with writing specs.

We have a separate repo for specs here.

[yangm97]

@ipfs come here real quick

[zkochan]

If I understand correctly, we can do this by storing the metadata and the tarballs in IPFS, and then use IPNS to reference the latest metadata files by package name.

[frani]

Hello there! I’m interested in collaborating to create a Distributed Package Manager System.

To be honest, I believe we should use blockchain as registry implementing a smart contract on any EVM to manage registry names, authorities, and more.

EVMs are currently in a developing phase, focusing on scaling their transactions per second (TPS), which makes this an opportune time for such a project.

Hey @P1R! They want to create a distributed package manager system. Are you interested?

[P1R]

Hey! @frani @zkochan a bit busy at least for this year. IPNS its useful but a bit complex to manage with the base configs in perspective, following many packages and metadata to update, perhaps and build on IPFS is the OrbitDB, if it requires security as @frani referenced you can use Smart Contracts to handle in a easier way the version (perhaps something like what I was working to manage my dapps listing as an alternative of verify my dapps integrity and future integration to a DAO https://git.decentralizedscience.org/DecentralizedScience/dapp_indexer), anyway the other hard thing can be how to incentive nodes to store and decentralized further, which @frani and I developed in the past some work on a proposal economic model to incentive hosting on a decentralized way. Also in terms of version control you can consider radicle.xyz, they released their first stable version, and the project is very cool, it used to work with ipfs in the past but I think they simplified and took git as the base structure to handle the version control and a way to avoid conflicts + using the local first philosophy.