Decentralized Data Ideas

[davidgasquez]

Random thoughts around decentralized and permissionless data lakes.

• Everything should be content addressed and, ideally, immutable. I should be able to query a dataset via its CID without caring where it is stored.
• Publish the CID of the something like a Delta Catalog JSON file on Ethereum. You can publish your fork or write contracts on top of it. Use any compute engine to run queries on top of that.
• Collaborate on data TrueBlocks style, where more people using the service means better data reliability and speed. If there is a section missing, I can send something like a PR to fill that data.

Also from datonic/datadex#22 (comment).

Reading "The Database I Wish I Had" and thinking about something like that for OLAP workloads. Feels like OLAP use cases might be the "killer database" for IPFS/Hypercore/Dat. For analysis, you want data to be inmutable, don't care that much about latency, and have to store large amount of data.

[davidgasquez]

Chatted with some folks working on Subsquid. They're doing interesting things on the decentralized data lake area.

This is more or less what I understood about how the Subsquid Archive works.

1. Right now, data is indexed by Subsquid itself (running substrate-ingest). In the future, anyone will be able to publish their arbitrary datasets.
2. Indexed data is packaged and into height partitioned Parquet files and sent to an orchestrator/router that distributes these across nodes in the subsquid network. This orchestrator takes into account dataset durability, response times, geolocation distribution, ...
3. Users send (and pay) queries to the Subsquid network (via a gateway or contract?), and the gateway will select the nodes to run these queries. Nodes will run the query (DuckDB on the nodes), and send back the results.

Subsquid Labs maintains public Archive endpoints and offers batch access via the Squid SDK free of charge.

Questions

• How can you join across different heights/datasets that are in different machines? This will need a proper decentralized query engine (perhaps something where Substrait, Datafusion and Ballista can help!).
• How do you guarantee fast response times?
• Is there any mechanism in which commonly accessed data is more distributed?

[davidgasquez]

Adding a small note that Dagster is already relying on "hashes" to check when runs are needed! A step closer to fully content addresses workflows.

[davidgasquez]

You can ATTACH to a remote DuckDB database! There might be a world where a bunch of people publish their small/medium databases and people just attach to them.

[davidgasquez]

Having a decentralized data lake paired with some good standards and incentives, could result in a universal repository of high-quality structured data ™️. A public data warehouse. Similar to Wikipedia, but for public data.

This might be possible now as the technology and theory is there. We have L2s, token based incentive systems, modular and composable data stacks, fast peer to peer connections, and data pipelines are virtually free.

[davidgasquez]

Working with Parquet on top of IPFS or other decentralized file system is tricky as Parquet doesn't have deterministic encodings and changing only one element might cause the entire file to be "reuploaded".

Adding here a few notes that I've collected in the past:

• Kamu folks have created a deterministic Arrow crate to have stable hashings.
• Might be interesting to rely on table formats like Delta to take care of some of the details on evolving datasets.
• HuggingFace is focusing on improving Parquet deduplication using Content Defined Chunking and other ideas also present in the Content Addressing world.

[davidgasquez]

Open Data via AT Protocol

The AT Protocol is an open, decentralized network for building social applications.

What if we tapped into the protocol primitives to create an open directory of (potentially interlinked) datasets? Some ideas:

• Use DIDs and AT URIs to identify data repositories and individual data sets uniquely.
• PDSs could store the data or point to the URLs that serve it.
• Lexicon as a way to define apis (e.g: publishData) and data standards (dates, locations, data types, partitions) that allows using (exploring, getting, joining) datasets.
• The Firehose will contain updates, new datasets and similar things.
• There would be signed official updates (weather.com updating a dataset) and forks (someone.com derives a new field and serve it)...

Quite hand wavy for now but looking at the success Bluesky is having inspired me to think more about this one.