Ideas of data pipelines leveraging the concept of function-addressable computation network

[albertdessaint]

Hi everyone, 

A month ago, the paper IPFS-FAN: A Function-Addressable Computation Network introduced concepts surprisingly close to thoughts our team has been actively developing since the end of March. Basically, we are working on a protocol that allows people to collaborate on data pipelines using publicly available data (open data, data on blockchains, data on IPFS) but also using publicly available data transformations (called functions in the paper). The ultimate goal is to allow anyone to reuse and build on existing data pipelines (kind of like DeFi composability, but for data). The protocol will be delivered and open sourced in September. 

We found appropriate to reach out to @adlrocha, one of the researchers at Protocol Labs who worked on the paper and ask him about the kind of data pipelines that he thinks should be developed first to demonstrate the potential of such a protocol (we plan on developing these first data pipelines although the final goal is of course for everyone to create and update data pipelines themselves). 

Alfonso kindly responded to us with some fields of application:

• Data access control, authentication, authorization, accountability.
• Log processing and any other type of stream data processing.
• AI/ML decentralized training.
• Network Function Virtualizations (NFV).
• Smart contracts computation decoupled from the data
• Verifiable computation/proofs.
• Decentralized serverless infrastructure.

Alfonso also suggested that we start an open thread, so here it is!

In this context, may we all share here any idea of data pipelines you would love to use and that would be useful to the community? Or do you know someone who would be interested in sharing their needs? Ideas can be general or more specific to certain use cases. 

Don’t hesitate to ask if you need more information. We can't wait to see what we may come up with :-)

[yiannisbot]

Hi @albertdessaint! It's great to hear that you've got interest in this space! I would be very interested to see how IPFS-FAN and related concepts can be applied to the following use-cases:

1. personal data markets: in those use-cases, personal data is stored in some arbitrary storage location in the network and the identifier of the content is shared with selected parties (or with anyone, depending on the data and the use-case). It would be great to be able to:

• count the number of times the data has been accessed and by which authority,
• charge the corresponding parties accordingly,
• be able to revoke access to the data.

2. secure and private remote computation on data: in those cases, again data is stored at some arbitrary storage location, but selected parties, who are given access (or can authenticate themselves) can carry out computation on the data with pre-selected functions in a decentralised fashion.

Both use-cases are largely covered by @adlrocha 's points, but I thought of adding some extra clarification on use-cases I had in mind.

Looking forward to seeing your work being released in September!

[albertdessaint]

Hi @yiannisbot, really appreciate your comment! Both ideas are interesting even though our current work is not focus on proof and revocation of access. There is a lot of cryptography involved in order to achieve this and given that our resources are limited it is not a priority on our side 🙂

We think that the first use cases could be based on fully available data on IPFS with no access restrictions. Do you have any use case in mind in this respect?

It can nevertheless be interesting to see if Ceramic could meet your needs on these two ideas!

[albertdessaint]

Hi everyone,

We have just released our project built on top of IPFS and WebAssembly! You can now transform data in a way that enables sharing of its history.

Link to the website of the project: holium.org

We would love to hear your feedback and ideas for future developments :-)