Open Grant Proposal: Interplanetary Library Catalog
Name of Project:
Interplanetary Library Catalog
Proposal Category: app-dev
Proposer: wesc
(Optional) Technical Sponsor: n/a
Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT and APACHE2 licenses?: Yes
Project Description
Curating, acquiring, and archiving data is a non-trivial combination of skills. Typically, all three functions are handled by a single organization. For example, Wikipedia maintains and archives its own data, but requires $10s of millions of USD per year to do so. More commonly, organizations or people are competent in a subset of those skills.
- In academia, it’s common to find broken references to data from field experiments. Either the FTP servers are ill maintained, academics shuffle data around thereby breaking links, or they accidentally delete files.
- A small church may generate monthly newsletters it wants to archive, but the organization lacks the IT capability to do so in perpetuity.
- A tinkerer has technical capability to run an IPFS node and is interested in supporting the arts but is not an artist or collector. An artist the tinkerer follows, though, loves to collect public domain photographs but does not have the ability to run an IPFS node.
- An activist has the ability to copy a one time snapshot of Turkish Wikipedia into IPFS, however does not have the bandwidth to keep the snapshot up to date.
IPFS provides a solid foundation on which we can store and mirror data, however what is missing is an ecosystem of easy to use tools that enable community members to find and mirror data that is meaningful to them. Like the SETI@home and Folding@home projects, it should be just as easy as installing a screensaver to maintain a local mirror of data or alternatively donate filecoin towards the archival of a dataset.
The long term tech tree for the project looks like:
- Schema for a dataset (maintainer, title, description, version, etc), which is an immutable view of data.
- Schema for a collection (curator, title, description, mutable list of datasets in the collection), a mutable list of datasets. Methods for permalinking to a collection (IPNS, ENS).
- Command line tooling for pinning and updating collections. Easy to use deployment to Docker, AWS, Raspberry Pi, and local Linux installations.
- Protocol for publishing new datasets and collections.
- Software that constructs a public search index from published collections. Addition of index into deployment targets.
- Cryptographic methods for protected or semi-public data. Methods for zero knowledge construction of derived data from encrypted private datasets.
- DApp for allocating filecoin to collections.
- Decentralized incentive and reputation system for dataset maintainers and curators. An incentive system should encourage the continual maintenance of datasets and collections, and a reputation system should discourage the archival of illegitimate material.
- Interface suitable for non-technical users to upload datasets and curate collections.
This proposal is scoped to the lowest technical branches 1 - 3. Completing those three branches will produce a system with low overhead that establishes reciprocal pinning relationships between different institutions. With minimal IT overhead, the MIT Center for Constructive Communication, for instance, may agree to mirror data from the Stanford Internet Observatory, and vice versa.
(Note the authors of this proposal are staff at MIT CCC, but this is not a commitment by CCC nor Stanford IO to mirror data.)
Value
A system functioning as designed would do three things:
- Reduce the friction to allocate storage towards archiving useful data.
- Democratize the curation of data. Organizations such as the Internet Archive and Media Cloud have an opinion on what data is important, and there does not exist an organic market or community mechanism to signal value.
- Democratize the allocation of capital towards archival.
There is a risk that the system can be used for wide scale piracy or CSAM, which IPFS already must deal with. To mitigate these issues, this system relies on the reputation of curators and peer pinning relationships. This proposal will initially involve only above board institutions and known entities, however future work will develop market mechanisms for reducing the likelihood of widespread illegitimate distribution.
Deliverables
The deliverables are branches 1 - 3 from the tech tree, as well as the active reciprocal mirroring of data between two or more organizations.
Development Roadmap
Milestone 1:
At this milestone, we expect a moderate level of technical proficiency from users in setting up pinning nodes, ie they should be able to spin up Linux servers, ssh in, and edit text files.
Goals:
- Dataset and collection schemas fully defined. Schema should at a minimum be human and machine readable, and include contact information.
- Defined unique permanent identifier scheme for collections.
- Reciprocal pinning proof of concept that takes as input collection identifiers and automatically updates the pinset.
- Manual procedures for adding, updating, and deleting datasets.
- Manual procedures for adding, updating, and deleting collections. Verification that unwanted data is unpinned from nodes.
Two developers will be working on this milestone at an expected cost of $1,000 USD. Work to be completed by February 1st, 2022.
Milestone 2:
Goals:
- Bring pinning proof of concept to production level quality. It should run with minimal downtime and oversight. The user should be able to “set and forget” a pin on a collection.
- Construction of a Docker image that brings up a pinning node. Provide scripts to bake an AMI or provision a Raspberry Pi or any other Linux system.
- Active pinning of the most recent two English Wikipedia snapshots. This collection should automatically update every month.
- Active pinning of an additional dataset, to be determined.
Two developers will be working on this milestone at an expected cost of $1,000 USD. Work to be completed by March 1st, 2022.
Milestone 3:
- Complete documentation.
- Establish a reciprocal pinning relationship between two groups such as MIT CCC, MIT Media Lab, Stanford IO, the Harvard Library Innovation Lab, Media Cloud, or interested individuals, involving at least two collections.
One developer at zero cost. Work to be completed by May 1st, 2022.
Total Budget Requested
Total requested budget is $2,000 and accounts for labor. The cost of running infrastructure for the reciprocal pinning relationships will be borne by the groups or interested individuals.
Maintenance and Upgrade Plans
Reciprocal relationships are maintained by the groups involved, and the design of this software should be such that this requires minimal IT effort.
Additional work for technical branches 4 and on will be the subject of future Filecoin grant proposals should this project demonstrate value.
Team
Team Members
Wes Chow
Sinclair Target
Team Member LinkedIn Profiles
Wes Chow: https://www.linkedin.com/in/weschow/
Sinclair Target: https://www.linkedin.com/in/sinclairtarget/
Team Website
n/a
Relevant Experience
Wes and Sinclair are developers working on computational social science in the MIT Center for Constructive Communication. This work entails a good deal of data engineering, for example the ingestion of Twitter firehose data, processing, and building data systems. This proposal is motivated by a common problem in computational social science around the reliability and access of public datasets. Many peer organizations independently acquire and replicate data, but there is no mechanism or coordination by which researchers can pool those efforts.
Team code repositories
Wes Chow: https://github.com/wesc
Sinclair Target: https://github.com/sinclairtarget
Unfortunately, much of our data engineering work is proprietary. There is some open source code at the Lab for Social Machine’s github repo:
https://github.com/social-machines/
Note, MIT CCC was formerly known as the Lab for Social Machines.
Additional Information
MIT CCC is not applying for this grant. That said, the developers intend to use their work networks to seed support for the project.
Open Grant Proposal: Interplanetary Library Catalog
Name of Project:
Interplanetary Library Catalog
Proposal Category:
app-devProposer:
wesc(Optional) Technical Sponsor: n/a
Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT and APACHE2 licenses?: Yes
Project Description
Curating, acquiring, and archiving data is a non-trivial combination of skills. Typically, all three functions are handled by a single organization. For example, Wikipedia maintains and archives its own data, but requires $10s of millions of USD per year to do so. More commonly, organizations or people are competent in a subset of those skills.
IPFS provides a solid foundation on which we can store and mirror data, however what is missing is an ecosystem of easy to use tools that enable community members to find and mirror data that is meaningful to them. Like the SETI@home and Folding@home projects, it should be just as easy as installing a screensaver to maintain a local mirror of data or alternatively donate filecoin towards the archival of a dataset.
The long term tech tree for the project looks like:
This proposal is scoped to the lowest technical branches 1 - 3. Completing those three branches will produce a system with low overhead that establishes reciprocal pinning relationships between different institutions. With minimal IT overhead, the MIT Center for Constructive Communication, for instance, may agree to mirror data from the Stanford Internet Observatory, and vice versa.
(Note the authors of this proposal are staff at MIT CCC, but this is not a commitment by CCC nor Stanford IO to mirror data.)
Value
A system functioning as designed would do three things:
There is a risk that the system can be used for wide scale piracy or CSAM, which IPFS already must deal with. To mitigate these issues, this system relies on the reputation of curators and peer pinning relationships. This proposal will initially involve only above board institutions and known entities, however future work will develop market mechanisms for reducing the likelihood of widespread illegitimate distribution.
Deliverables
The deliverables are branches 1 - 3 from the tech tree, as well as the active reciprocal mirroring of data between two or more organizations.
Development Roadmap
Milestone 1:
At this milestone, we expect a moderate level of technical proficiency from users in setting up pinning nodes, ie they should be able to spin up Linux servers, ssh in, and edit text files.
Goals:
Two developers will be working on this milestone at an expected cost of $1,000 USD. Work to be completed by February 1st, 2022.
Milestone 2:
Goals:
Two developers will be working on this milestone at an expected cost of $1,000 USD. Work to be completed by March 1st, 2022.
Milestone 3:
One developer at zero cost. Work to be completed by May 1st, 2022.
Total Budget Requested
Total requested budget is $2,000 and accounts for labor. The cost of running infrastructure for the reciprocal pinning relationships will be borne by the groups or interested individuals.
Maintenance and Upgrade Plans
Reciprocal relationships are maintained by the groups involved, and the design of this software should be such that this requires minimal IT effort.
Additional work for technical branches 4 and on will be the subject of future Filecoin grant proposals should this project demonstrate value.
Team
Team Members
Wes Chow
Sinclair Target
Team Member LinkedIn Profiles
Wes Chow: https://www.linkedin.com/in/weschow/
Sinclair Target: https://www.linkedin.com/in/sinclairtarget/
Team Website
n/a
Relevant Experience
Wes and Sinclair are developers working on computational social science in the MIT Center for Constructive Communication. This work entails a good deal of data engineering, for example the ingestion of Twitter firehose data, processing, and building data systems. This proposal is motivated by a common problem in computational social science around the reliability and access of public datasets. Many peer organizations independently acquire and replicate data, but there is no mechanism or coordination by which researchers can pool those efforts.
Team code repositories
Wes Chow: https://github.com/wesc
Sinclair Target: https://github.com/sinclairtarget
Unfortunately, much of our data engineering work is proprietary. There is some open source code at the Lab for Social Machine’s github repo:
https://github.com/social-machines/
Note, MIT CCC was formerly known as the Lab for Social Machines.
Additional Information
MIT CCC is not applying for this grant. That said, the developers intend to use their work networks to seed support for the project.