Two months have passed since our last update on the “state of the clusters”. In our previous post I mentioned we were tracking 25 million pins on a 9-peer cluster.

Today that cluster (which stores content for NFT.storage) has grown to 18 peers and 50 million pins. Our average usage rate keeps at around 4 new pins per second.

The new peers were added and were able to sync the cluster pinset in about 24 hours. This is a cluster with a crdt-DAG-depth 500k, which, given the multiple branches, likely involved syncing millions of CRDT-dag blocks. Because the new peers are empty and have more space that the older ones, they started storing and taking the load, relieving others as intended (older ones have up to 70TB of data pinned).

In the last version (v0.14.5), which we rolled out everywhere, we included some changes to improve performance and CRDT-DAG syncing. We have also started rebuilding older nodes with LVM-striped, XFS and flatfs/next-to-last-3 datastore layout configuration for IPFS. In our experience, XFS performs better than Ext4 for folder with large number of files, which is essentially what flatfs does. Next-to-last-3 is a sharding strategy that shards blocks over folders with 3 letters (the default is 2). By having more shards, there are less items on every folder, which is better for very large nodes.

The main issue now preventing unbounded scalability is that the huge pinset causes RAM memory spikes whenever a cluster peer needs to check that the pins that are supposed to on ipfs are actually there. This is because every item on the pinset is loaded on memory to be able to iterate on them. At this point, the memory spikes are very noticeable and steal memory which IPFS would gladly use.

The next release of IPFS Cluster will address this and other issues through a major shift on how things work internally, which will not only fix the memory spikes, but also unlock lots of performance gains when adding content to cluster peers. With these changes, IPFS Cluster will graduate to version 1.0.0, having proven its reliability and scalability properties while serving production infrastructure.

Today, we would like to provide a few details and figures on where we are with regards to cluster scalability, particularly as ensuring IPFS storage allocation and replication behind the NFT.storage platform.

We have started 2022 with a new release (v0.14.4). A few months ago, we were happy to report that we were tracking around 2 million pins.

Today, cluster is tracking over 25 million pins for NFT.storage in a single cluster, made of 9 peers with around 85TB of storage each running go-ipfs v0.12.0-rc1. On average, we are ingesting more than 4 items per second (normally add-requests that put the content directly on the cluster). We know we can take many hundreds of pins per second when needed.

These numbers are not overly impressive when compared with, for example a PostgreSQL instance for pinset tracking, but we understand cluster as a distributed application with seamless pinset syncing which also supports things like follower-clusters and scalability to hundreds of peers based on its pubsub+crdt pinset distribution mechanisms.

In terms of configuration, we have set the cluster peer to let IPFS pin up to 8 items in parallel. This is what we found was a well-performing value when going through pinning queues of several million items. Bitswap performance, disk usage, network bandwidth all affect the right values. The cluster-peers are configured using the crdt consensus mode, with replication factors set to 3. Each node is tagged with a datacenter tag, and the allocator is set to allocate per datacenter and free-space. Thus, we get global distribution of every pin, which are then allocated to the peers with most free space in each DC. We make use of the crdt-batching function, creating commits every 300 items or 10 seconds (although we tune them as we need, sometimes increasing the batch size or delay). For reference, one batch (crdt-delta) can fit almost 4000 pins with 3 allocations (actual number depends on the pin options and allocations).

The 20x pinset growth in the last few months has necessarily been accompanied by several releases to get IPFS Cluster up to the task of handling multi-million setups:

• The cluster-peer datastore can be setup with LevelDB and Badger, and the latter is GC’ed regularly so that it does not grow to take too much space per pin.
• We heavily sped up operations reading the full pinset (pin ls or status). For example, it is now very efficient to check all the pins in error or queued states because filtering has been improved. Listing all pins in the state has improved an order of magnitude.
• State export and import functions have also been improved to allow for cluster pinsets to be moved around (to new clusters), which facilitates maintenance, for example by setting new allocations for pins.

The next steps are to keep iterating towards supporting much larger pinsets. One of the improvements in the pipeline will be streaming-RPC support (cluster components communicate via RPC). This will allow us to speed up many operations, such as listing or adding to the cluster.

Filecoin recently announced the launch of NFT.storage, a pinning service to provide perpetual IPFS storage specifically catered to NFT creators and collectors.

The service is backed by storage provided by Pinata and Protocol Labs, with the service on the Protocol Labs side relying on IPFS Cluster for pin tracking and replication.

The service has been setup as a collaborative cluster with 3 main storage peers run by Protocol Labs. The Cluster currently tracks and pins 1.900.000+ items, including many existing NFTs from around the web, which are preserved for posterity.

To better support the requirements of the project, a couple of upgrades have been added to IPFS Cluster:

• First, we have enabled batch-pin ingest in CRDT-mode. This allowed us to easily ingest over 400.000 pins to the cluster in less than 1 hour, with a very low cost to the system. From that point, the cluster peers make sure that IPFS pins the items in an orderly fashion, restarting stuck pins as needed.

• Second, we have added the possibility of adding arbitrary DAGs to the cluster directly, by enabling CAR-file imports on the /add endpoint. This powers the storage of CBOR-encoded DAGs, that include metadata and links to the actual NFT-material.

These features have been included in IPFS Cluster 0.13.3, which we just released.

Happy pinning (and now, at very high rates)!

We just released IPFS Cluster 0.13.1, with some bugfixes, dependency upgrades and a couple of improvements.

While development efforts have been moved to other parts of the ecosystem in the last few months, the IPFS Project continues being maintained, although without active development of large features. What users can expect is:

• Support over the common channels, responses to issues etc.
• Bugfixes and pull request reviews.
• Dependency upgrades and project maintenance, with a slow but stable release candence.

IPFS Cluster continues to be used in production to replicate and distribute important data hosted on IPFS. Happy pinning!

IPFS Cluster 0.12.0 is here! It comes with the new ipfs-cluster-follow application, a super-easy way of launching a “follower” peer.

Follower cluster peers join clusters to participate in the replication and distribution of IPFS content, but do not have permissions to modify the Cluster peerset or perform actions on other peers of the Cluster. When running ipfs-cluster-follow, peers are automatically configured with a template configuration fetched through IPFS (or any HTTP url) and run with some follower-optimized parameters. Additionally, ipfs-cluster-follow can setup and run multiple peers in parallel, so users can subscribe to several clusters at the same time.

Would you like to try it out? Grab ipfs-cluster-follow and run:

./ipfs-cluster-follow ipfs-websites init ipfs-websites.collab.ipfscluster.io
./ipfs-cluster-follow ipfs-websites run

Your IPFS daemon will start pinning a list of IPFS-related websites (you will need about 600MB of available space). You can stop and re-start your followers any time and they will catch up to the latest state of things.

We have also added a bunch of new features. Pins can now have expiration times so that they are automatically unpinned at some point. And Cluster operators can now use the Cluster-GC command to trigger garbage collections on all the managed IPFS daemons. These should be very useful for IPFS storage providers.

Finally, users running clusters behind NATs or in Dockerized environments will benefit from improvements in NAT traversal and connectivity. Cluster peers now support the new libp2p QUIC transport and TLS handshake.

You can read more information about all the new things for this release in the changelog.

We hope the possibilities opened by this new release will make IPFS Cluster a very useful tool for hosting and re-distributing IPFS data in a collaborative manner, building communities around archives based on user interests and strenghthening content distribution in the IPFS network by doing so.

A few days ago we shipped IPFS Cluster 0.11.0. This was a huge leap forward as it finally crystallizes the journey to replace Raft with a system that allows peers to come and go freely from a cluster while keeping consistency guarantees on the shared pinset. The effort to find a suitable replacement started almost a year ago and resulted in a new crdt component that is based on go-ds-crdt, a datastore implementation using Merkle-CRDTs.

As mentioned in the changelog, version 0.11.0 is the biggest release in the project’s history and it comes with many other features and improvements.

We have also started running IPFS Cluster workshops on several conferences. We keep an updated list of past and upcoming events at our workshops repository. In these workshops, participants install and run cluster peers with a IPFS-hosted configuration and they automatically discover each other, form a Cluster and try out all commands.

During the upcoming months we will be shipping more features but also start taking advantage of IPFS Cluster’s new features by launching public collaborative clusters: we will publish instructions for anyone to join specific clusters to backup pieces of important IPFS data such as the distributions page, wikipedia mirrors or community websites.

We wish you a lot of success using the latest version of IPFS Cluster.

Today we release 0.10.0, a release with major changes under the hood that will make IPFS Cluster perform significantly faster with large pinsets and less memory demanding.

For those upgrading, this release is a mandatory step before any future upgrades, as it will upgrade the internal state to a new format which prepares the floor for the upcoming addition of an alternative CRDT-based “consensus” component. The new component will increase IPFS Cluster scalability orders of magnitude and unlock collaborative Cluster where random invididuals can collaborate in replicating content.

We also have a few new features:

• Path resolving before pinning and unpinning
• Ability to manually specify pin allocations
• Environment variable override to all configuration options
• Added the possibility to store custom metadata with all pins

Finally, the Cluster team would like to thank @alekswn and @roignpar for their awesome contributions!

Be sure to check the changelog for a detailed overview of changes and upgrade notices.

Happy pinning!

IPFS Cluster version 0.9.0 comes with one big new feature, OpenCensus support! This allows for the collection of distributed traces and metrics from the IPFS Cluster application as well as supporting libraries. Currently, we support the use of Jaeger as the tracing backend and Prometheus as the metrics backend. Support for other OpenCensus backends will be added as requested by the community. Please file an issue if you would like to see a particular backend supported. We are looking forward to digging deeper into how IPFS Cluster peers operate and communicate with each other and accurately measuring how they are performing in real world deployments.

The one other significant change that comes with the 0.9.0 release is the removal of the Snap distribution of IPFS Cluster. Due to difficulties in getting Snap builds to work reliably without a disproportionate amount of time spent debugging them, we decided to deprecate the distribution mechanism.

Happy Measured Pinning!

Since the beginning of IPFS Cluster, one of our ideas was that it should be easily dropped in place of the IPFS daemon in any integration. This was achieved by adding an IPFS Proxy endpoint which essentially provides an IPFS-compatible API for Cluster. Those endpoints and operations which does not make sense to be handled by Cluster are simply forwarded to the underlying daemon. Add, pin and unpin operations become, however, Cluster action.

IPFS Cluster 0.8.0 comes out today and includes a revamp of the IPFS Proxy endpoint. We have promoted it to be its own API-type component, extracting it from the IPFS Connector (which is just the client to IPFS). We have additionally made improvements so that it truly mimics IPFS, by dynamically extracting headers from the real daemon that can be reused in the responses handled by Cluster. Thus, there will be no CORS-related breakage when swapping out IPFS for Cluster, and custom IPFS headers (i.e. X-Ipfs-Gateway) can be configured and forwarded by the proxy.

The increasing importance of browser integrations prompted us to fully support Cross-Origin Resource Sharing (CORS) in the REST API as well. It will now handle CORS pre-flight requests (OPTIONS) and the configuration allows the user to set up all the CORS-related headers as needed.

This is the first release with @kishansagathiya as full-time member of the team. Apart from extracting the IPFS proxy component, Kishan is behind the new --filter flag for ipfs-cluster-ctl status. You can now list all the items that are pinning or in error without the need of a complex grep line. A few more useful features will be coming up in the future.

For the full list of changes and update notices for this release, check out the changelog.

Happy pinning!

We are proud to introduce the 0.7.0 release today. It comes with a few small improvements and bugfixes.

We have slightly changed the /add endpoint response format in a non-compatible way, to return more adequate objects than the ones mimic-ing the IPFS API. It’s not the best but, better now than later.

We have also fixed the proxy /add endpoint to work correctly with the IPFS Companion extension and js-ipfs-api. Thanks to @lidel for helping figuring out the problem!

Regarding features, @kishansagathiya has been making a few contributions lately and now, among other features, we have new commands like ipfs-cluster-ctl health metrics freespace which show the list of last received freespace metrics and their validity.

Finally, we have included a default docker-compose.yml template, which launches a stack with 2 ipfs daemons and 2 cluster peers.

As usual, for the full list of changes and update notices, check out the changelog.