The Web3 Data Storage Challenge

Until recently, blockchain data storage has been fundamentally inefficient. Take Ethereum, for instance: every archive node must store all information, resulting in massive data duplication. While this approach ensures security, it’s far from optimal for modern Web3 applications.

The industry has attempted to address this challenge through various solutions:

• Data Availability Sampling
• Proto-Danksharding in Ethereum
• Arweave
• Avail
• Celestia
• Eigen Layer

Solutions like Avail, Celestia, and Eigen Layer offer short-term storage using Reed-Solomon codes and polynomial commitments. However, their commitment schemes differ from those typically used in ZK-proofs, creating bottlenecks in scaling. Moreover, these protocols inherently exclude long-term storage.

For long-term storage, we have Filecoin and Arweave. These solutions are significantly more cost-effective as they economically incentivize long-term storage. They employ ZK technologies for sealing and random challenges to verify data retention.

However, each solution has its limitations. Filecoin, while innovative, doesn’t guarantee data preservation and availability at the consensus level. It’s essentially a decentralized P2P network where storage providers and users enter into smart contract-based storage agreements. Arweave ensures data persistence at the consensus level but lacks flexible data management capabilities.

Why Existing Solutions Fall Short

The key issue with Filecoin in a Web3 context is the potential for malicious consensus leaders to deliberately select unreliable nodes. This vulnerability could enable targeted denial-of-service attacks. For Data Availability, this is critical because we can’t preemptively prevent such attacks — we can only minimize the size of the dishonest majority capable of attacking the network.

While Arweave guarantees data persistence at the consensus level, its immutable, pay-once-store-forever model doesn’t suit many dApp requirements. Modern decentralized applications need mutable storage. Consider a decentralized social network: storing every historical state of mutable objects would be unnecessarily costly. We need storage that can be modified according to smart contract logic while maintaining security.

The Power of Reed-Solomon Coding and Zero Knowledge

Instead of simple replication, ZeroPool employs Reed-Solomon coding — the same technology used in RAID arrays. Here’s a practical example: from four original numbers, we can generate sixteen numbers through Reed-Solomon encoding. Any four of these sixteen numbers can reconstruct the original data.

This approach allows us to achieve the same reliability level as hundreds of replications with just 8x or 16x data expansion. The efficiency gains are substantial:

• Reduced data transfer requirements
• Lower storage costs
• Improved processing efficiency
• Maintained security through Zero Knowledge proofs

Zero Knowledge proofs play a crucial dual role:

1. Proving that each fragment is a valid part of the whole dataset
2. Enabling recursive scaling with minimal bottlenecks

The Scalability Breakthrough

Current rollup solutions don’t store data long-term in the blockchain. Instead, data is excluded from consensus and stored centrally, assuming that two weeks to a month of availability suffices. This approach, dating back to Plasma in 2017, forces users who don’t trust centralized data sources to maintain their own copies — an inefficient and resource-intensive solution that compromises Web3 principles.

The game-changer here is recursive zkSNARKs. While they’ve existed for years, recent technological advances have made them practically viable for large-scale data handling. Using FRI on 31-bit prime fields, we can now achieve 1 Gbit/s throughput for polynomial commitments on standard hardware. These commitments are recursion-friendly, avoiding the performance penalties associated with elliptic curves.

Technical Architecture

The system operates through a clear workflow:

1. Client submits data to validators
2. Validators verify data encoding correctness
3. Data is distributed to nodes
4. Nodes verify Zero Knowledge proofs of their data fragments
5. Nodes and validators sign the transaction
6. Transaction is recorded on the blockchain

Importantly, validators aren’t critical for blockchain security. The codes used are optimistically correctable — assuming an honest minority (a fundamental requirement for any storage system), data can always be recovered even with dishonest validators.

Mathematical Foundation

The technical implementation leverages polynomial mathematics: four original numbers can be represented as a degree-3 polynomial. Computing this polynomial at four points uniquely determines it. We can then compute values at sixteen points, where any four points will uniquely determine the same polynomial, enabling data recovery.

Open Source Collaboration

The project operates under MIT and Apache 2 licenses. Contributions are welcome through pull requests and issues at GitHub

Summing up

ZeroPool Sharded Storage represents a breakthrough in blockchain data storage, offering Web2-level efficiency with full Web3 compatibility. The convergence of advanced cryptographic techniques, improved prime fields, and mature ZK technology has made this solution possible at precisely the right time for Web3’s evolving needs.

This article outlines a technical solution that could fundamentally reshape how we approach data storage in Web3. For developers and projects interested in integration or collaboration, visit the GitHub repository for detailed documentation and contribution guidelines.

Website: zeropool.network

X: https://twitter.com/ZeroPoolNetwork

Discord: https://discord.gg/TNesyTqFdf

ZeroPool Sharded Storage: A Novel Approach to Blockchain Data Storage was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

ZeroPool is a fully private, multi-blockchain solution offering low transaction fees, atomic swaps, and a common anonymity set. It uses innovative technologies, including zkSNARK, and Fawkes-Crypto circuit API library, to hide balances and transaction graphs.

In this article, we will delve into the components that power ZeroPool: the zkSNARK circuit for private transactions and the private transactions contract for the Substrate pallet and client library.

Documentation

The documentation is available at https://zeropool.network/docs/intro.

Fawkes-Crypto

Fawkes-Crypto is API for building r1cs and plonkish circuits. All ZeroPool circuits are implemented using Fawkes-Crypto.

The idea of Fawkes-Crypto is programming like native Rust, but using special circuit-allocated variables. So, when you type `y=x*x`, Fawkes-Crypto automatically initializes a new circuit-allocated value, adds constraints, and optionally set the value.

The implementation is available at https://github.com/zeropoolnetwork/fawkes-crypto.

zkSNARK Circuit for Private Transactions

ZeroPool is using zkSNARKs to prove the correctness of join-split transactions:

1. transaction inputs are corresponding to the Merkle root
2. the sum of inputs equals the sum of outputs
3. spender has the right to spend the inputs
4. nullifier is corresponding to the input account

The hybrid account&UTXO-based model provides to publish only one nullifier per transaction not depending on the number of joined notes. Information about spent notes is stored in the account, so double spending is prohibited.

The implementation is available at https://github.com/zeropoolnetwork/libzeropool/.

The documentation is available at https://zeropool.network/docs/implementation/zksnarks-and-circuits/.

Private Transactions Contract for Substrate Pallet and Client Library

ZeroPool’s implementation on Substrate involves several components, including the pallet, relayer, client, and faucet. Each of these components plays an important role in facilitating private transactions within the ZeroPool​.

ZeroPool Pallet

The ZeroPool pallet is a Zero Knowledge proof-based anonymous transaction engine. It provides functions to perform confidential transactions where users can send and receive transactions privately without revealing any transaction details. The pallet uses the alt_bn128 elliptic curve for generating proofs and a Merkle tree structure to store all notes.

Moreover, the pallet includes a storage map to store nullifiers, which are unique identifiers for spent coins to prevent double-spending. The pallet also includes two verification keys for transfers and the Merkle tree, which the owner of the pallet can manually change. The pallet provides functionality to transfer funds, deposit funds, and withdraw funds privately.

The documentation is available at https://zeropool.network/docs/implementation/contracts-and-circuits/substrate/.

The implementation is available at https://github.com/zeropoolnetwork/zeropool-substrate.

ZeroPool Client Library

The ZeroPool client library, also known as zeropool-console is an essential component for testing and interaction with the ZeroPool network. It provides an interface for users to interact with the ZeroPool substrate pallet and perform operations like transfers, deposits, and withdrawals.

The implementation is available at https://github.com/zeropoolnetwork/zeropool-console.

Relayer

The relayer is responsible for submitting new transactions to the pool and updating the current Merkle tree’s root. Relayer helps to improve the node throughput by pre-checking the transactions, which is a computationally complex procedure. The issue is that without a relayer nobody can pay for gas for failed transaction

Relayer is available at https://github.com/zeropoolnetwork/zeropool-relayer.

We are also working on a new version of relayer, implemented in Rust. The source code is available here https://github.com/zeropoolnetwork/zeropool-relayer-rs.

Faucet

On the other hand, the faucet is a testing tool that provides tokens to users for testing purposes.

The faucet is available at https://github.com/zeropoolnetwork/zeropool-faucet.

Try ZeroPool

Testnet

Our testnet is available at substrate.testnet.console.v2.zeropool.network. Also, we have a video example, of how to use our testnet: https://www.youtube.com/watch?v=DQ8gbNTOP-g

Take the docker-compose file from https://github.com/zeropoolnetwork/zeropool-substrate/blob/main/docker-compose.full.yml and launch it. For test purposes, you may use params (proving and verifying keys, circuit data, etc) from https://github.com/zeropoolnetwork/zeropool-test-params. For production usage of our tech spec in your products we hardly recommend you run MPC trusted setup or wait til we stabilize the version based on recursive PLONK.

Plans of ZeroPool

The current version of ZeroPool is stabilized, but we are hardly researching the next generation of privacy and scalability technologies to improve the next version of ZeroPool.

Sequencer

Currently, we are supporting a set of relayers with Burning Auction consensus between them. But we suppose that a decentralized private sequencer is a more stable solution with more simple economics.

Recursive PLONK

Currently, Fawkes-Crypto supports PLONK, so ZeroPool can be built in PLONK with minor changes. BTW, to bring the most advantages of PLONK we should wrap all ZeroPool blocks to recursion.

Cloud Proving

We have fresh research on how to run Sangria in special mode, allowing us to do only O(n) complexity computations on the client side and the rest on the cloud prover. This approach allows us to improve performance on thin clients.

More details available at https://zkresear.ch/t/running-sangria-final-proof-in-shielded-mode-on-untrusted-3rd-party-prover/133

General Privacy

The next versions of ZeroPool will not be focused only on finance privacy but on general blockchain privacy, including multi-purpose transactions with user-defined scripting.

ZeroPool: Implementing Privacy-Preserving Transactions with zkSNARKs and Substrate was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

We have successfully published testnets for NEAR, Polkadot, Waves, and Ethereum. You are welcome to try them all on our website.

We also have reissued our website with an improved user experience. Moreover, we have now made our documentation available directly on the site, allowing users to quickly and easily access the information they need.

As you know we are not going to provide the final solution by ourselves but we go on our policy helping other teams to build privacy.

We have a collaboration with zkBob. Our team helps them to build easy-to-use privacy solutions for everyone.

We are pleased to announce that our cryptographic library, Fawkes-crypto, now supports PLONK, a highly efficient and versatile zero-knowledge proof system. This addition expands the capabilities of our cryptographic offerings and paves the way for the development of more sophisticated privacy applications.

Lastly, we are currently working on a groundbreaking project: creating general privacy solutions based on general zero-knowledge technology. Our goal is to develop a comprehensive privacy framework that is versatile, secure, and easily adaptable for various applications across different blockchain ecosystems.

In conclusion, we are thrilled with the progress we have made over the past year and are eager to continue our journey in building a more secure and private future for the blockchain space. We are deeply grateful for your ongoing support and enthusiasm for our project. Stay tuned for more updates and exciting announcements in the coming months!

As always, feel free to reach out to us with any questions or suggestions, and join our community on social media to stay connected with the latest developments.

You could always support us on gitcoin.

Website: zeropool.network

Twitter: https://twitter.com/ZeroPoolNetwork

Discord: https://discord.gg/TNesyTqFdf

ZeroPool news was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

We had some changes in our team. As one could expect, the whole of our team is out of Russia now. We go on work from different countries. Due to the relocation process, we had to miss some events. However, we took part in Gnosis meetup in Dubai. In spite of all global changes in the world and in crypto, we go on working on ZeroPool and developing it.

Here are some updates from the tech part of ZeroPool. We have delivered the solution compatible with EVM and Substrate. So here is one more option for our multichain solution. And we are welcome to cooperate with other projects in Substrate community to build privacy solutions.

We have updated the multithread prover for the computation of zero-knowledge proof on the frontend. So we got the process up to 3 times faster.

Our research part is also going on. We see the future, and partly presence belongs to PLONK. So we make our R&D for moving to PLONK and new polynomial protocols. Our target is to get the radical acceleration of computation. Research is always our priority. We believe that private technology has to be at the top of new technologies in the blockchain.

It was a short update. More news is coming soon.

Short update was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

We always say privacy is a fundamental right of businesses and individuals. And now some good news is coming. We are ready to tell you more about collaboration which brings more privacy onto blockchains — ZeroPool and Gnosis Chain (formerly xDai Chain).

Overview of ZeroPool

ZeroPool is a privacy solution for blockchain. We research a lot about new technologies and features to build privacy solutions in the blockchain. ZeroPool was created as a product for private transactions. But we believe that privacy on the blockchain is much more than private transactions. That’s why we decided to develop new technology and help others team build their own different solutions for privacy in the blockchain. One of our first partners we are ready to tell more details about is Gnosis (formerly xDai) Chain.

We use zkSnarks to hide the graph of transactions. So there is only visible the fact of the transaction but not the details. So nobody can see who and how much assents transmit until the transaction participants open this information.

What is Gnosis (formerly xDai) Chain?

In production since 2018, the Gnosis Chain (formerly xDai Chain) has provided a stable EVM (Ethereum Virtual Machine) environment for countless projects looking to scale and optimize transactions while maintaining Ethereum compatibility. The Gnosis Chain is a stable payments EVM blockchain designed for fast and inexpensive transactions. The chain uses a unique dual-token model; xDai is a stable token used for transactions, payments, and fees, and STAKE is a governance token used to support the underlying POSDAO Proof-of-Stake consensus.

The team is behind the first widely used bridge infrastructure (OmniBridge) for relaying assets and arbitrary data cross-chain, and BlockScout, a full-featured open-source block explorer. The team is currently readying a beacon chain implementation designed to support ETH 2.0 with faster block and epoch times, permissionless bridges, and rollup integrations while serving as an essential canary network for the ecosystem as a whole.

How does Gnosis Chain (formerly xDai Chain) use ZeroPool’s technology?

Large-scale privacy will eventually come to EVM chains as financial applications are getting more traction over time. Privacy is a vital requirement for the vast majority of financial services, thus introducing requirements for the solutions working on lower levels. Gnosis Chain leverages the ZeroPool technology to implement built-in privacy on EVM chains and cross-chain bridges. The data obfuscation solution on the Gnosis Chain will be closely integrated with the OmniBridge software. This solution plans to provide an ability for end-users to transact privately after entering the Gnosis Chain inside a zero-knowledge pool. Eventually, users will be able to deposit any arbitrary amount of any token to the zero-knowledge pool via OmniBridge, make private transfers using zero-knowledge proof technology (without disclosure of the sender, recipient, and value), and afterwards withdraw any amount of tokens owned by particular users from the pool.

Integration with the OmniBridge allows users to enter into the zero-knowledge pool by making only one transaction on the Ethereum Mainnet or other chains connected via OmniBridge to Gnosis Chain. Users will also be able to exit the zero-knowledge pool by initiating a transaction on Mainnet or other chains.

We go on working in collaboration and improving our technologies to increase privacy in blockchain.

ZeroPool and Gnosis Chain (formerly xDai Chain) was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

You could notice that Cheshire Cat changed anonymous in all our social media. Why? We decided Cheshire reflects our vision of privacy in a better way.

When we created ZeroPool on EthBoston it was like a fantastic experiment, a nice try, and an unknown future. Now it’s a little bit different.

We are still based on research, and we go on creating new technologies to protect your privacy in the blockchain. At the same time, we insist that privacy is an essential user’s right even in the blockchain. It’s not a crime or smth illegal. That is why Anonymous doesn’t reflect our vision of privacy on the blockchain. We are one step ahead from release, and that is why we would like to clarify our position.

It’s time to change was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

Privacy in the blockchain is always a tricky point of discussion. Some view it only as something secret, criminal, or what honest people never need. In reality, transparency is the fundamental property of blockchain.

People think if someone wants to hide something it’s something bad. But don’t you have private territory and secret passwords from your accounts? So let’s discuss who needs privacy in blockchain?

Blockchain is a specific area with great people who work to build a better world. But even in such a “perfect” world, founders are often faced with an old school question in the team “Why do I work more than that guy and get much less”. It’s so easy to check payments in the blockchain. So founders have to spend their efforts and time to solve these useless and unpleasant points. Such questions don’t play for team spirit. Blockchain projects are different from classical business and usually don’t have a great amount of HRs. Privacy could easily solve this problem. When you pay your team with private transactions they don’t have a way to check out how much their colleagues get.

Another important point for any business is corporate espionage. If you have some soft commitments with an important client or with a partner and you have to make some transactions your rivals can track it and act faster or try to influence it. So in this case privacy is also a solution. With private transactions, you can hide all important transactions from your competitors and better protect your business. With private transactions, you can feel safe and spend your efforts only on developing great projects.

Many of us remember last winter crypto whales were under tractions. The market was extremely interested when and where they put accents and what kind of transactions they do? And if you are a crypto whale, sometimes you need to make a big transaction but at the same time, you don’t want to influence the market. And obviously in this case you need private transactions.

Sure there are some other ways to solve all these cases but it takes much more time and effort compared with private transactions because you have to take several steps instead of just one transaction.

Privacy in the blockchain is an essential right as in the real world. Sometimes people need to keep financial secrets from others and it’s ok. It doesn’t mean one is a big boss of the mafia, it just means they have a private life. Private transactions are not a threat to the transparency of the blockchain.

Who needs privacy in blockchain? was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

Also, I would like to write some results for the year.

We are fortunate to have an impressive experience of decentralized work, so the new reality of 2020 wasn’t a challenge for us. The whole of our team worked smoothly on developing a solution for private transactions.

We could participate in the final offline event before COVID-19 EthCC in Paris where we presented our beta version.

During the pandemic time, we participated in zkSummit and talked about account-based privacy. It was a fantastic experience.

What we havee done in November and December

We concentrated on the implantation of our solution on polkadot and the absolutely new interface of ZeroPool.

So as a result of our hard work we have proof of concept of zkSNARK verifier on PolkaDot pallet. It’s done to have a strong base for ZeroPool implementation on PolkaDot.

During the development of ZeroPool implementation on NEAR Protocol, we faced there wasn’t support no_std support in borsh. But it was needed to implement our product on PolkaDot. So we solved this problem most simply. Our team added no_std support in borsh.

Besides this, we decided to make a totally new interface for our product. We spent a lot of time and effort on UI/UX. As a result, we have an application layout. For this, we used Redux (managing state of the application), Storybook (isolated debug), Jest (unit tests).

We are ready to share some pictures of our future interface. So it’s completely different from what you can see on our testnet and mainnet. And you are always welcome to share your thoughts about our interface in comments or in our Telegram

What we are doing just now

We have almost finalized our first step in cooperation with Web3. We are delivering the report. And we are working on our development of the implantation of ZeroPool on Polkadot.

We are in touch with some other blockchains about cooperation. We do it to give more opportunities to our users.

We hope to participate in the first offline events in the blockchain area after canceling all restrictions. And also we are looking for good online events to tell you more about privacy and our solution.

Our plans

We have to think of how to organize the audit of our product. Obviously, it’s a hard product for audit and not all researchers and developers can do it. There are too many innovations, details, and responsibility. But we already have some ideas.

ZeroPool November and December news was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

We keep on sharing our news with you. It can help you to be in touch with us and learn more about our product.

What we have done in October

As we mentioned last time we were working on a ZKP module standard on the base of the current tech stack and one more step is done to make transaction costs cheaper and bring even more privacy to ZeroPool. We implement account-based privacy in our circuit. More details you can find here.

Also, we optimized Fawkes-Crypto. It made our codebase more optimized with no loss in functionality. We keep in mind the upcoming audit and we want our code to be as optimized as possible. It can help to avoid bugs and reduce the time and efforts of auditors.

One more blockchain joined our solution. It’s Waves. This is important for us to make usage of ZeroPool as smooth as possible. That’s why we create a multi-blockchain solution. We believe it could solve more of your tasks.

What we are doing just now

Now we are working hard on dApp. We trying to do UI/UX as comfortable as possible. There are some challenges. It is a multi-blockchain solution, so it should be comfortable to change blockchains you are working with and at the same time, it’s an anonymous solution so there are private and public addresses.

Also in spite of hard technologies we use this dApp should be fast. It’s also an important reason to work on code base optimization.

Our future plans

We are going to make a pre-audit version in Q1 2021. So we have to rewrite all components in RUST and prepare a mobile version of our product.

Also, we are going to work more carefully on our white paper. Now we have some drafts of it but we want to have a full document

Our October was look like this. We are always open to the dialog and are ready to discuss the details of our product. You are welcome to try our beta version.

ZeroPool October update was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.

We decided to share some updates with you. In a perfect world, we are going to do it every month.

What has our team done?

We have added in Fawkes-Crypto several blockchains (Ethereum, NEAR, and Waves). Due to this step, we can get zkSNARK proofs acceptable for these smart contacts. It’s one more step in building a multi-blockchain private solution. There is more information about Fawkes-Crypto here.

Also, bls12–381 is added. It shows that Fawkes-Crypto helps to make zkSNARK proofs for new curves in Ethereum.

Besides we go on working hard on solutions for NEAR. It’s a perfect way of cooperation from both sides. Everything is smooth and in time. So we are extremely close to finalizing the first step of our common work. News is coming soon on this point.

As we decided to make a multi-blockchain solution we are working on attracting more projects to our solution. We won Web3 Foundation Grant Wave 7 and will tell more about it in detail soon. However, you can be sure our solution is going to be as comfortable in use as possible.

And what are we working on right now?

We are working on a ZKP module standard on the base of the current tech stack. It allows us to proof of anonymous transactions in the main blockchains, supporting smart contacts.

Our team is keeping in touch with the market and we were impressed by recent improvements in recursive composition. So we are working on supporting more up-to-date protocols like Plonk in Fawkes-Crypto. We are building a clear flexible solution which is ready to meet all changes and, first of all, new technologies. It gives us an advantage for building universal and truly safe product.

A couple of words about our plans.

We think about liquidity mining. We should compare all pros and cons before making a final decision. It is the solution with the anonymous distribution of reward token proportionately the amount and time of liquidity in the pool.

Our team goes on to build a universal solution for different blockchains for your comfortable use. So we are looking carefully for new projects for cooperation.

ZeroPool September update was originally published in ZeroPool on Medium, where people are continuing the conversation by highlighting and responding to this story.