ICYMI, TEEs are the latest crypto craze after dunking on 100 million dollar raises and the complicated social relationships between the ETH clique.

After explaining TEEs more broadly in a recent blog, we decided to dive deeper and invite speakers who are already using TEEs to share their experiences.

You can listen to the recording of the space here, or read on for all the insights.

On TEEs in general

Our speakers explained that TEE, short for Trusted Execution Environments, is a type of hardware-based security. In their most common form, a chip has a secure compartment. All the computation inside that compartment is inaccessible even to the host. This isolation of data and compute from the rest of the chip is a key characteristic of TEEs.

Interestingly, TEEs are not a new technology but have been around for decades. As Dan mentions, Apple has used the term secure enclaves to talk about TEEs in many of its keynotes for years to assure users of their privacy.

Why are TEEs becoming popular in crypto?

On that topic, Esli explains that as the AI and crypto hype continued, we quickly realized that ZK technology was too expensive and wouldn’t scale for verifying compute. Fortunately, TEEs have become widely available and can easily be integrated with other cryptographic technologies like MPC or ZK-Proofs.

“It’s not new tech. We’re already using it whenever you open an app on your ledger or unlock your phone using your fingerprint.” — Nukri

What’s more, TEEs aren’t just widely available; they are also much easier to implement for developers while allowing them to run more complex computations inside them. For example, all existing open-source code can be run inside of TEEs without modification.

So, in short, TEEs are popular in crypto now because:

• Widely available
• Affordable esp compared to use of ZKPs
• High Performance
• Verifiability for general compute
• Fast learning curve
• Ability to integrate with other cryptographic tech
• Necessary to make AI x Crypto work

David added to the AI topic that there was a lot of excitement around using autonomous agents in Web3, and rightfully so. However, once you entrust an AI agent with your data, you’ll want reassurance that all models are running as they are supposed to. TEEs then become a quick and cheap way to offer these guarantees without the necessity of trusting a centralized company. Beyond AI agents, any RAG setup will also benefit from verification provided by TEEs. You can read some more about RAG in the context of SQD here.

On using TEEs in Products

Currently, SQD has not yet integrated with TEEs. But it’s part of our roadmap, and there are three aspects we’ll likely rely on them:

• Trustless ingestion: allowing other participants to add onchain data to the SQD data lake
• Verification of query results: by running queries inside of TEEs
• Trustless indexing: solves the scenario where one users wants to rely on someone else’s indexer without needing to trust it.

For investors, as David explains, it appears that confidentiality is playing a big role, but what’s often lost is the verification aspect. With ZK, we could prove that computation was done correctly, but it was expensive and took time. TEEs offer a faster, practical way to verify data hasn’t been tampered with. This then opens up two categories:

• Co-processing: where smart contracts are hooked up with off-chain sources, expanding their capabilities, basically making them smarter
• Trustless computing: An example of that is Super protocol, which allows users to run apps in a decentralized cloud, offering verifiable trustlessness.

Speaking of Super, Nukri provides some areas where its customers have found a decentralized compute platform useful, including intellectual property management, digital marketing, and AI.

In Marlin, TEEs ensure the integrity of computation, providing devs a way to verify on- and off-chain that computation has happened while adding an additional layer of confidentiality.

Beyond deAI…

So far, you could end up thinking that TEEs are only helpful for doing AI x crypto things, but there is much more, from block building to decentralized frontends. Decentralized frontends are a topic that makes a comeback ever so often, but this time, chances are they are here to stay.

“An increase in exploits and compromises of large dApps has lead to an increasing popularity of decentralized frontends.” — Esli

After all, the piece that gets exploited is usually the most centralized part, and for most dApps it is the frontend hosted in the cloud. While previous iterations of decentralized frontends were complex, with TEEs, one can easily host whole dynamic websites, allowing users to ensure they interact with the correct app without UX compromise.

Surely there are downsides, though?

As Dan shares, there have been multiple exploits on intel TEEs in the past, which has given rise to the perception that TEEs are vulnerable. In the end, everything has trade-offs.

“There’s always a need to weigh the cost of attacking the TEE against the value of what it’s protecting.”

Still, TEEs are already used in a billion devices, and with increasing attempts to build open-source TEEs, developers will have further ways of mitigating attacks. One can always throw some MPC or ZKP at the TEE for added security.

What does the future hold?

Esli hopes for fully open-source TEEs and TEEs to become the base layer for anything done off-chain. Nukri foresees the rise of fully compliant web3 infrastructure and the rise of useful applications. He’s most interested in AI agents on private data, big data, and IP management.

David believes we’ll see a lot of innovation in open-sourcing chips, which will affect how private keys are created. For example, when the hardware owner creates the key, not the manufacturer.

For Web3, novel functionality should unlock the experimentation necessary to discover new use cases. Think of DeFi, which started as a big experiment and remains the largest sector of Web3 so far.

The future for TEEs x Crypto is bright. The current hype is justified.

Thanks to our speakers and everyone tuning in.

Exploring the potential of TEEs was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

GM fellow SQD community member,

With the end of August also comes the end of brat summer, which, let’s be honest, we won’t be that sad about.

At SQD, instead of focusing on keeping on top of all the trendy jargon among the terminally online, we spent our time integrating with new networks and launching a new website.

But before getting into all the details of what we’ve been up to in August, here is a look at the numbers:

SQD in Numbers

• Worker Nodes online: 1016
• Data Stored: 775 TB
• Queries in the last 24 Hours: 565,783
• Data served in the last 24 Hours: 15 TB

New SQD Website

Network usage numbers aren’t the only thing that went up at SQD. We also completed our rebranding with an overhaul of the SQD website.

It’s now up and live at sqd.dev. Go check it out and let us know what you think. In line with our new domain, our X handle is now @helloSQD. Please be careful to follow the official account for official communication.

Hacking to the Gate

And yes, this title is an Anime OP reference. Listen to it here while you BUIDL.

On August 8th, The LATAM Onchain Open Hack officially kicked off, incentivizing builders to create dApps and services on Base using SQD. Since then, devs have been busy building and we’re looking forward to seeing project demos soon. After all, SQD is all about supporting devs in creating with data that they otherwise couldn’t.

In the meantime, check out this presentation our very own Natalia gave during the Opening Ceremony.

New integrations

The Hackathon participants weren’t the only ones busy. Our devs were cooking and integrated countless new chains in our data lake.

Among others, we integrated Aleph Zero’s EVM L2, Arthera, Testnets for Berachain and Shibarium, Gravity Chain, and Plume. For a list of all the networks integrated, go to our docs.

All things TEE

Keeping in line with the current hype cycle, first, we wrote a blog post explaining why everyone is suddenly talking about TEEs…. TL;DR: They are widely available and cost-efficient and can provide verifiable compute, connecting off-chain and onchain trustlessly.

And to deepen the conversation we brought together speakers working in and researching TEEs to discuss whether the hype is justified. Spoiler: reasonable hype.

Listen to the entire conversation here and if you have more TEE questions, you know where to find us.

SQD in the Press

In August, SQD was also featured twice in the press.

• Searching for the Google of Web3
• Blockchain Data Indexing key to Web3

Both articles highlight the data challenges developers face when building in the current multichain and modular crypto landscape. SQD is mentioned as a platform that addresses these issues and facilitates data extraction at the scale necessary to support the broader adoption of crypto.

Thanks for being part of the SQD community.

By the way, SQD now also has a Farcaster account; if you’ve reached your limit with X, connect with us there. We’re looking forward to any feedback on the new brand and continue working on executing our game plan.

Stay tuned and hydrated.

Your SQD Team.

Monthly Recap: August 2024 was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

After announcing our partnership with M31 Capital, our two co-founders went live with M31’s David Attermann to discuss our plans and share some insights into the reasoning behind SQD’s development.

You can view the full recording here: https://x.com/i/broadcasts/1nAKEpvkYpbxL

Why did M31 invest in SQD?

Normally, you see announcements of investments after the fact, and there is little transparency in investors’ reasoning. Not this time. During the Broadcast, David from M31 shared their thesis and explained how SQD fit into that.

In general, M31 is focused on infrastructure and middleware, and in particular, the data layer. While well-understood in the traditional web it is a very underrated and misunderstood section in Web3.

Ever since the Google Cloud partnership, SQD has been on his radar. He started doing due diligence on it, diving deep into the network architecture and quickly realizing that it was a completely different way to do data. Moreover, he quickly understood that once fully developed, it would enable data functionality not even possible in Web2.

In short, David had come across exactly what VCs dream of: an investment opportunity that is unique but currently misunderstood. As he put it, SQD will provide valuable functionality in the future and, therefore, is a long-term investment.

A decisive factor was also that SQD emphasized verifiability and designed the network with unit economics in mind to ensure that even growth in the billions would not break it or make it extraordinarily expensive to use.

Overally, M31’s thesis aligned very closely with SQD’s values, and David shared he knew instantly that our Co-Founders were the real day (which regrettably is rare in this industry).

Why is SQD excited about working with M31 Capital?

Marcel, SQD Co-Founder, shared that it’s very rare for a VC in crypto, especially a liquid investment fund, to feature such technically sophisticated people as we encountered at M31. Since SQD is laser-focused on product, it’ll be highly beneficial to tap into the expertise of the M31 team.

Origin Story of SQD & Differentiation

SQD CEO and Co-founder Dmitry Zhelezov went down memory lane, sharing that SQD started three years ago as an attempt to fix a specific problem: trying to index blockchain data for a video streaming platform. They quickly discovered that the bottleneck for accessing onchain data was the RPC taking hours to sync without providing any flexibility. As a quick fix, they started adding data to a more efficient database, and the first prototype was born. After a Hackathon win, projects started expressing interest and Dmitry realized he had to build something that scales and not just rely on a simple database. That’s how SQD was born.

In the long run, SQD isn’t just for indexing but an attempt to completely overhaul the way Web3 data is accessed at scale. It combines DePIN with the advances in data technologies from Web2.

Scalability

Through decentralizing the data lake, it scales with every node. Each worker contributes 1GB per second in throughput capacity, and each node is a small DB in itself. Therefore, as a whole, they are able to pool Terabytes of data and deliver them quickly to consumers.

By tailoring to a specific problem, SQD achieves a 100x more efficient solution for querying. It’s not a one-size fits solution as other indexers.

Flexibility

There are probably hundreds of data use cases we haven’t even thought of. That’s why SQD stores raw data directly, providing devs with a familiar experience while also allowing them to be creative with it.

By putting all the tooling to handle data on the client side, we solve data access at scale without worrying about how data is encoded or the different implementations that exist. Everything is solved on the client side, where devs can split and manage data as they desire.

Network Economics

As David points out, this is an area that is not enough discussed in crypto. Yet, at SQD, we got it sorted. The main idea is to balance supply with demand and decentralize over time. To achieve decentralization, hardware requirements for running worker nodes need to be low to allow people to run nodes even when rewards are small.

Of course the question then is: but how do you know the right amount of tokens? You don’t. That’s why, instead of fixing inflation from the start, SQD opted for a fixed reward pool for the bootstrapping period — structured in a way to achieve maximum decentralization. This also means that some investors weren’t happy to realize that the more they delegated the lower their rewards became.

[At this point, Marcel mentions that people in crypto need to realize that to earn rewards one has to work — like in the real economy — and not expect to get money for just being there…]

Anyway, there is a good reason for this, as it’s best for the network to have many people delegate and run worker nodes rather than just a select few.

Once the bootstrap period is over, we’ll better understand rewards and costs and set inflation accordingly. At the same time, we’ll likely trigger a fee switch on the supply side, where people who have so far locked tokens for higher rate limits will switch to a subscription model.

Game Plan

The entire game plan can be found here.

Gateway 2.0

Currently, to access the SQD network, consumers have to bond SQD and run a gateway — basically a piece of software that acts as a proxy service. It’s sequential, and you have to wait for responses. With the introduction of 2.0, all of this is getting smarter. Gateway 2.0 will pre-fetch and allocate data, eventually eliminating the need entirely to have an RPC on the downstream client.

For users, it’ll be a simple interface where they can stream data based on their filters in real time. Some big names will be coming as partners to facilitate this — especially from the RPC space. RPC providers will be able to join SQD as sellers streaming data with little upfront investment. Watch out for more on this 👀

Light Clients

Another area of excitement among all the speakers are light clients. Already pretty hyped in Web3 for their ability to verify chains without needing to run an entire node. On SQD Light Clients refers to facilitating the handling and cloning of a DB straight on a users device.

Dmitry explains that usually Web3 is always a few steps behind the Web2 data world.

“In Web3, we’re still stuck in the postgres area, while Web2 has long moved on.” — Dmitry

At the moment, SQLite is the latest innovation, a lightweight production-ready database that can be put on any device. It’s already widely used in mobile apps. Dmitry uses Instagram as an example to illustrate its power.

“When you open instagram, even offline, then you still will see a feed full of things, and that’s largely thanks to local caching and storage.”

Trustless access, decentralization, and SQLite are a perfect combination. Once implemented, users could run indexers locally and even enrich them with private data, using secure enclaves for processing without latency.

While use cases are limited so far, Marcel is confident that builders will discover hundreds of them. David adds that this also allows projects to scale efficiently without the usual data infra overhead — after all, smartphones are everywhere and powerful.

Verifiability

The last topic the three discuss is verifiability.

All agree that most people don’t care that much about decentralization. What matters, though, is verifiability, especially with the rise of AI. As David explains, in many of the current use cases alleging that OpenAi (or another AI company) used data without consent, it’s nearly impossible to prove. Verifiability is lacking.

With blockchain, there is a way to add verifiability to AI agents. As a large data provider ability to verify also plays an important role in SQD.

There are two parts to it:

• the ability for users to verify that the data providers have added correct onchain data
• the ability to verify that the query has run correctly

For the first, adding simple hashes might be sufficient since data is mostly public anyway. For query verification, Dmitry points out that TEEs are a promising technology as they allow low-cost computation with integrity.

ZKPs, while more expensive, can also be used in a setup with optimistic sampling, where disputes are only submitted when a user is 99% sure that an output is invalid.

For any further questions on this partnership or SQD’s game plan, please join us on Discord.

Insights from SQD’s Broadcast with M31 Capital was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Crypto sure loves its acronyms. If you entered the industry early enough, you’d recall learning about PoW vs. PoS, and if you ventured further, you’re also familiar with BFT. In the last cycle, it was all about ZKP, and then FHE entered the room. In between, we also talked about MPC, not to be confused with NPC, but we quickly realized that it didn’t have the trust guarantees many of us wanted and was expensive.

Even though we still lack real implementations of FHE and MPC remains niche in crypto, we’ve moved on to the next three-letter acronym: TEE.

If you go on X, or watch talks from the latest crypto conferences, everyone is trying their best to come up with witty titles for their talks, such as “How to win friends and TEEfluence people” — this is a Carniege reference, in case you weren’t aware — when “TEEnage dream” was right there or you could have ripped off the French Revolution’s call for LiberTee, EgaliTee…

Anyway, where there is smoke, there is fire. The only question is: how big is that fire, and why is it there in the first place? Why does everyone talk about TEEs as if they were a panacea for all of crypto’s problems?

The best place to start in addressing that question is:

What are TEEs?

Just like ZKPs or MPC, TEEs are not a crypto-native invention. Instead, they’ve been around for a few decades, and if you have a smartphone, you’ve unknowingly used TEEs. TEE is short for Trusted Execution Environment and describes an environment in which a program can be run with special, powerful protection.

This sort of environment is also often referred to as an enclave. TEEs exist as a set of instructions on hardware or can live in the Cloud. They offer a secure area of the main processor that guarantees that code and data run inside are protected and kept confidential while ensuring their integrity.

TEEs are an isolated spot that no one can mess with, not even the host of the TEE itself. Hence, their appeal to crypto people. Imagine being able to put a TEE on your enemy’s blockchain node, and you still don’t have to worry about the integrity of the computation.

A good mental model to apply to TEEs is that of a trusted third party, which runs arbitrary computation for you in an inexpensive way. The opposite of a blockchain, which makes running computing very expensive.

In addition to offering a magic box that adds privacy to whatever you put in, TEEs can also offer attestations to prove to any party that a program is running, allowing users to know exactly what they are interacting with.

In short, TEEs make everything better.

Understood, but surely there is some downside?

Well, if you are very web3 native, you’ll be unhappy to hear that TEEs are all manufactured by large chip manufacturers and open source is still trying to catch up.

Currently, the largest in the TEE market is Intel — the same Intel that just announced the layoff of 15,000 employees, and it’s not because they are being replaced with AI. Rather a realization that the market has become a bit of a bubble.

Still, for now, their TEE business is going okay. Their TEEs run as instruction codes available on existing hardware alongside encrypted memory. Others in the TEE market include AWS with their AWS Nitro cloud product, ARM TrustZone, mostly in mobile devices, and AMD.

Even if TEEs don’t take off in crypto, they already enjoy widespread adoption in traditional IT infrastructure, from use in financial systems to authentication and digital rights management.

So, we’re supposed to trust that intel?

Ironic, isn’t it? But in a sense, we already trust AWS that they won’t rug Ethereum, so how big of a deal is it to add to the cognitive dissonance?

Fortunately, though, using a TEE does not imply trusting intel. And although there is a crop of crypto people hating on TEEs because of a supposed centralized entity controlling them and the existence of bugs, one should take a more nuanced approach.

“TEEs are extremely cheap and extremely fast, but they don’t have the sex appeal for some reason. We’re trying to change that.” — David Atterman, M31 Capital

For one, intel can’t really control the TEE, which is kind of their point. They cannot censor your applications as they have no way of knowing what you attest to (remember, the compute is in the box, sealed off from outsiders).

To prevent the risk of physical access to the machine, it helps to run them in multiple locations and to prevent the potentially devastating side-channel attacks; there’s actually some neat workaround running light clients inside of enclaves to ensure the integrity of activities inside of it.

And it gets better, you can also combine TEEs with ZKPs to get privacy and reduce the dependency on the TEE alone for correctness.

TEEs are great, I get it. But why is everyone talking about them now?

Well, for one, every cycle needs its new acronym. Add to that the ability of TEEs to provide privacy without being overly expensive, and you get a few interesting use cases. TEEs are like ZKPs but already production-ready and well-established.

It’s practical, widely accessible, and allows for prototyping while we figure out how to make the other technologies (MPC and FHE) good enough for wide-scale use.

A few potential use cases are:

• Managing Decentralized Identities with TEEs
• Creating private mempools or dark pools inside of TEEs
• Using TEEs and ZK prover in combination with 2-FA for rollups enhancing integrity and privacy
• Running block builders inside of TEEs (this warrants a whole separate post to cover)

And then there’s decentralized AI, the other hot topic du jour. Centralized AI is coming increasingly under scrutiny for its questionable practices of using other people’s content without their consent to train models (OpenAi is guilty of this, and so is Nvidia). Even Elon Musk, not really a man of the common people, is now alleging that OpenAI is up to no good. Well, whoever sits in the glass house probably shouldn’t throw stones…

What all of these issues with current AI monopolies fuel, though, are the aspirations of teams building solutions that distribute power while training their models on data that retains its privacy and pays data owners.

The reason AI companies try to get as much data as possible, even if it means spinning up new servers to bypass scraping blockers of YouTube and Netflix, is that any model is only as good as the data it’s trained on.

Decentralized AI promises to create collaborative environments where access to sensitive data is limited. Blockchains are used only to verify computation but not for the actual computation. TEEs offer an attractive solution to provide security for sensitive AI computation, throughout which data remains encrypted and protected.

This means data owners could share private data without the risk of loss of confidentiality or ownership. TEEs make the system even more trustless, especially when run in combination with ZK proofs and blockchain.

TEEs add a spark of magic by bringing privacy and integrity to arbitrary compute. No wonder everyone is talking about them in crypto.

“Any sufficiently advanced technology is indistinguishable from magic.” Arthur C. Clarke

At SQD, TEEs are part of our game plan and are scheduled to be released at the end of 2024. They will allow the network to retrieve data through TEEs. We’ll share more details on implementation as we get closer to pushing it live.

In the meantime, stay tuned by following us on X.

Why is everyone suddenly talking about TEEs? was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

“In an information industry, the cost of monopoly must not be measured in dollars alone, but also in its effect on the economy of ideas and images, the restraint of which can ultimately amount to censorship.” — Tim Wu in The Attention Merchants.

Chances are you’ve found this article while browsing through X. It’s every crypto person’s favorite pastime. This endless scroll shows us anything from the latest memecoin rug to VC hot takes and, occasionally, comedy gold.

Mingled within are ads that tend to be completely irrelevant to what you care about. I, for example, am constantly getting gaming ads, although I have not even touched my Steam account in over a year nor played games outside of chasing yield. Even as an Elon Fan boi, you’ll have to admit, the experience has gotten worse, from not seeing content from people you follow to a stream of misinformation with a pinch of more questionable takes in community notes.

The problem is that even as crypto users, we are somewhat hostage to X because where else could we go? If suddenly, X started to ask us all to pay for use of the service, we’d probably do it — and countless companies have opted to pay for premium instead of quitting the platform, hoping it’d at least aid their reach.

The process we’re witnessing of platforms providing people with value to ones that extract from them is one that Cory Doctorow has dubbed “enshittification.” Instead of explaining in length what it means, here are a few suggestions on what you can do to feel it:

• Try finding a product on Amazon where the first page isn’t full of ads
• Do a Google Search
• Try to see a tweet from someone on your X timeline you actually follow.

For the super courageous, go to Facebook, which turned from a place where we poked friends to a place where tinfoil hat aficionados get triggered by AI-bot-supplied rage bait. It’s worse than the dead internet; it’s the Zombie Internet. Social connections? No mas.

Then the question becomes, why don’t we just opt out? This is the point where we have to realize that these platforms have a monopoly, which they’ve managed to sustain thanks to linguistic acrobatics. If you are trying to grow your professional network, where do you go outside of LinkedIn?

At first, these social networks were actually useful. Remember the early days of Facebook (if you are old enough, then you might want to join the Zoomers in crypto telegram group)? The timeline was simple, full of your friend’s stupid takes. That was the stage where there was actual value provided to end-users. Then things shifted, as Facebook realized there was much money to be made from businesses by selling them ads and giving them advanced targeting capabilities. The platform has gone entirely unhinged, squeezing both companies and users.

Look at any platform that reached a big scale in recent years, and you will see the pattern repeat. Potato patata. Even TikTok already displays the dynamics, and that’s the youngest among them. Not even the CCP can fix this. Technology beats ideology.

But maybe crypto can provide an alternative.

Let’s quickly recap what’s bad about the current state before exploring why crypto is the messiah (or isn’t).

• Privacy is non-existent, although, to be fair, we’re primarily willing participants in our own panopticon so it’s not clear how much we shall blame that Google knows what we ate for breakfast on them.
• Data ownership is impossible when all the data we generate miraculously belongs to and is mined by the platforms to show us ads (aka monetizing attention)
• Algorithms shape culture, yet decisions are opaque and made by a handful of mostly tech bros in Silicon Valley Ivory Towers.
• No Freedom to exit: you exit, you lose all your connections. The current social media platforms are closed black boxes, offering limited access to their data and infrastructure.
• Misinformation and censorship
• Monopoly powers : every time a competitor grew too large, the Big platforms simply bought them up. There’s no incentive to innovate or improve when the current system works. Businesses have to use these Social Media because that’s where their users are.

“We are all mine sites now, data mine sites, and despite the intimacy and import of what is being mined, the mining process remains utterly obscure and the mine operators wholly unaccountable.” — Naomi Klein in Doppelganger.

The one comfort is that at least if we’re to believe Cory Doctorow, the current state of these platforms will lead to their death.

Ok, but if X really died (this is purely hypothetical anyway), where would we go from there? There are different answers to that. Maybe just go live in a hut in the woods, reconnecting with Nature.

Alternatively, there are social networks that try to rid themselves of some of the problems with Big Tech by distributing power and giving users more control.

Social networks, but decentralized

People in crypto weren’t the first to notice the issues with how big social networks operated. Already in 2009, researchers developed an alternative to the likes of Facebook, a fully peer-to-peer architecture that was called PeerSon. It never took off, but the efforts continued.

Nowadays, Mastodon is probably one of the more successful examples of a network that isn’t controlled centrally. By providing interoperable code deployed across websites, it allows developers to create their own servers — called instances — which they can operate, own, and moderate as they desire.

And then there is Bluesky, the pet project of the previous Twitter CEO, now turned Bitcoin Hippie Jack Dorsey. Currently, Bluesky has 6 million users, of which 1.3 million are active at least once a month. Unlike Twitter, it’s built on an open-source framework, giving anyone outside the company insights into what’s happening.

Sounds already pretty crypto-aligned, right?

Decentralized socials like Lens and Farcaster take things even further. These networks prioritize user autonomy and censorship resistance and provide resilience against single points of failure. They aren’t based on email addresses but operate on top of crypto rails, with users relying on their wallets and public key encryption for sign-ups.

Lens currently has around 30,000 interacting users, and Farcaster counts a total of 621,000 users, but only 10% of those are daily active users.

Compared to Bluesky, one can’t help but think:

Sure, but a bigger number isn’t always better, or is it?

At least, you can tell yourself you’re still early.

Even though technically, we’re not because decentralized social networks that use crypto have been tried various times before, as with Steemit. They just never took off.

Maybe now is the time. At least in theory, DeSo sounds great as it:

• Offers enhanced resilience
• Enables users to take their data and leave
• It tends to be open-source, enabling suggestions and improvements
• Provides increased censorship-resistance
• Integrates native crypto payments
• Empower developers to build on its distribution

And there aren’t a whole lot of other significant internet use cases beyond commerce, finance, and social, so we might as well give it a try. What’s the alternative anyway?

Be a willing contributor to your own (data) exploitation without any guarantee that your account will still be available the next day?

But the reasons to at least try DeSo aren’t just that the current centralized platforms aren’t serving us well; there’s also an opportunity to shape how we envision social experiences — making social platforms actually social again.

On using DeSo

As the author of this post, my primary exposure has been to Farcaster, even though I got my hands on a Lens handle way earlier than an FID. So, these thoughts are somewhat subjective, although they illustrate some bigger challenges and opportunities for these platforms.

While architecturally different, both Lens Protocol and Farcaster aim to provide an ad-free social experience where users get to choose how they interact. The underlying protocol aggregates all the important data, such as follows, likes, and posts, while anyone communicating on top does so through a client of their choice.

The good

Open Playground

This creates an open playing field for developers to build on, with countless dApps built in and around Lens and Farcaster’s social stats. On Farcaster, for example, there has been a rise of “mint your profile and trade it on a bonding curve” dApps. Not that this fosters mass adoption, but at least it’s the type of experimentation where, eventually, someone comes across a mechanism people actually want to engage with long-term.

Choose your algo

Another benefit of having a social protocol is that one can design a client with a different algorithm. Currently, the main client on Farcaster is Warpcast, built by Merkle — the company that developed the protocol. However, not everyone is happy with the way content is aggregated to the main feed by some way of prioritization.

Instead of just complaining to DWR, the founder, a handful of devs have chosen to simply build their own client that has a reverse chronological feed.

Vibes

This is extra subjective. Still, it would seem that at least the more toxic X gets, the more people can be enticed to try another social network simply based on Vibes. Lens and Farcaster were initially running on an invite-only base, allowing them to curate a fairly cohesive group that created its own subculture, where “rules” were loosely enforced through community and a /dont-do-this channel.

I didn’t start using Lens frequently because when I signed up, there was mostly talk about NFTs, and I wasn’t all that into those despite holding quite a few. On Farcaster, however, the existence of channels (theme-specific feeds similar to sub-reds in Reddit) provided a place to nerd out about non-crypto topics and look at fun memes, all while staying on top of what’s happening in EVM land.

It also created a way to connect with people over your niche interest. I ended up joining a boardgame meetup, discussing great books, receiving countless postcards from fellow casters, and being “promoted” to co-host of the classical music channel.

These are more wholesome interactions than are often possible in a crowded, opaque place like X. But not everything is rainbows and unicorns on DeSo.

The challenges

Insiders and Outsiders

For now, there seems to be a sentiment on Twitter that people on Farcaster are their own in-crowd and should stop telling everyone to join their platform. This happens with every new platform, and it’s somewhat logical that people who joined early have accumulated a higher reputation than those who just joined. With a little effort, anyone can find their crowd.

There’s no guarantee that either of the current DeSo will survive, but you might still make frens along the way or learn a thing or two.

Bots

The Degen Airdrop was a watershed moment for Farcaster, making some early users a little rich while attracting a whole new audience of airdrop farmers to the platform. The rise of the bots began, and even though one has to pay storage fees, the infestation seems unstoppable. While users can report suspected bots, chances are whoever supplies them will always be winning that race.

Spam & Freedom of Speech

Bots often bring the next nuisance to platforms: spam. Warpcast has a filtering mechanism that hides replies that are most likely from spammers, yet this will likely be a challenge for every social network, decentralized and not, for the unforeseeable future.

While freedom of speech is often touted as the one thing that’ll fix all of social media, as we’ve seen with X, when you take guardrails of it, a lot of weird stuff finds its way on the timeline, from blatant lies to sparkling conspiracy, all sort of non-reliable content fills the feed.

Just allowing everyone to post everything is one thing. Freedom of speech is a noble goal, yet what’s often forgotten is that it should not mean being free of accountability or consequences. The ideal state for social protocols will likely be permissionless on the protocol level and filtering/moderating on the client level.

How that’ll be achieved in a decentralized ecosystem remains to be seen. Facebook spends around $3.7 billion on content moderation, more than X revenue — and still, the feed is not one you’d want to browse. Moderators frequently quit due to gruesome working conditions and PTSD.

Solving censorship by allowing everyone to post anything is only one part of the equation. In my view, creating a place where real humans want to spend time will require consistent effort in moderating what they see while balancing that with not throwing them too deep into their own filter bubbles.

“What if the powerful can use information abundance to find new ways of stifling you, flipping the ideals of freedom of speech to crush dissent, while always leaving enough anonymity to be able to claim deniability?” — Peter Pomerantsev in This is not Propaganda

Even with those challenges unsolved for now, it’s worth spending some time on decentralized social networks. Try them out, and who knows, maybe you enjoy them more than you expected.

Just like any social platform, they give rise to their own insider jokes, memes, and movements. The noise-to-signal ratio is still significantly better on the DeSo, also because they are still small.

For now.

And if the one topic or movement you want to see in the world isn’t there yet, why not go and create it? Instead of chasing big numbers, the way to get the most out of a social network is to chase quality connections and conversations. Unlike blockchains, human relationships don’t scale.

“It seems that in a world where people compete with numbers, it is the numbers that always win” — Ronan Hession in Leonard and Hungry Paul

SQD also has a channel on Farcaster, so feel free to drop by to share feedback, memes, or questions.

Obviously, in a world where all social experience is powered through onchain data, we expect SQD to play a significant role, with our CEO frequently alluding to the possibility of having customized feeds powered by local indexing on a user’s device.

If there is any DeSo data you’d like to access using SQD, get in touch.

Written by Naomiii

The Crumbling of the Ivory Towers was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Did you know that July was named after Julius Caesar?

And just like him, at SQD, we have grand aspirations, although we do our best to avoid dinner parties iykyk.

This month, there’s been a lot happening, from a major rebranding to enterprise partners and the release of our Romane Empire, aka the game plan. Here’s a recap of it all.

But first, the numbers.

Network in Numbers

• 950 Worker Nodes online
• 703 TB Data stored
• 138,545 Queries served in past 24 hours
• 93,551,227 SQD delegated

So Long Squiddie, thanks for all the Fish

At the start of the month, SQD announced a significant transition, waving Goodbye to the Squid on our logo and welcoming a more mature interpretation of what Subsquid Labs is trying to achieve. The rebranding marks our transition to a production-ready platform ready to onboard major clients from Web2 and Web3.

Read more here.

SQD on its first DEX

A few days later, the native SQD token went live on Arbitrum on its first DEX: PancakeSwap. The PancakeSwap team was already a heavy SQD user, making it a logical choice for our first DEX appearance. And with its high liquidity, user-friendliness, and focus on being an all-in-one DeFi platform, it’s also a fun place to trade.

New Game Plan + AMA

Another milestone for SQD was the release of our game plan. People often put SQD in a box, comparing it to other indexing protocols, without realizing that, the flexibility of our layer enables much more than that.

For an impression of what we’re aiming to build in the long run (spoiler: a data access layer that eventually outcompetes Web2), read our game plan or listen to the AMA we hosted with SQD CEO Dmitry Zhelezov.

Partners & Integrations

In July, we also entered one new enterprise partnership and shipped a Proof-of-Concept, showcasing what’s possible when combining AI analytical tools and SQD onchain data:

• Deutsche Telekom running worker nodes on SQD. LINK
• Kelp Analytics’ integration provides an interactive tool to analyze Uniswa Price Impact powered by SQD data LINK
• M31 enters strategic partnership with SQD LINK

Events

Centuries after Caesar, Charles V was the most powerful man ruling the Holy Roman Empire — and despite the name, the center of his power wasn’t Rome but Brussels — the city that, in 2024, became the venue for ETH CC.

While you might have seen Tweets about getting robbed or Vitalik jumping through the rain, at SQD our focus was hosting an Open Data Brunch to discuss all things data, together with our partner Google Clouds.

Thanks to everyone who joined! No seats were left empty; no one was left standing in the rain.

Other notable mentions

AI x Blockchain Use cases X Space

Although we build regardless of the current meta, we can’t fully escape the AI trend. We’ve previously clarified that we’re as much an AI project as a DePIN project, and SQD’s main proposition for AI is enabling data access at scale. But then it might matter less what we say than what others perceive us to be.

Undeniably, a lot is happening so for a deeper dive into what’s possible at the intersection of AI and Blockchain, we invited speakers from Guru Network and 0G Labs to discuss

“Once the infrastructure is in place, AI agents can do anything humans do onchain, from trading in DeFi to checking in on your portfolio to minting NFTs.”

For further insights from that space:

AI meets Blockchain

dAGI House

Further fuelling the demand for AI-related content, SQD CEO Dmitry spoke during dAGI House, an online event series, about what he believes will happen with LLMs in the future and how SQD fits into the picture.

The main takeaway here is that the edge eventually won’t come from building the most sophisticated neural network but from the data you access to fuel it and how expensive that is. You can watch the talk in its entirety here or read our recap of Dmitry’s insights on the SQD Blog here.

Thanks for being part of the SQD community.

FYI, a shorter version of this has also gone out via email to all newsletter subscribers.

Monthly Recap: July was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

There have been many changes at SQD in recent days, from a major rebranding from Subsquid to SQD to a blog post that shares our game plan for the future. If you still think that SQD is just an indexing protocol, you’ll be surprised.

For now, the niche SQD is serving is certainly indexing, but that’s a small percentage of the whole vision. During the X AMA on July 18th, SQD CEO and technical Co-Founder Dmitry Zhelezov answered community questions and shared further insights into what’s next.

You can listen to the recording here. Read on for a recap of the most important things discussed.

Thoughts on rebranding

Dmitry also occasionally uses Subsquid instead of SQD, quickly correcting himself and pointing out that it’s still a rather recent transition and that we’re still getting used to it. More importantly, though, it signals that SQD went from a research and development project to being actually production-ready.

“At a lower level, we’ve already enabled data accessibility. The next step is elevating it to a larger market share and going beyond indexing to serve niche verticals such as AI, analytics, and, of course, the evolving ecosystem of dApps.”

On philosophical association

“You can discover more about a person in an hour of play than in a year of conversation.” — Plato

Since Dmitry’s official description on LinkedIn for the longest time was Chief Philosopher, it begs the question of what type of philosophy he pursued. Now we have achieved clarity on that.

SQD CEO sees himself as a disciple of Plato, who gave platonic relationships their name and was more a vibes person than his platonic friend Aristotle. Together, they belong to the greatest thinkers of Western philosophy, even though Stoicism sells more books these days.

“I’ve been in academia doing math for ten years. One of the themes you notice is that there is a world of ideas, and the most fruitful ones are often found by different people simultaneously without them being in communication. That suggests that things aren’t necessarily invented but discovered. At SQD, this also applies in that things evolve based on their internal logic.”

SQD’s current focus

For now, the most common perception of SQD is as an indexing solution that sits in the same bracket as The Graph. Dmitry explained that usually, developers have smart contracts or other events they need to observe to display in a user interface. To facilitate that with blockchains, a process that constantly listens, extracts, decodes, and puts the data in a digestible format. To facilitate even something seemingly simple as showing a user the volumes traded on a specific NFT collection requires organizing a lot of onchain information. And it needs to be fast since people are impatient.

That’s what indexing facilitates. And the current main audience for SQD has been dApp builders who need indexing. He further highlighted that more have been migrating their infra from the Graph, thanks to SQD’s speed and flexibility.

A few dApps Dmitry is very excited about are Railgun, who rely on SQD’s indexing of holistic onchain data to provide a privacy solution to using Web3 and Decentraland.

“Decentraland is this OG project with two cycles behind them. There’ve been some big Metaverse and NFT hypes followed by long winters. The team continued to build and eventually, it’ll be those teams pushing the industry forward. That’s why I’m very excited that they are migrating their infrastructure to SQD.”

SQD’s “end-game”

In the end, it’s not just about providing a better indexing solution for current dApp builders. For SQD Co-founder, the big vision is magnitudes larger than that. He explains that the big idea behind SQD is that you need a powerful way to extract raw data at scale. Eventually you won’t need one indexer per user, but all you need is an indexer producing the data and countless endpoints from where it can be consumed.

How could that work? Not with the current Graph design. The data will need to be much closer to the user. One technology that might play a big role in that is SQLite. Leveraging it, the data could be directly on the client, enabling private data while also making it easy to replicate.

Replication could be combined with AVS to maintain Web3 values and resilience. Imagine Nodes from networks like Eigenlayer spinning up replications on demand. Overall, this allows the separation of storage and computing, which has been part of the SQD thesis from the start.

Taking on Web2

Web2 has very successfully separated compute and storage. And still, Dmitry foresees a future in which using SQD for data access could have a significantly lower cost than using existing Web2 solutions. That’s because SQD taps into a huge retail hardware market where the power of the network scales with nodes.

Why we’re still alive

It’s unclear what the person asking was really on about, but in short, SQD is still alive because we continue building on our mission regardless of web3 vibes or the token price.

On what happens if $SQD becomes so valuable, being a worker node becomes a huge investment

If the price of the SQD token increases so sharply that no one can afford to run a node anymore, that would be negative for the network. Still, it’s important to balance security, rewards, and requirements.

It’s worth pointing out that current rewards flow from a fixed reward pool, and the model will be updated in two years. By then, 20% APR might be too much for the resources required. Further, Dmitry points out that it’s quite common that running one node in a blockchain is expensive, see Solana or Ethereum. However, we then saw an increase in liquid staking protocols making it more accessible. In theory, it’s pretty simple to create that for SQD.

To further optimize, requirements could be more flexible in terms of the bonds workers have to provide. The more they bond, the more they will potentially earn, but they also have to commit more hardware.

For more in-depth on current rewards calculation, go here.

Any big partners?

Yes, the type that requires signing an NDA from us. Still, SQD is going heavily into the enterprise market and has already built partnerships with Google Cloud and Deutsche Telekom. An interesting observation Dmitry mentioned was that the flexibility makes SQD quite attractive as it allows these enterprises to experiment and then deploy in their own enterprise cloud, all without vendor lock-in.

dApps only possible once the game plan has been executed?

Dmitry highlights two use cases in particular: privacy enabled by indexing on the client, which unlocks new use cases, and client-side indexing, which could, for example, provide a responsive mobile app that shows an AI-customized Farcaster feed to the user.

Most excited about short-term?

Gateway 2.0 is part of our bigger mission to replace how dApps access data from the chain. In short, there will be no more RPC for data access.

For more questions, please ask away on the Discord Server.

AMA Recap: Discussing the Future of SQD was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

For some time, “data is the new oil” was a slogan crypto data companies used to appeal to VCs and convince them that there were enormous profits to be made. This throws up a few questions:

• Why would you want to associate with leaks in the ocean killing birds?
• How are VCs making money from you empowering users to own their data and not allowing Big Tech to exploit it for advertising revenue?
• Is it, though?

We cannot and won’t deal with the first two questions to set expectations straight. It’s something to ask the people who made these claims. For SQD, the more critical question is data’s role in a web increasingly navigated through the lens of A.I. agents.

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin. […] Samsa was a traveling salesman — but the only places he traveled were the roads of 0s and 1s. “Whatever”, he thought and tapped the screen of his phone with one of his legs, opening the interface for Alexis. “Hey Alexis, please pull up my most recent sales conversation and all relevant research notes I made,” he instructed the artificial assistant. With the other arm (or was it a leg?), he lifted the glass of water from his nightstand and took a sip. Gregor Samsa took his new state in with a nonchalance only found in modern-day nihilists. Shifting through his notes while scrabbling with one leg to fetch a sock off the floor only to realize he wouldn’t need it, it crossed his mind that maybe having so many legs (or arms) would give him an advantage in navigating the world of 0s and 1s. After all, he could now type on three devices at a time…

It’s pure speculation, but had Gregor lived a few centuries later with RAG and LLMs fully established; he might never have gone through the painful process of hiding the metamorphosis. He could have just stayed home and still accomplished his work.

But how far are we from this scenario? In a recent talk, our CEO, Dima Zhelezov, shared his thoughts on AI, RAG, and the role SQD will play in it. Listen to it here or read on for an overview.

AI & RAG are here to stay.

Let’s face it: regardless of whether you fall in the d/acc or e/acc camp, the underlying belief is that technology is here to stay and is not going anywhere. The same applies to Large Language Models, which quickly gained adoption by anyone from LinkedIn influencers looking to out-cringe each other to housewives (and husbands) just trying to figure out what to cook with their random assortment of leftovers.

AI agents are convenient and alleviate the mental load of their users. In the context of crypto, such agents take on- or off-chain actions on behalf of their users.

The way this works is that they consume data from a web3 data source, further data provided by the user, and then execute actions without the user needing to do anything. An easy example would be maintaining a certain portfolio of DeFi assets. A user can specify the allocation to Top 10 coins, and the agent will ensure that the ratios are maintained throughout market cycles selling and buying as necessary.

One way to think of these types of agents is as “trading bots on steroids.”

Further use cases include governance, other DAO activities, and automated trading. Adding RAG takes this a step further.

What’s RAG?

RAG is short for Retrieval-Augmented-Generation and describes a way to fill the gaps of existing LLMs.

Have you ever wondered where Chat-GPT got its answers from or what it means when it answers in a non-sense paragraph? Obviously, unlike humans, GPT-4 relies on statistics to combine words without understanding their actual meaning. And it’s limited to the data it was trained on.

Not anymore.

RAG adds access to current information while allowing users to peek behind the curtain and understand where an answer came from. The benefits are obvious: increased transparency on the sources, enhanced factual accuracy, improved consistency, adaptability, and coherence.

Take Github Co-Pilot. It doesn’t just rely on the data it’s been trained on; it also accesses commits, issues, and responses from GitHub relevant to a developer’s project to provide more relevant responses. Similarly, you could imagine an organization super-charging its support system with a bot that has access to all historical and current open tickets.

For crypto, RAG presents a new way to shape user experiences by leveraging AI. Instead of requiring users to navigate the kafkaesque labyrinth of bridges and chains, they could simply say something like “Send $200 USDC to my friend Gregor.” It’d be accomplished, even if the funds in their wallet are on Arbitrum, and Gregor only accepts TRC-20 tokens. The system could even look up the best times to transfer funds on given chains to optimize for gas fees and so on.

The User interface could even be a voice interface where tasks run in the background, simplifying all Web3 interactions.

SQD CEO Dmitry foresees LLMs in crypto could be leveraged this way for on-demand risk assessments (taking in relevant on- and off-chain data), human-intent-centric UI, data visualizations, and analytics, among other use cases.

Does that mean we should all now train LLMs?

If you have deep pockets, maybe.

However, you might want to consider our CEO’s take.

LLMs will become a commodity; Data won’t

Data isn’t the new oil; it’s something different. He explains that, eventually, AI interactions will become cheap while data explodes. At the same time, chances are that data retrieval won’t drop as much in price, and on top of that, blockchain’s size continues to grow with every block.

Before you go ahead and build your business entirely on AI agents, you might want to consider the cost of retrieving the terabytes of data you’ll need.

Our solution to the onchain data retrieval dilemma is modular indexing. SQD supports bulk extraction and filtering to the instruction level for more than 150 blockchains, including leading L1s, L2s, and native VMs.

It’s fast and scalable at a significantly lower cost than the competition while providing the dev tools necessary to power any AI and non-AI blockchain data use case you can think of.

Whether you want to facilitate future Gregors’ Hikikomori lifestyle or simply build better ways to interact with Web3, get in touch if you need data to accomplish that.

Data is not the new oil was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

16 July 2024 — Zug, Switzerland

“Every new beginning comes from some other beginning’s end.” — Seneca

We began to build SQD in 2021–which could be considered a lifetime ago by crypto startup standards. Following mainnet launch, however, the energy at Subsquid Labs is entirely different than what might be expected. We feel that we are at an entirely new beginning. The protocol is live, but the path toward true decentralization and developer utility has only just begun. This article is intended to publicize just some of our internal thoughts and give shape to our vision for SQD Network over the next year. Decentralization is, by definition, a community effort, and we hope the reader feels moved to become a part, whether it be by contributing code or through other talents, skills, or abilities.

Background

The Hyper-Scalable and Modular Decentralized Data Platform–SQD Network is a decentralized data lake that unlocks granular, permissionless data access at a petabyte scale. The network in its current form is a cluster of worker nodes that stores and serves data to blockchain indexers (in particular ‘Squids,’ but also Subgraphs and other ETLs) in a peer-to-peer manner.

In broad strokes, we have pursued and continue to pursue a vision for the network that aims:

• To provide permissionless access to all sorts of data (both on-chain and off-chain).
• To create a token-gated query interface (‘query engine’) to extract only the data relevant to the developer or other data consumer from the network.
• To enable extremely inexpensive, trustless, and auditable queries with opt-in TEE, ZK, and self-certification.
• To achieve virtually infinite horizontal scalability for the indexing of historical on-chain and off-chain data.
• To build out a token-based curation mechanism for node operators and the datasets that are served to the network.

Much of this vision was outlined in the latest version of the SQD Network Whitepaper, published on August 28, 2023. We encourage everyone to familiarize themselves with that document, and in particular to learn the roles and responsibilities of each network participant, including Worker Node Operators (who make up the data lake cluster), Delegators (who curate the nodes), Data Providers (who submit datasets to the network), and Data Consumers (who run gateways to the network or access ones operated by third parties).

Why SQD Matters

At Subsquid Labs, we believe that the key to unlocking Web3’s next significant boost in adoption lies in enabling fast, cost-efficient, and decentralized access to the ever-growing universe of unstructured data. This is just like how Google accelerated Web2 adoption by accumulating the data available on the World Wide Web and making it easily queryable through a search engine, and how Snowflake and Databricks did the same for analysts and developers.

Of course, there are several fundamental differences between our pursuit and that of Google:

• Owing to how the data is used, the value-per-byte of data in Web3 is orders of magnitude higher than in previous iterations of the web.
• In Web3, the data needs to be consumed by many additional entities and technologies, including AI agents, smart contracts, dApp indexers, and analytics APIs.
• Unlike the Web2 giant that depends on the monetization of data, we aim to create a credibly-neutral access protocol with a lean and simple low-level interface to consume the data. This is key to programmatic adoption, as was the case with HTTP in the early days of the web.
• In Web3, the value of data increases, and the network effect is cemented when it is not fragmented.
• A 100x improvement on decentralized indexing (as pioneered by the Graph) is a perfect launchpad niche for us to bootstrap the adoption flywheel.

While it is hard to predict how the dApp landscape stands to change and develop over the next three to five years, it is clear that a credibly-neutral solution to providing and consuming Web3 data will become a self-enforceable MOAT for SQD. Indeed, we anticipate that SQD Network will be able to serve billions of queries daily, capturing significant market share, including for:

1. dApp indexers (our primary focus up till now)
2. Smart contract and oracle data access (primary for DeFi use cases)
3. Analytics
4. Next-gen use cases including AI agents and TEE coprocessors.

In SQD Network’s current form, operation costs per query are already very low ($<0.001). Taking advantage of this fact, we are now entering into a bootstrap phase that only token-gates access to the network without taking fees, enabling organic scaling alongside free data access. After this initial bootstrap phase, a small SQD tax for gateways will be introduced directly into the protocol. The SQD revenue model is and will be discussed elsewhere. Interested readers can check the manner of calculation as outlined in the SQD Network whitepaper.

Past achievements

We have previously told our founding story in some depth on our blog. In short, SQD has gradually gained industry adoption as an indexer since its launch in 2021. Taking a modular approach to data access, Squid-based indexing became the industry’s most flexible and performant data-retrieval solution in the years that followed. In 2023, Subsquid Labs ran the largest data infrastructure testnet ever, wherein over 60,000 decentralized indexers were deployed by at least 17,000 verified developers over the course of multiple testing phases. In January of 2024, SQD held the fastest CoinList sale ever (and the first major token sale of the year), selling out 5% of the token supply in just 19 minutes.

Decentralized Data Lake

On June 3rd, 2024, Subsquid Labs announced the launch of the SQD Network mainnet. At this initial launch, SQD Network is a decentralized data lake composed of data providers, worker node operators, delegators, and data consumers (gateway operators).

Data providers, acting as oracles for ‘big data,’ upload raw blockchain data, which is then compressed and distributed among worker nodes. During the initial bootstrapping, the unique data provider is Subsquid Labs, however this (including data validation) will be decentralized according to the plan presented in this article.

Worker node operators bond a security deposit (100,000 SQD), which can be slashed for byzantine behavior. When requested, the worker node queries its local data using DuckDB. Any query can be verified by submitting a signed response to an on-chain smart contract.

Delegators are generally regular SQD token holders who indicate trustworthy nodes in the network through delegations. Nodes that receive the most delegations have a higher chance of receiving work to do and, as a result, higher rewards. There is currently no minimum delegation requirement in the network and no locking period, meaning curation acts as a real-time, permissionless system of node curation. Delegator rewards are calculated based on the rewards paid out to the relevant worker nodes.

To query the network, data consumers have to operate a gateway or use an external service. Subsquid Labs’ ‘SQD Cloud’ product currently runs a gateway, and several infrastructure providers are in the pipeline to run these as well.Each gateway is bound to an on-chain address.

The number of requests a gateway can submit to the network is capped based on the amount of locked tokens. Consequently, the more tokens locked by the gateway operator, the more bandwidth it is allowed to consume.

One can think of this mechanism as if the locked SQD yields virtual “compute units” (CU) based on the period during which the tokens are locked. Much more information about this model will be made available soon, along with the release of Gateways 2.0, as outlined later in this article.

Since the launch of the mainnet just over one month ago, we have observed significant traction for several of the network roles. At the time of this article’s publication, 811 worker nodes are online, storing 594 TB of onchain data and earning an average APR of 22%. Approximately 71.2 million tokens have been delegated, earning delegators an average APR of 8.1%.

At this early stage, the vast majority of production SQD use cases have yet to be migrated to the decentralized network. Nevertheless, the decentralized data lake has already served 433 TB of data to data consumers.

SQD Indexing

Long before the launch of the decentralized network and the token, SQD built its reputation by providing the industry’s most performant and flexible indexing service. Squid SDK is a TypeScript-based toolkit for high-performance batch indexing, sourcing the data from SQD Network, and previously from the centralized iteration of the data lake. SQD Cloud is a hosted service for custom indexers and GraphQL APIs, that also sources data from the data lake.

Additional plugins include SQD Firehose, a lightweight adapter for running subgraphs against SQD Network, without accessing an archival RPC node, and the ApeWorx SQD plugin, which uses SQD Network as a fast data source for that framework. Moreover, Python developers can access data from the data lake using DipDup SDK (built externally by the Baking Bad team).

As an indexing service, SQD has set itself apart for its speed, reliability, and developer-flexibility. A typical SQD indexer, for example, can process historical blockchain data at tens of thousands of blocks per second–tens and hundreds of times faster than competing tools. SQD uptime is unprecedented for Web3 infrastructure, as can be seen on the official uptime monitor.

Squid SDK is modular, extensible, and compatible with dozens of popular tools and data targets, including Google BigQuery–this feature was recently announced as a part of Subsquid Labs’ official collaboration with Google Cloud.

In terms of traction, SQD has achieved approximately 20% market share, second only to the major incumbent. Using SQD, developers can already index over 150 chains, including any EVM, SVM (Solana), or Substrate (Polkadot) network, as well as Fuel, Tron, and Starknet.

At least 300 dApps depend on Squid SDK each and every day, and well over 5,000 indexers are currently in production. These numbers are conservative, and we at Subsquid Labs expect them to increase drastically as decentralized gateways go live. Gateways will enable us to see how SQD Network data is being used onchain.

A New Game Plan: Why and How

Anybody who reads our earliest whitepaper and our most recent will see that very little has changed in our overarching mission: to re-architect the way decentralized data is handled in order to unlock the peer-to-peer internet for everybody. However, in the over three years since our founding, there have been significant updates in the space, and new technologies made available. For these reasons, we consider it a good time to refactor our tactical game plan.

Keep in mind that SQD Network is a decentralizing protocol with the ambition to become fully decentralized. This means that aspects of the plan may change according to the feedback of the community, and indeed we very much encourage any interested person to provide their input on any part of the vision, just as we welcome open-source contributions to the technology.

Technological Progress and a Changing Landscape

When we first set out to build SQD, Web3 was very much a different place–data at that time was provided to dApp front-ends by simple indexers, and to smart contracts by somewhat primitive oracles. Solutions to expand access to blockspace were quite different too, as higher-layer rollups and appchains were still largely theoretical.

Back then, there was no expectation nor need to provide truly real-time access to data at any kind of significant scale. It wasn’t so important to be able to query across multiple networks, as these chains simply didn’t exist or have any usage. Simple smart contracts on EVM were completely dominant.

Since then, the landscape has completely changed. More data than ever is being posted on-chain. Contributors to this trend include restaking layers like Eigenlayer and Symbiotic, high-throughput L1s like Solana, appchain ecosystem like Cosmos and Polkadot, and the ever-growing availability of data availability solutions as provided by decentralized projects like Celestia and Avail, and by centralized RaaS providers like Caldera and Gelato.

Of course, the more networks are live and the more data t is onchain, the greater the challenges in terms of accessing it. The large incumbents have mostly failed, allowing SQD to gain market share as a result. Developers of applications across use cases have become more ambitious, with ambition going beyond the infrastructure presently available. For today’s DeFi and social use cases, scalability in terms of historical data access as well as real-time data streams has never been more relevant.

Indeed, the old indexer/oracle dichotomy is becoming less and less pertinent. Now, developers want to stream on-chain data to smart contracts, as is certainly necessary for many use cases including AI agents. Particularly in onchain social media use cases, developers need to aggregate both on- and off-chain datasets to be displayed by front-ends.

Data validity is prescient, and fortunately ZK, TEE, and co-processor technology is coming to its own, thanks to providers like Polyhedra, Brevis, Super Protocol, and Phala, to trustlessly provide these proofs.

The new SQD developmental plan, as presented below, takes all these innovations and market demands into account.

A Note on Timeframes and Dates

“The only thing that is constant is change.” — Heraclitus

The time estimations and dates shared in this document should not be construed as a contractual obligation or any other kind of ‘promise’ from Subsquid Labs. While it is true that today, and likely for some significant amount of time going into the future, the direction of SQD Network is and will be largely driven by Subsquid Labs, the tenants of decentralization may lead to developmental delays or ‘changes in schedule.’

This could present itself in many ways, via governance or through community activism. New opportunities may arise in the ever-changing world of Web3, and network participants may decide to redirect protocol development in that direction. We embrace this spirit of innovation wholeheartedly and look forward to seeing how the protocol develops in unexpected ways as time passes and adoption grows.

Another consequence of decentralization is that development is unlikely to always be linear: just because an SQD product feature becomes implemented and accessible to network users does not mean that it is yet fully decentralized.

Subsquid Labs follows a product-first protocol development strategy, meaning we always develop features for the network first as testable MVPs before making them production-ready and decentralizing them. In other words, we make sure that real users need and want a feature and bring them onboard to help improve the feature before we move on to decentralizing it.

An example of this is the ‘ArrowSquid’ release of SQD Network blockchain datasets, which was released back in early 2023 and which architecturally are identical to data sets (also called ‘archives’) in today’s mainnet. First, we built the release; then we tested it with a few users; then we onboarded the rest of the users, and only then did we decentralize them fully with two testnets and our recent mainnet launch.

To sum up, while features in this network may go live on a specific date, it is still likely to take some time to coordinate their decentralization across the very rapidly growing network.

SQD Network’s 2024 Development Game Plan

Throughout the rest of 2024 and until 2026, the primary focus of SQD’s technical development will be on data streams (hot blocks/real-time data), trustless data validations, implementations of ZK proofs and TEE co-processors, SQL queries, permissionless dataset provision, and full indexing decentralization via light squid indexers and AVS deployments.

By the end of this plan, data consumers in SQD Network will be able to retrieve data from the network using decentralized indexers, TEEs, and SQL. It will be possible (as it is today) to send this data on to dApp front- and back-ends, and it will become possible to stream it to smart contracts. The provision of data will be decentralized, and users will be able to submit datasets to the network to make them available for permissionless queries by anybody, anywhere.

Data Streams (Gateways 2.0)

The status quo — To query SQD Network, data consumers have to operate a gateway or use one operated by an external third-party. Presently, SQD gateways are architected in such a way that they can only serve finalized blockchain data at a delay of a few dozen to a few hundred blocks, depending on the particular blockchain’s finality time and how prone that blockchain is to re-orgs. As a result, to get real-time data, data consumers today must depend on an RPC to read/access the latest blocks, including non-finalized (‘hot’) ones.

The goal — With the launch of Gateways 2.0, we will remove data consumers’ dependence on the RPC endpoint, allowing for much more predictable real-time data streams. Thanks to certain architectural changes, Gateways 2.0 will also make squid indexers run against the network with far reduced latency against the network.

Value unlocked — In addition to the increased performance and more trustless access to real-time data for data consumers, the native implementation of data streams to SQD network represents the first step towards unlocking a wide range of additional use cases, including the ability to send data directly to smart contracts (Oracle use case), as will be outlined in more detail below.

The implementation — Gateways will be rearchitected to include hot storage and to handle computations that will no longer need to be handled by the squid (indexer) or client side.

Time estimate — A minimum viable implementation of Gateways 2.0 is on track to begin testing in mid-August. The first gateways will serve Solana data, with EVM, Substrate, and other VMs to be released in quick succession after adequate testing.

Query Verification & Validation

The status quo — The standard for data validity in Web3 today is far too low. Developers and end-users are far too willing to trust data providers. People trust oracles, typically centralized RPC providers, indexers like the Graph, and their centralized alternatives.We believe this is wrong, and are in good company with projects like Space and Time, Kyve, and a range of ZK-protocols that aim to develop solutions to this problem.

The goal — Trustlessness for data access through ZK validity proofs and greater trust at a lower cost for many downstream use cases through TEE implementations.

Value unlocked — SQD’s implementation of data validation technologies will unlock trustlessness for data access from the network, enabling greater decentralization power for use cases that are down the pipeline, like indexers, oracles, and AI agents.

The Implementation — Historically, we at Subsquid Labs have tried our best to ensure the data that is included in the data lake parquet files is sound. However, there is no way to control how worker nodes respond to data consumer requests. This opens up significant problems in terms of data validity, which is a blocker to many trust-minimalized indexing use cases, and certainly not to be allowed when sending data onto a smart contract. Already, we have begun implementing and testing different methods of data validation, taking into account each method’s unique trade-offs.

Ultimately, we intend to offer developers multiple options, which they may choose based on cost considerations and the degree to which trustlessness is required for their use case.

ZK proofs, for instance, are expensive and slow but fully trustless. This may be preferable for sending data to smart contracts. TEEs, on the other hand, are far cheaper but also less secure, and may be chosen for use cases that don’t require such a high level of trustlessness. We expect that some developers may even choose to validate data on centralized servers, which are even lower cost than TEEs but far less secure. This may be adequate for some front-end use cases.

Time estimate — The first ZK and TEE implementations ready for beta testing will be made available in Q4 2024.

SQL Queries

Status quo — Currently, SQD Network filters and stores raw data on a minimal level and validates it. On top of this, developers build squid indexers/ETLs in order to develop richer datasets for quick queries by the apps. For ad-hoc queries, this is not the best solution.

The goal — Much of the data people are interested in can be reduced to a SQL query when available from a specifically designed dataset. These datasets and this query functionality will become available directly from SQD network. In other words, in a permissionless manner, users will be able to ask questions (and get answers relatively quickly) to questions (written in SQL) like:- “What was the average trading volume of SQL/ETH across DEXs on 10/09/2025?”- “What is the current TVL of PancakeSwap exchange on Arbitrum and BNB Chain?” — “Tell me my token holdings across all EVM chains”

Value unlocked — The primary value-add from the introduction of SQL to SQD Network is decentralization and the second-order effects of decentralization (reduced infra cost), enabling users to receive rapid responses to questions relating to Web3 in a P2P manner.

On top of this, one could build a front-end for a permissionless Dune Analytics-like service, a natural-language search engine, and many other demanded use cases.Furthermore, we expect developers to use the SQL query feature for the development of APIs on top, and in general for more advanced aggregations on the client side.

The implementation — We have no intention of making the SQD network to be computationally heavy. Instead, the network must provide developers with primitives that can allow for more sophisticated computations on a higher layer of the stack. In line with this philosophy, the SQL functionality will likely be built by breaking queries down into smaller pieces, which can each be executed by workers. Another alternative is that a bulky part of these aggregations will be carried out by the Gateways.

Likely, it’ll be a combination of this. Time estimate — We are still at the research phase in the development of this feature, and we expect its release to be non-linear. The first minimum-viable implementations may become available in Q2 2025.

Decentralized Dataset Provision

Status quo — The network currently depends on Cloudflare R2 as worker nodes’ main source of data. Further, there is no way to see or verify on-chain the status of the work done by worker nodes. The goal — Replace R2 as the data source for worker nodes with a decentralized mechanism and ensure the on-chain accessibility of metadata about the inner workings of SQD Network.

Value unlocked — As a result of this implementation, data will always be able to enter the network, no matter what, in a permissionless manner. This unlocks better distribution for the data itself, and the ability for the community to validate information about the data contained in the data lake (i.e. how many blocks have been indexing, when this data was provided), further improving the trustlessness of SQD.

The implementation — We will develop a series of smart contracts and a more sophisticated query engine on the worker side. Data will be stored using a decentralized storage provider (likely IPFS). The way it works will be as follows: The data provider puts data into the decentralized storage, creating a transaction on-chain which describes the data. Whenever a new piece of data is submitted, an update will be transmitted to the smart contract so that the workers know that the data is there.

Time estimate — Smart contracts will be deployed, and the R2 dependency should be eliminated by Q4 2024. The first data providers may begin to be whitelisted in early 2025. Eventually, whitelisting of data providers will likely be transferred to the community, and will be carried out through token governance.

Light Squid Indexers

Status quo — By default today, squid indexers send data to postgres databases, which are heavy and hard to replicate. As a result, builders of squids are forced to use centralized production environments as hosting, including SQD Cloud. While the SQD mainnet launch made great progress towards decentralizing the ingestion Web3 data, and at making this data available in an efficient in P2P manner, it has not so far provided a way to transform and present this data as APIs in a decentralized way.

The goal — With the introduction of Light ‘Squid’ Indexers, Subsquid Labs will introduce a decentralized way to serve APIs with maximum performance improvements over legacy protocols (i.e. the Graph, SubQuery). The light indexers will serve as fully decentralized back-ends for decentralized applications that are both easy-to-run, provide validated data in real-time, and scale with increasing usage.

Value unlocked — Light squid indexers will enable very low-latency data for end users. Replicated on an AVS, they will offer unlimited scaling and will be rapidly forkable and deployable in different regions, offering additional performance improvements. For certain use cases (for example when privacy for aggregated data is of the utmost importance), light squid indexers could be deployed on the device or in the browser of the end user.

The implementation — Inspired by recent advances in database replication technology (such as the recent implementations of Turso), we will begin by making squids natively compatible with SQLite, in lieu of the very bulky Postgres. For initial testing, these light squids will be deployed to the cloud service, enabling current users to be able to fork their existing indexers in order to make them available in multiple regions. This is a product that has been demanded for some time.

Once light indexers on their own are production ready and tested, we will begin to deploy them on decentralized rails. Since SQLite is light, it can easily be run on an AVS. We are currently deciding on the best platform for SQD and weighing the pros of and cons are all popular staking layers, including Eigenlayer and Karak.

Time estimate — The first viable light indexers will begin to be tested for production usage on the Cloud in September 2024. Multiple-region forkability should go live soon after. We intend to migrate light squids to an AVS towards the end of Q4 or the start of Q1 2025.

Community Call to Action: Provide Feedback

Upon reading this article, we highly encourage anybody knowledgeable in the areas we are building to provide their suggestions, comments, and criticism. The best possible place to do this is in the SQD Discord. Feel free to @ an Admin. We are looking forward to your feedback.

Data Streams & Light Squid Indexers: SQD’s New Game Plan was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Regardless of whether you believe that alternative Layer-1s will become the backbone of Web3 or that L2s will facilitate widespread adoption, the data they generate daily is already the Terabytes.

The ledger sizes of legacy blockchains, such as Bitcoin or Ethereum, have surpassed 5000 and 1000 GB, respectively, and this is just onchain data. Add to that other native chains, L2s, and even L3s, and the amount of information is staggering.

This presents a unique challenge for businesses operating in the space. For one, they need to access data to build their projects, but they might often be in the dark about success metrics when it’s hard to retrieve and analyze data. In a modular multichain blockchain ecosystem, there are few tools (if any) with the analytical capabilities necessary to make real-time decisions.

That seems contradictory in a space where real-time monitoring would be all the more valuable for risk management, threat prevention, and optimizing tech for demand.

SQD has built a decentralized stack that facilitates querying onchain data at scale without relying on RPC access to blockchain nodes. Instead, SQD ingests the data in its decentralized data lake in the form of parquet files, which offers builders a playground where they can define their own schema and queries for retrieval. The infrastructure already supports 150+ EVM chains, L2s, and native Virtual Machines such as Solana.

With the data access problem solved, the only thing missing are operational tools to derive critical insights from the data.

Kelp is filling that gap with its visual development platform. It’s designed to simplify the development of data processing and data visualization tools using an intuitive drag-and-drop interface. With a comprehensive library of data connectors, operators, and UI widgets, users can build powerful applications faster than with traditional coding. It supports secure and reliable connections to nearly any REST and GraphQL APIs, databases, and SSE streams, providing seamless integration with existing infrastructures and data sources.

With Kelp and SQD, you can transform onchain data into actionable insights and operational tools.

Analyzing the Price Impact on Uniswap

To illustrate, we built a tool that helps analyze the change in the price of a token resulting from a trade on the Uniswap Dex.

• Uniswap uses liquidity pools rather than traditional order books. Prices are set algorithmically based on the ratio of tokens in the pool.
• Understanding and managing price impact is crucial for traders on Uniswap to optimize their trades and minimize costs associated with large price swings.
• Each trade affects the balance of tokens in the pool. Large trades relative to the pool size cause significant shifts in the token ratio, leading to greater price impact.
• High price impact leads to higher slippage (the difference between the expected price of a trade and the actual executed price), making trades more expensive for the trader.
• Larger pools tend to have a lower price impact for a given trade size because the token ratio changes less dramatically. Conversely, smaller pools are more susceptible to large price swings from individual trades.

Example

The example below shows a low price impact. In this transaction, the price impact rate results in a slight loss of 0.053%.

The following example shows a high price impact. In this scenario, Uniswap will initiate a price impact warning because you will likely experience a significant loss.

To mitigate price impact, traders have several options. Splitting a large trade into smaller trades can help reduce price impact, though this may not always be feasible due to gas fees. Trading in pools with higher liquidity is another option, as it reduces price impact by ensuring that the ratio change for a given trade size is smaller. Additionally, traders can set slippage tolerance to control the maximum acceptable price impact. If the price moves beyond this threshold during execution, the trade will not execute.

Without special tools, it may be hard to reason about it. Visualizing the chart of price change in relation to the trade value offers several benefits for traders and liquidity providers on Uniswap. Traders can better plan their trades by understanding how different trade sizes will impact the price. Liquidity providers can assess how their liquidity impacts the pool and adjust their contributions to optimize returns. Both traders and liquidity providers can gain insights into the liquidity depth of different pools, understanding how robust or fragile a pool is to large trades.

Explore a visualization of the Price Impact chart implemented on Kelp using the data feed from SQD. You can also fork this app and customize it for your own use case. For the math behind the calculations, use the following resources:

• Uniswap v3 Core
• SwapMath method

Conclusion

Visualizing on-chain data transforms complex blockchain information into actionable insights, offering significant value to both traders and developers.

For traders, it enhances decision-making, risk management, and strategy development by providing precise and accessible data on various metrics, including price impacts, market trends, liquidity levels, etc.

For developers, it improves user experience, facilitates efficient monitoring, and fosters transparency and trust through intuitive interfaces and real-time analytics.

Beyond these two groups, practical insights and flexible dashboards are also beneficial for researchers and crypto businesses.

By making on-chain data more understandable and usable, visualization tools empower users to optimize their interactions with decentralized finance systems and contribute to a more robust and transparent blockchain ecosystem overall.

Using the SQD data lake as a source of comprehensive blockchain data, combined with the flexible and interactive capabilities of the Kelp platform, users can create powerful tools tailored to their specific needs. The fusion of real-time data access and interactive visualization can transform the way we interact with and understand blockchain ecosystems.

“At Kelp, we’re all about giving developers the tools to turn their ideas into reality. Visualizing and interacting with onchain data opens up so many possibilities. It’s exciting to see what we can build together when we have the right data and tools at our disposal.” — Vitaly Malyshev, CEO and Founder of Kelp

“This collaboration is a unique opportunity to showcase the power of SQD in surfacing blockchain data and how it can contribute to better decision-making. Kelp enables something that other analytics tools, such as Dune Analytics, don’t: the ability to interact and figure out, for example, what the impact of a specific trade you have in mind will be. That is just one of countless potential use cases. ” — Dmitry Zhelezov, CEO and Co-Founder SQD

Stay tuned for future posts as our teams explore the technical details of building further tools to visualize and make sense of blockchain data.

Developers interested in leveraging the Kelp x SQD integration, please contact us on Discord.

How Kelp and SQD Enable Interactive Onchain Analytics was originally published in SQD Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.