There's a particular kind of sigh you let out when a platform you once loved starts making the same moves you've watched a dozen other sites make right before they got worse. Well, Reddit just made one.

On May 28th, a Reddit admin posted in the moderator news community, and tucked inside the cheerfully-titled "protecting communities from scrapers and platform abuse" (archived) was a line that made a lot of us wince. Reddit wants to know how people use its RSS feeds, because they're "another common surface for scraping."

If you've been around long enough to remember when all these large sites started walking the feeds we rely on to the woodshed, you know how this tends to go.

What Reddit actually said about its feeds

To be clear, Reddit hasn't announced it's dropping RSS feeds, at least not in this post. They're just requesting feedback from the mods on how they use them in their work so that, as they "develop secure solutions," they can account for the tools you depend on.

Not sure if anyone really knows what that gibberish means, but hey—it sounds nice. It's also the sound of someone telling you they aren't redecorating while measuring your living room for new furniture.

First the open developer access got a huge price hike that killed a wave of third-party apps we loved, which none of us were happy about. Then, some data endpoints got cut as announced in this very post, and now their feeds have entered the chat too. It's not hard to connect the dots.

Reddit isn't entirely wrong

The Reddit post implies that the site is being abused and their RSS feeds are being used to do it. Feeds are kind of our thing around here and we're no strangers to how readers and feed bots tend to misbehave, so including some of them in the abusive category isn't unwarranted, or all that surprising.

In order for feed readers to work properly, they have to keep checking a site's feeds for new content, and a lot of them tend to do these checks far more often than they need to. Pile millions of those together while hundreds of millions of users are simultaneously using the site for non-feed content, and even a site like Reddit, with a beefy infrastructure and a ton of servers, could buckle under the load.

The Slack and Discord integrations Reddit mods rely on are automated fetchers, and can be adding to the problem too. When built well, no abuse. But built badly, and they behave exactly like the data-hungry bots Reddit says it's fighting, which is why so many sites block feed readers now — right along with the bad bots. So when Reddit says its feeds are a scraping surface contributing to the abuse, they're not just making it up.

Cutting access to feeds won't fix the abuse problem

So Reddit doesn't want a bot or automated tool using their RSS feeds to abuse the site — fine. But if they're thinking about cutting off access to the feeds from everyone else using them legitimately, not only would that be a bad move, it won't fix the problem. RSS feeds aren't the only way the site can be abused. And if bots and tools are checking the site (or the feeds) for content too often, that's what rate limits and caching are for (which people already suggested in that thread, btw). You fix a leaky faucet by fixing the faucet, not by shutting off the water to the whole house.

Although if a platform is going to kill its feeds, it usually doesn't happen overnight. Their feeds just magically stop working as a side effect of some other change they announce, almost exactly like what Reddit is doing in this post:

• Twitter didn't announce the end of feeds. It retired the old version of its API in 2013, and support for feeds went with it.
• Facebook never said it was dropping feeds either. The endpoint that powered its page feeds was buried in a routine API update and stopped returning data in mid-2015.
• Google framed killing Reader as routine housekeeping, and a chunk of the open web's plumbing went with it.

Where this leaves us

Fortunately, Reddit has kept feeds working long after other platforms let theirs rot, so it's nice to see they're at least asking for feedback about it. But you can read a post like this two ways. Reddit genuinely wants input. Or the whole post is just corporate double-speak for "we're about to get rid of our RSS feeds, we just wanna know how loud the blowback will be."

So good for all the Reddit users who are rightfully defending keeping RSS feeds available in that post, especially while it's still a question and not an announcement. The best argument against Reddit flipping the switch on their feeds is a pile of real people describing how much they depend on them.

Honestly, Reddit. You may have been a little tone-deaf lately to your most dedicated users and lack the ability to read the room on occasion. But when it comes to feeds, you've always been one of the good ones. So please, not you too.

In case you haven't caught on yet, some of us will just never be interested in being manipulated by those brain-rotting, never-ending homepage feeds you love shoving in our faces the moment we log in.

We would rather use the feeds you offer for each of your channels. You know, the ones you're hiding? The feeds we can subscribe to in our own feed reader to follow our favorite creators without having to be on your platform at all?

Well, your relationship with these feeds has gone from neglectful to borderline hostile, and we're tired of pretending otherwise.

Your feeds keep disappearing

Let's start with the fact that when using your feeds in a feed reader, they're unreliable. Users have been reporting for a while now that their feeds either go silent without warning or vanish altogether. No announcement, no error message, no explanation. Just... gone.

And sometimes they're out of commission for so long that people genuinely think you've just said "screw it" and axed them.

Is it a bug? Probably. Is a fix being prioritized? That's a harder question to answer. But when a platform your size lets something like this slide, it stops feeling like an oversight and starts feeling like a choice.

Hiding your feeds in plain sight

Another thing that annoys us: you make no effort to surface the link to these feeds. When visiting a YouTube channel, there's no link to follow it in a feed reader, no "add feed" button, nothing.

Instead, we're stuck trying to glue together a channel's feed from a bunch of jumbled letters and symbols like channel/UC4a-GbYw7vOacCHmFo40b9g, a hot, garbled mess that's unmemorable and clearly not designed for human beings.

It's sad to see when you compare that to the early web, when feeds were a first-class citizen and sites like yours wore their feed links at the top of their pages like a badge.

We just don't get it. You have the infrastructure and every opportunity to let people subscribe to your feeds in a feed reader with a single click. But you keep choosing not to. It's like you just don't want us to use them.

Nobody asked for shorts in their feed

Apparently somewhere down the line, you've begun a multi-year mission to become another TikTok, and that's fine, platforms evolve. But when that mission starts bleeding into the feeds of users who don't want it, it becomes a big problem.

Shorts are showing up in feeds whether we want them or not, and we've tried to express how much we don't want it as politely as possible (How many ways can we say "Not interested"?), but there they are.

When we subscribe to feeds in our feed readers, it's intentional. So if we add a feed to specifically follow the channel's full-length, higher quality video content, that's what we want to see. Shorts are the opposite of that. They're impulse content, designed for infinite scroll, not for a feed reader. And mixing the two isn't just annoying, it's a fundamental misunderstanding of the purpose of what feeds are for.

So feel free to chase TikTok. But it should be okay if some of us don't wanna be dragged along for the ride.

There's a much bigger problem at play

Sadly, you're not the only platform letting their feeds rot. It's part of a broader pattern across the web where large platforms like yours have subtly, over time, made their feeds less visible and harder to use.

Why? Because offering feeds that can be used in feed readers lets us follow our favorite content without having to log in and constantly check your platforms. It gives us control. It removes your algorithms and the ability to manipulate us. It doesn't let you decide what we see and when, and that's bad for those fancy engagement metrics and ad revenue you all love so much.

Unfortunately, you're not unique in this. But you are one of the few platforms that still offers feeds that can be used in feed readers. So even if you're trying to make us forget they exist, we can't be too hard on you. You haven't removed them... yet.

Our feeds will be here even if yours aren't

Here's the thing: the technology behind the feeds we use in our feed readers has outlasted every platform that ever tried to make it irrelevant.

It survived when Google killed its feed reader while trying to take the entire technology down with it. It survived the rise of social media timelines. It even survived the podcast industry trying to wall off its own open ecosystem (looking at you, Spotify).

So your indifference is just the latest chapter in a long, boring story we've all read before. But if you're going to offer feeds, make sure they actually work. And if not, guess we'll have to keep trying to do it for you.

After subscribing to a website in a feed reader, the reader needs to keep constantly checking the website for new content to show you.

But an increasing number of websites are either severely limiting feed reader apps or outright blocking them because they're checking websites for new content way too frequently and often unnecessarily.

Too many requests around the same time from a feed reader can cause a website to use up all of its resources and bandwidth, making it extremely slow for its other visitors. In severe cases, the excessive requests will knock a website completely offline, leaving it unresponsive and inaccessible to its visitors. This is especially the case for smaller websites that aren't run by big companies with big budgets to pay for large servers and unlimited resources.

Feed readers are being blocked with bad bots because that's how they behave

Back before the internet became a battlefield of who can get the most clicks or gobble up the most user data, websites could just exist without being flooded with requests for their content. Now, malicious bots have taken over the internet, hammering websites with tons of requests as they try to suck up every bit of data they can, wherever they can, at any time they want, and as often as they like.  

To prevent these nasty bots from taking down their website, website owners are left with no choice but to block them. Unfortunately, the app you're using for your feeds may be behaving just like these malicious data-hungry bots, so it's probably getting blocked too, leaving you with feeds that don't work or that never update.

Excessive feed requests are wasting a website's resources

When a reader checks a website for new content for a feed, the site already has to use its limited resources to process the request and respond to the reader—even if there is no new content. So when many of those feed requests are unnecessary, the website is wasting resources on the reader that could be used to serve content to its other users.

We often see feed readers checking a website every second even though the website only publishes content every few days. Even worse, we've seen a reader make as many as 40 requests every second for content that only changes once a week!

Feed readers don't have to make so many requests

You may not see it in your feed reader, but when it's checking for feed content too often, websites will typically respond by notifying it to reduce the amount of requests. Some websites respond with the specific amount of time to wait before checking the site again. Other sites will just become much slower to respond, indicating to the feed reader that the website is temporarily overwhelmed with requests and to back off for a while.

In these cases, the feed reader should adjust the frequency of its requests based on this information, which would make it a lot less likely the website will block it.

Helping to get blocked feeds working in your feed reader

When feeds are blocked in a feed reader, often its developer isn't aware there's even an issue. To help with this, when an issue with the reader is identified either by our organization or reported by a user, we contact the developer about it and work with them to get it resolved. The issue then gets added to the Issues page.

After navigating to an individual issue, you'll see a running log of our progress as we work with the developer towards a resolution. There's also a feed available that you can subscribe to to follow along. Each log entry includes our communication when reaching out, their responses, and any assistance we may provide them.

If your feed reader is successfully connecting with the service, another way to see its issues is in the Apps section. After navigating to a feed reader's page, you can view all of its recent errors that can cause it to be blocked by websites.

If a developer needs more information on how to resolve issues with their feed reader, we provide a Developer's Guide that offers step-by-step recommendations on how to request feed content without overwhelming websites with too many requests.

Unblocking your feeds is our mission

Open RSS exists to help ditch the cesspool of algorithmic feeds controlled by the companies running the apps and websites we visit. For the sake of humanity and our mental health, their hyper, anxiety-inducing feeds can't continue to trigger, entice, and manipulate us. These feeds must be under our control, and feed readers are critical to making that possible. So we'll continue working through all issues feed readers have until there's a world where we can always rely on the feeds we add to them.

The feature in this article is available exclusively to those donating $10 or more monthly.

If you're well acquainted with Open RSS feeds, you may have noticed that, when a website publishes new content, there could be a delay before the feed shows the new content. Now, you can see when this delay starts and expires, so you'll know exactly how long it'll take for new content to appear in any feed.

Why the delay?

Each Open RSS feed has a website it gets content from that needs to be checked repeatedly to keep the feed up-to-date. But these checks don't happen every second of the day because that could overwhelm the website with too many requests. Instead, the feed will show a stored version of the content and the feed's website won't be checked again for new content until after a reasonable amount of time has passed.

How it works

Just visit any Open RSS feed in your web browser to see the last time new content was requested and how long it'll take before it happens again.

The left-hand side shows the last time the feed's website was checked for new content to add to the feed. On the right, you'll find a timer that counts down to the second of when new content from the feed's website will be checked again. When the timer ends, you're given an option to refresh the page to show any new feed content available.

How to use

To use the feature, you must have a donation subscription of $10 or more every month. Then follow these steps:

1. Login using the information you received when you began your donation subscription.
2. Navigate to the preview page of any Open RSS feed in a web browser.
3. You'll then see the feature at the top of the RSS feed preview.

This feature is a part of a larger effort by the organization to provide more transparency to our supporters and give more insight into the behavior of Open RSS feeds. You can read more about how Open RSS feeds work here.

Many users prefer to use an RSS feed reader to stay up to date with the content on the websites they visit. But if you've enabled Cloudflare on your website, you're likely blocking these RSS users from accessing your website content without realizing it.

The Cloudflare features that block RSS readers

In Cloudflare's dashboard, you'll find tools designed to block bot traffic to your website. Particularly, the Bot Fight Mode and block all "AI scrapers and crawlers" options below. When enabled, these features end up blocking users who access your website through RSS readers, even though RSS readers are legitimate and aren't malicious bots.

How Cloudflare blocks RSS readers from your website

When enabling the tools, Cloudflare will evaluate each visit to your website and determine whether the visit is from an AI scraper or "bot" based on a score , which ironically Cloudflare uses AI to generate.

Then, when a user's RSS reader attempts to read your website, Cloudflare presents it with a number of challenges that the reader would never be able to fulfill.

Here's an example of the Human Verification challenge that an RSS reader would be shown when it tries to visit your website. The challenge requires a human to solve and, because an RSS reader is not a human, it can never complete them.

In other cases, Cloudflare will simply block the RSS reader from accessing your website without a reason.

The only way to resolve when Cloudflare blocks an RSS reader from accessing your website is by contacting you directly and asking you to make a custom rule to unblock it. But Cloudflare shouldn't expect people to contact every owner of every Cloudflare website that blocks their RSS reader. And you shouldn't have to waste time logging into Cloudflare to add an exception every time they block an RSS reader, either.

Unblock RSS readers while still using Cloudflare

Even though Cloudflare blocks RSS readers from your website, you can whitelist RSS readers as a workaround. This would at least unblock RSS readers without having to turn off any security features that you may have already been enabled until Cloudflare better addresses the issue.

First, find the user agent of any blocked RSS reader in Cloudflare's analytics dashboard. The User-Agent of most good RSS readers usually include the name of the reader, it's URL, or a word like "RSS" or "feed" that makes it obvious that it's an RSS reader.

Once you've identified an RSS reader's user agent, you can create a custom rule that explicitly whitelists and allows all traffic by the reader's IP address or by it's user agent string. Note that user agents can be disguised, so it's often better to whitelist the reader's IP address instead of the user agent. If you'd like to whitelist Open RSS, please contact us for the required information.

Cloudflare needs a better way to allow RSS readers

Cloudflare offers a bot verification program to which RSS readers owners can manually apply to avoid being blocked by websites, but this program isn't guaranteed to work and it suffers from quite a few problems.

• The verification process is flimsy — They're using a Google form for applications to the program. Then after applying, no notification is sent that they're working on it or even received the application successfully (we've tried applying twice), with no progress updates or expected timeframe for completion.

• Verified RSS readers are still being blocked — There are reports that RSS readers Cloudflare has verified as "good bots" are still being blocked from websites. If Cloudflare has successfully approved an RSS reader as a "good bot", it shouldn't be blocked or still require website owners to add any custom exception rules.

• Unblocking RSS readers across multiple websites is cumbersome — Cloudflare's only resolution to unblocking RSS readers is for the owners of the readers to contact each website owner directly and ask for an exception to be made. While that may work for one-off cases, this is unreasonable for RSS readers that have to access thousands of different Cloudflare-enabled websites each day. It's also overwhelming for website owners to configure exceptions for each and every RSS reader.

To be clear, there's nothing wrong with using Cloudflare's security tools on your website to help deal with malicious AI bots, scrapers, and potential attacks. But Cloudflare needs to ensure that people who use RSS tools aren't blocked from accessing your website content, and make it easier to resolve when they are.

If you're interested in following this Cloudflare issue as it relates to Open RSS, we're tracking it in issue 144, which has its own RSS feed you can subscribe to for updates.