A new form of human interaction

Wed, 06 Aug 2025 00:00:00 +0000

Few days ago, Cloudflare published a blog post saying Perplexity is using undeclared crawlers to evade no-crawl directives. This sparked some interesting debates and I was particularly piqued by one arguing Perplexity is no different from a regular browser and should ignore no-crawl directives. As ridiculous as it initially sounded, I found the argument surprisingly valid and worth further thought. (Perplexity has since replied in the same defense).

Bots, Crawlers, robots.txt and no-crawl

Bots and crawlers are important elements of the web. These bots and crawlers (also called spiders) are automated programs that browse the internet without a human controlling them directly. (Crawlers are special types of bots that follow links and jump from one page to another).

Search engines like Google use crawlers to discover and index web pages, so they can show them in search results. SEO tools use them to analyze how search engines see your websites. Monitoring tools use them to check if your site is up or down. And there are many more examples like price tracking, content aggregation, and social media auto-posting. But not all bots are good. Bots can also be used to do harmful things like spamming comments or forms, trying passwords on login pages, or for DDos attacks.

To give website owners control of how bots interact with their site, robots.txt was designed. The robots.txt file sits in the root directory of the website (example.com/robots.txt) and contains a directive on how bots should interact with the site—bots allowed/disallowed, pages to crawl, etc. In other words, the robots.txt says don’t crawl this site or this list of pages and the bots comply. But do they?

Browsers vs Bots

The argument “But Google Chrome browsers by users are not blocked, why is Perplexity different than a regular browser” falls apart by the single fact that browsers are fundamentally different from bots. Browsers are programs as well, but they are human-controlled programs (generally called clients) designed for direct human interaction. This human interaction is a core component of the web that we have designed around:

Discussions in forums and boards
Commenting on blog posts
Interacting with ads
Recommending a post to a friend
Support via creator support platforms
Followers and likes

Clients: browsers or bots?

The way we interact with the web is rapidly changing. Human interaction that used to mean manually visiting a link and doing stuff on a web page can now be done with programmatic instructions to a [special type of] browser. Better still, tasks can now be performed by simply describing them in natural language: “open example.com and login with this username and password”. AI assistants interact in a similar way with webpages to answer specific real-time questions or process human requests like bookings.

While we can argue that this is still different from manually visiting a browser to perform the tasks, we can agree it definitely blurs the line between what we define as “real” human interaction and not. Should this type of interaction follow robots.txt directives or ignore them like browsers? Should we have a new directive for this type of human-interaction? Should a website not allow me to place a food order or book air travel because I didn’t manually visit the page to place the request but did via a different form of client?

It is a lost war

Blocking bots is at best a won battle but a lost war. For starters, there is no easy way to enforce robots.txt compliance or any non-client preventive rule. Unethical bots don’t follow them anyway. Overly strict methods like full-page captchas on content pages may work in the short term but will harm usability in the long run and ostracize that content from the web.

I will repeat, “the way we interact with the web is rapidly changing”. I can’t remember the last time I visited Stack Overflow. Or Wikipedia. As web authors (owner, manager or whatever role you play), we need to redefine what we think human interaction is and redesign to accommodate the changes happening; designing expecting that the page will be accessed outside the standard human-visit methods we are familiar with.

I get it. Human interaction on any web page is what converts to revenue—ads, subscriptions, audience, purchase. Losing this is the big fear (other fear of bots like spam have working or at least somewhat working answers). However it won’t get better. We need to start thinking of alternative ways to “fund” the web.

On Prayer

Wed, 11 Jun 2025 00:00:00 +0000

“I still maintain my stand that prayers change nothing. If prayers need you to put your effort first to function, that means prayers do absolutely nothing; what is working is your efforts.” - X

This is a genuine thought a lot of us have had at one point or another. These questions are good as they challenge what we think we know, what is true, and help us seek deeper understanding.

Here is how I think about it:

1.

A good place to start is defining what prayer is. For a lot of us (Christians), we understand prayer primarily as a channel of petition and request. This is part of it but if this is all you do in prayer, you will run out of things to ask and get bored. You can try it; I have.

Prayer at its core, is fellowship. It is simply being present; like spending time with a friend, your kids or your partner. It is less about the ask and more about communion (Romans 8:26).

2.

So if request is part of prayer, meaning I can ask for things, why is effort needed?

(The fact the effort is mentioned means this is a request that effort alone can fulfill—usually material things. When people ask questions like this, they are often referring to wealth).

Close your eyes and imagine a world where all you need to do to get things is pray. I think it would be a world filled with religious facade and devoid of purpose or fulfillment. It wouldn’t be a good place.

Prayer isn’t to bypass the human experience of effort and growth. That will be unfair of a just God. Perhaps the “answer” to prayer is often the strength or wisdom to engage with life’s challenges, rather than their removal.

As humans we are limited. We get tired (Isaiah 40:30, 1 Kings 19:4). We make mistakes. Prayer gives strength (Psalm 138:3, Luke 22:43), peace (Philippians 4:7), and clarity (James 1:5, Jeremiah 33:3).

By the way, I really doubt anyone can attribute all they have achieved in life to their sole effort—maybe most things, but definitely not all.

3.

So are you saying “Don’t pray for things”?

It’s ok to pray for things. We’ve all been in situations where we could use some help. I could really use a couple thousand pounds right now. But God doesn’t have pounds. He doesn’t spend them. The world’s value system is designed by man. So if I am praying for money, it has to come through this system—it won’t fall from heaven. What channel can this prayer be answered through? Have I invested in building strong relationships with friends God could nudge to give me that? Have I worked on acquiring high-value skills that someone can pay me for?

Miracles do happen but systems and principles govern the everyday life (John 6:26).

4.

What about the time I prayed for stuff (especially non-material things) and it didn’t happen?

Towards the end of December last year, J dreamt B was seriously beaten and she had to take her to the hospital. Before they got to the hospital, she woke up. She shared this with me and I nudged her to give B a call and check on her. Turned out B had been sick. J forced her to the hospital for a proper test and the doctor discovered there was fluid in one of her lungs. Some weeks and 3 operations later, a cancerous mass was discovered in the lung.

This year, I prayed like I have never prayed before. And fasted. At least 30 minutes every morning and 1 hour every night, praying for a miracle. J was on another level. Besides our own sessions, she does at least another hour with her prayer group.

Even when the doctor had a meeting with the family and close friends after her failed radiotheraphy and chemo, we still had faith. I mean, why did J had the dream in the first place? One of those nights, I was literally just crying in prayer. If anything happened to B, we would be shattered.

We lost her in April. And we were broken. B was the sister I never had. It was really devastating.

I had many questions. What did we miss? Was it that we didn’t pray enough?

There are many possible reasons we don’t get what we pray for. There are all those things you have heard—wrong kind of prayer, character development, faith building, etc. Sometimes, even our own actions sabotage our prayers. Personally, I try to focus on two things:

Unbelief—a lack of that inner alignment and conviction¹ about what you are praying for. God can, but will he? Have I reached that place of conviction that he will?
God’s will (1 John 5:14) and sovereignty. As humans, our focus and interests limit our perspective. In the grand scheme of things, our desires may not align with God’s plan (Job 42:2–3).

I havent really been able to pray in the last 2 months. The few times I have managed to, all I pray for is strength. Some days are fine, but some evenings, I still see J cry.

So yes, what did we miss? Why wasn’t she healed?

Honestly, I don’t know. But as I try to find my way back, I find solace in two truths:

My role is to put in the effort (Titus 3:8) and confidently ask (1 John 5:14).
I can trust in His wisdom and plan to take care of the rest, accepting the outcome, whatever that is (Romans 8:28).

I don’t have all the answers. These are just my own thoughts. Maybe they are wrong, maybe they don’t fully answer it, but I am curious to know what you think about prayer.

Dolapo Lawal has one of the best teaching I have heard on this—how to move mountains 1, 2 and 3 ↩

Inflowdash - A dashboard and calendar of your upcoming revenue

Tue, 06 May 2025 00:00:00 +0000

I have been using an internal tool for some time now to check upcoming revenue for Engage. It connects to Stripe, pulls data for upcoming revenue and displays it in a calendar form. That’s it; nothing magical. This tool is now available at inflowdash.com

Why?

Current metrics dashboards are focused on data after the fact (after revenue is in). This is rightly so as they tell a story of growth. For inflowdash, it is the other way - data before the fact, which is perfect for forecasting. This helps:

Forecast and plan cash flow
Identify problems early
Manage churn and expansion revenue
Set accurate goals

If you do usage billing (like we do), where pricing is based on usage and not particularly fixed, this is even more accurate.

So check it out: inflowdash.com, share, and send me feedback: @kehers

Email/Password Authentication Checklist

Tue, 17 Sep 2024 00:00:00 +0000

Here is a fairly rough checklist of things you should think of when building an email/password authentication system for a fairly sensitive application. This assumes the basics like securely hashing passwords, proper session management, and transmitting data over SSL are already covered. For a more comprehensive list, see the OWASP Cheat Sheet on Authentication.

Ensure passwords are a minimum of 8 characters. Support all printable ASCII characters and spaces.
Implement two-factor authentication with an authenticator app at the minimum.
If your app allows multiple users, allow the admin to be able to “force” 2FA for all members.
Require password to update password or email.
Send an email notification on password update.
Send email notifications to both old and new email when email changes.
Lock account (could be for a specific period) after multiple failed login attempts. (OWASP recommends also logging the failed attempts)
Log authentication events with metadata like IP, client, and OS details.
If a user is accessing from a “new location”, and has no 2FA on account, send an email verification code. “New location” can be based on the client, IP, or a special fingerprint algorithm.
Throttle codes sent via email (or SMS) to the same user. You can use an exponential backoff strategy to increase delay between each message sent and have a limit over a specific period, e.g., maximum of 4 verification codes to the same user over a 1 hour period.
Allow users to be able to log out of all devices they are currently logged in on. This can be a button on their settings page. To implement this, you need to be able to map users to their session IDs and delete sessions by IDs. This may mean adding a DB store (Redis is a good choice) to your session layer.
When users change their password, log them out of all other devices they are currently logged in on. A more subtle implementation of this that I have seen makes this optional. A provided checkbox lets you decide if you want to or not.

If I remember more, I will update the list.

Serverless vs Kafka

Mon, 16 Sep 2024 00:00:00 +0000

I have been moving a lot of Engage’s architecture to Kafka of late. Even though the monolithic side has been winning the microservice vs monolithic debate in recent times, I am still a big fan of serverless microservices—using PaaS (we use Google Cloud) to host microservices. To be fair, what we do is more of a serverless architecture than a proper microservice architecture but our design goal aims to achieve the same as with a microservice architecture. This has helped us handle millions of events daily from the API and integrations with less worry about scalability.

So why move off PaaS? A couple of reasons. Top of it are two things.

Bringing together many separate services to make them compact—easier testing, faster interpolation of shared resources, more flexibility, and simpler deployment.
Predictive service cost. Note that this is not the same as being cheaper, although our current Kafka setup is, which is great. Being predictive is that I know I am going to pay just about the same amount whether or not I processed 50 million or 1 million events during the month.

But is the grass really greener on the Kafka side? Nope.

There is the initial overhead of setting things up. (This is if you want to self-manage. You can get another PaaS that can do that for you. There are a couple of managed Kafka services). There is also your consumer design, which is a crucial part. You need to be able to design your consumers in a way that they can consume messages concurrently and efficiently. If for example, you have a topic with 100 partitions that is receiving messages at that throughput, or even more, then you need consumers processing at that speed as well. Now think of 10 topics of similar throughput. So what are you looking at? Threads? Multiple container instances? Node pools? With a serverless architecture, Google Cloud Function for example, you can specify a minimum and maximum number of instances to be deployed to handle such throughput and everything automatically gets taken care of for you. And that’s the beauty of serverless. You are less bothered about the abstraction.

There are other things. Kafka doesn’t have any delay mechanism. Most serverless PaaS do. Amazon SQS has delays. Google Pub/Sub doesn’t but you can use Cloud Tasks.

There is also monitoring and logging. Monitoring is how you know when something is wrong with Kafka or your consumers are failing. Logging is how you ensure messages are properly processed. I have found heartbeats monitoring a good way to monitor consumer uptime. For every consumer, register a heartbeat every time a message is processed. If no message is processed within a time frame (this will depend on the type of message the consumer processes), trigger an alert. (Betterstack has a good heartbeat monitoring service).

All that said, things are holding up fine so far. Would our dependence on [serverless] PaaS be zero? I honestly doubt it.

[Building Engage] #5. Working with unstructured data

Wed, 14 Feb 2024 00:00:00 +0000

A lot of content around data wrangling talks about dealing with structured data—data with a well defined schema. You can clearly define your database columns based on this schema to fit the incoming data. Most apps deal with this kind of data, so I understand. But for the other few, Engage for example, dealing with unstructured data is big work.

I am not talking about storage here. Storing unstructured data is easy. If you are using an SQL db, you can normalize the unstructured data (JSON stringify for example) in a column. If you are using a NoSQL db, even better. MongoDB for example supports schemaless data by design. Just dump your data in JSON style and you are good. But how do you store the data in a way that is easy to “search” the different fields? How do you sort on any of the fields? What about aggregation?

At Engage, users can track custom user attributes, events and properties. This means, each business have unique user data points and can do things like segment users based on those data points.

Consequently, this means in total, the different user attribute fields from all businesses can be in hundreds, thousands, and eventually millions. Let me explain properly. Business A can have the following user attributes:

name
gender
location
avatar
city

Business B:

name
age
website
occupation
address
rating

By the time you take that across a thousand businesses, 4 unique user attributes per one, that’s about 4,000 unique attributes. How for example do you store this in a way that the business can easily do a search like “Find all users that live in Lagos”? 4k unique columns that are indexed? Then do this for user events data. And events properties.

Unfortunately, this is not an indepth post to explore this. I hope to be able to gather more conversations, posts, and talks about this. But here are some pointers based on our experience.

Find a database suited for your need. What you should be looking at are databases that are specifically designed for logging. I know you are probably looking for a recommendation here but business needs varies. This sounds easy but it’s not. You will have to experiment with different options and finally decide on what really works for you. That is the hard work.
You can actually store “all” the unique fields using Elasticsearch’s dynamic mapping. The fields are indexed and you can perform search and aggregations on them. Well, maybe not all. There is a setting that limits this to 1k fields which of course you can change. But remember, the more the fields, the more degradation on performance and resources.
“Mapping explosion” happens when the number of these mapped fields in Elasticsearch is too high. One trick to prevent hitting it is to create multiple versions of the same index and route each query to the right index. So for example, you can cluster every account into a specific group (I love using the 16 hex digits: 0-9 and a-f). Suffix your index with the group, e.g, “logs” become “logs-a”, “logs-b”, etc. Requests goes to the specific index for the account making the request. What this means is that each of these indices would have 1k fields (so 16k altogether). The performance impact is way better than using a single index (say “logs”) and increasing the mapping limit to 16k.
ClickHouse has JSON fields that allow you store unstructured data as JSON objects. It also has JSON functions to extract data from these fields and manipulate them. PostHog has a post about working with this. Personally, I found the Pairwise Arrays option more performant so it’s something you can also explore. (Read the whole post about Working with JSON in ClickHouse)

I would love to hear about your own experiences, what has worked for you, talks, posts, videos and everything about working with unstructured data that you have seen. I will try and do more detailed posts about working with unstructured data as well.

I am @kehers on Twitter (or X as it is now called).

2023: Reset

Fri, 29 Dec 2023 00:00:00 +0000

2022 was rough. The year started with lots of energy - a new baby, Engage growing, and then new realities started kicking in. I watched Engage’s runway drain as we struggled with reaching product market fit. We tried many ideas from the playbook - inbound sales, community, socials, and ads, but conversion growth was slow.

The reasonable choice was to raise a new round but I didn’t want to get distracted from the product. I knew what effort went into raising the first round and how distracting and stressful it could be. That coupled with our slow growth, I decided against it. I kept convincing myself we would be able to figure it out and be profitable before we ran out of funds. You guessed right, I was wrong.

But we kept pushing. I had stopped paying myself way earlier and was really struggling. And piling on debts. At this point, it was obvious the kind of growth I was waiting for to justify our next raise wasn’t going to happen that fast so I started engaging my investors about raising a new round. Then my dad died in August.

I felt numb. Then pain. Not from the loss. But that I didn’t get closure on a lot of things I wanted to. Then anger.

Towards the end of the year, a lucky break finally happened. I got a windfall from a secondary equity sale. I was able to settle my debts and loaned a big chunk to Engage to extend our runway while I continued my engagement with my investors.

2023 started with lethargy. I kept thinking about money, growth, raising, what could change, and everything in between. On the home front, there was peace. Even when everything around was crumbling, that, and my faith were my solace. I would wake up every morning, look at the family, and get the energy to face the day.

In February, I got wind of Tech Nation shutting down. My friend had been pressing me for over 3 years to apply but I would shrug it off. This time, it felt the timing was right. I needed that reset, especially for the family. So I prayed and applied. A bit over a week later, I got a yes.

The relocating experience wasn’t as easy as I thought it would be. But that is a totally different post I need to take time to write in detail. I traveled to the UK in May for about a month to explore and get accommodation. By the end of June, myself and the family relocated to the UK.

In May and the following months, I suffered serious burnout. At first, I thought it was because I was away from family and support systems, especially with faith. But even after the family relocated, it took some effort to get back in shape. It was strange. I was happy about the move, although not ecstatic (I don’t get ecstatic about a lot of things). I was happy for a lot of things – settling in a place with proper and better systems; where I do not have to worry about low-level problems like electricity or water. But more importantly, seeing my daughter jump ecstatically when we took her to her new school and the new radiance on my wife’s face brought me lots of joy and made it worth it.

At this point, Engage was at the fatal pinch. Raising in 2023 was another level of hard. And I failed at trying to do so. By Q3, it came down to 2 options: increase revenue and/or decrease spend. Increasing revenue wasn’t something that could be forced, so I started with the other option. I had a call with the team and explained the situation to them. I had to let people go and drop expenses in many other ways. Fortunately, by Q4, revenue, was enough to cover expenses.

Things are calmer now. My wife started a new job and I am adjusting to the reality of the new environment, culture, and life. Of course, I am still worried about money. I don’t have a proper source of income yet and living expenses are more expensive here. But I am focused on the positives. It was a crazy year, but a lot of things went right. Every morning, the first thing on my mind is gratitude for grace, privilege, family, and how far we have come. I have seen the future. It’s beautiful. I choose to focus on that.

[Building Engage] #4. Avatars

Sun, 08 Oct 2023 00:00:00 +0000

Features like avatars look trivial in products with user profiles, but most times, they are not. (Is anything actually trivial?). It’s even more interesting, as with Engage, when users can’t upload their profile pictures themselves. Where do we get one from?

Gravatar!

If you are familiar with this, this would be your answer as well. Gravatar is one of the oldest forms of getting people’s profile pictures. Make a request with the person’s [encrypted] email address as a parameter and get a profile picture in return.

const hash = md5(email)
return `<img src="https://www.gravatar.com/avatar/${hash}">`

There are two major drawbacks though.

Gravatar uses email address as the unique identifier. This means if you don’t have the user’s email (a product that doesn’t collect user’s email address for example) there is no way you can get the user’s profile picture.
The user needs to have set up that email on Gravatar. While a lot of people have their emails set up on Gravatar, some people do not. This means having a user’s email does not automatically guarantee that they will have a profile picture on Gravatar.

Default/fallback image

Gravatar provides a solution to the second challenge – if the profile does not exist. You can provide a fallback image (as a URL) that is displayed instead.

const hash = md5(email)
const fallback = encodeURIComponent('https://my.app/images/default.png')
return `<img src="https://www.gravatar.com/avatar/${hash}?d=${fallback}">`

This is good. But it means every profile with no gravatar will have the same default image. If you are Facebook or Twitter (X), that may be fine. For us, I wanted some uniqueness. Gravatar has a way to generate unique defaults but I am not a fan of the designs. Are there alternative avatar services that can be used instead of Gravatar or as a fallback for Gravatar? A lot. But what are we looking for to start with?

Service Checklist

I created a checklist of what we want in a service.

It should be able to use email as an identifier, i.e., pull the avatar attached to the user’s email.
Emails are optional user attributes on Engage, so if it doesn’t exist, the name (first or last name) should be used. Usernames would have been the next best thing but usernames are not standard attributes on Engage. Names can’t be used to pull a profile picture but we can use it to create an avatar of initials.
If the user has an email but it doesn’t return a profile picture, get initials from the email and use that to create an avatar.
If the profile has no names (names are optional as well), generate a unique random avatar using the user’s ID. We do not want a default profile picture for all users with no email or name.

Unavatar

I love Unavatar! And for many reasons. It’s open source, has simple implementation, and pulls profile pictures from many places.

Just append the user’s username, email, or domain to unavatar.io and that’s it

<img src="https://unavatar.io/engage.so">

Unavatar checks preference 1. It doesn’t check preference 2 but has a fallback option so we can get an initials avatar service as fallback. It doesn’t check preference 3 as well but we can use a different service as a fallback if the user has no name.

Vercel Avatar

Vercel has an avatar service that generates beautiful gradients and can generate initials from a text parameter. We can’t use it to pull profile images using emails as reference but it is a perfect fallback for other services because it can generate a unique avatar from an ID and also generate a unique avatar with initials.

Generating a unique [gradient] avatar from the user’s ID works well as fallback for Gravatar and Unavatar but initials avatar does not work for Gravatar. The unique avatar can be generated in PNG format (default) but the initials avatar has to be SVG. Gravatar does not support SVG as a fallback image.

Tiley

Outline has Tiley, an avatar service that solves the SVG problem. We can generate an initials avatar in PNG, JPG, or even SVG format.

Tiley can’t generate a unique avatar without the initials. And if we don’t want to go with Vercel Avatar as a fallback for unique avatars when we don’t have the user’s email or name, then we can use another service for unique avatars.

Boring Avatar

My recent favorite is Boring Avatars. I love the styles and there is a documentation on how to use it as a service.

<img src="https://source.boringavatars.com/${variant}/${size}/${id}

What should I use?

As with everything, it depends. What user attributes do you have? Email? Username? Name? Do you mind the same default image for all users? Or do you want something dynamic? What design style? Can users upload their profile pictures themselves?

In the end, we went with Boring Avatar for profiles without email or name and Gravatar for profiles with email or name with Tiley as a backup for initials. Earlier versions of Engage used Unavatar and it will always be my first choice. However, even though I doubt we would ever hit it, I was [subconsciously] conscious the hosted version is rate-limited and I didn’t want to self-host yet another service.

Re: Passkeys

Sun, 28 May 2023 00:00:00 +0000

“Passkeys are a new, passwordless authentication method that offer a convenient authentication experience for sites and apps, using just a fingerprint, face scan or other screen lock.” (source). Don’t think too much about how it works, you can check out the demo at passkeys.io. I just tested it myself and couldn’t help but comment on the ease.

I love this – anything that replaces password authentication actually, but in the end, ease (of use and technical implementation) will determine its adoption and consequently, if this will totally replace passwords.

Google claims it’s easier to use but I’d argue “easier” is relative. It feels technical experiencing it for the first time.

Below is a signup process (implementations will vary but it’s typically going to be close to this):

Step 1 - create an account with your username/email

Step 2 - set a passkey

For a non-technical person, Step 2 may be confusing. Or at least cause a pause. I actually paused to understand what exactly was going on there. One question I’d assume the user will be asking at that point is, “What am I supposed to do here?”

If I save the passkey on the device, logins are actually easier.

I click “Continue”, I am requested to unlock my laptop and I am logged in. But what happens if I don’t have any lock on the device? I tested this on my phone and I couldn’t create a passkey or log in without a lock on the device itself. This makes sense as it ensures access to your unlocked device does not mean access to your logins.

The experience of logging in on a device when you have a passkey on another device is not what I will call easy as well. There is no “easy” way to connect to the passkey saved on another device. Below is trying to log in on mobile while my passkey is on my laptop.

Looking at the friction of these options, give me password authentication please.

Passkey is a good improvement compared to other passwordless methods: multi-device authentication compared to WebAuthn and less friction (not to mention delivery delays and failures) compared to one-time codes/links. I am keenly looking forward to what adoption will look like for this one.

[Building Engage] #3. Fighting spam

Tue, 22 Mar 2022 00:00:00 +0000

If you give people any tool for engagement, expect it to be used for spam. This is worse for marketing tools. There are always bad actors looking to exploit this to send phishing emails and spam to people.

For Engage, the earliest fail-safe we placed was to not provide email sending infrastructure ourselves but allow customers connect their existing infrastructure. To Engage, you had to connect your ESP (email service provider). (We support Mailgun, AWS, Sparkpost and Sendgrid). This solved the issue on two fronts:

These services have processes in place to ensure you are genuine. For you to have a production-ready AWS SES account, for example, you needed to have made a valid argument for your use case and successfully passed the review process.
These services track your use according to policy limits and penalize when necessary. They actively track things like your bounce rates and complaint rates and if these go above particular thresholds, your account gets disabled.

But there was a problem with this experience. Since our primary targets are product marketers/managers, telling them to connect an ESP before they get value on the product creates friction. They probably don’t know what ESP they use or care enough (yet) to connect the ESP. And not everyone wants to connect their ESP anyway. To reduce the path to value as much as possible, we had to automatically provision a default way to send outbound emails without the need for customers to connect an ESP. And we were back at solving the spam issue ourselves.

There are different approaches to solving spam depending on what you are building. For marketing tools, the general approach is to limit what customers can do, review the account and whitelist. There are slight variations of this approach but the general concept remains the same. This was the same approach we took.

Our original plan was to limit to a particular allowed daily sends (50? 100? 10?). Customers can then request full access. One other approach we considered was starting accounts in sandbox mode where we only allow sends to a whitelisted list of emails (emails need to be verified to be added). To move to production, a request is made and we review. This is the common pattern with transactional ESPs. But it adds another layer of friction and complexity that we would rather not include. In the end, we decided to completely limit email sends for new accounts. Customers can explore and use other features 100% but email sending (both broadcast and automation) is disabled till we manually review.

The drawback with the approach is that we need to review every single signup (and quickly too). But that’s not a bad thing. It lets us know our customers. At scale, we may not be able to manually do it fast as necessary but there are ways to solve that, e.g. automated checks, machine learning flagging.

But signup is only the beginning. Fighting spam is continuous work. After signup and whitelisting, it is important to continually monitor and ensure people use the service responsibly. Bounces and complaints need to be regularly monitored to ensure it falls within an acceptable range. I particular love how Intercom also checks links in outbound messages for web threats and this is something we have also adopted.