Building a safe and respectful group chat experience is more important than ever, especially in online gaming communities where conversations can scale quickly. A dedicated group moderator can make this easier by enforcing community standards, reducing toxic behavior, and keeping discussions welcoming for everyone.

With Stream’s AI Moderation built into the Chat API, moderation becomes even more seamless by automatically detecting harmful messages, filtering profanity, and supporting moderators in creating a positive environment for players.

In this guide, you’ll learn how to combat a toxic community by building a group chat with an AI chat moderator. You’ll implement it using Stream’s AI Moderation Service, Chat API, and React Native SDK to review user messages and actively monitor them for harmful content. The goal isn’t to police users but to create a space where they can engage freely without worrying about spam, hate speech, or toxic behavior.

Prerequisites

For this tutorial, you will need:

• Node.js and npm are installed.
• React Native CLI installed.
• A free Stream account to get your API key and secret.
• An Ngrok account to expose your local server to Stream webhooks.

Building the Group Chat

Once you’ve completed all the prerequisites, you’ll set up a basic group chat app using React Native and Stream. This app will include the essential components for a chat experience: a channel list, a message list, a message composer, and a new group creation flow.

Start by creating a new folder to store all your project files. A good place for this is your desktop, as it’s easy to access. Inside this folder, create a new file named \.env\.

Next, copy and paste the following code into your newly created \.env\ file. This file will hold important configuration details for your app and token server.

STREAM\_KEY=YOUR\_STREAM\_KEY  
STREAM\_SECRET=YOUR\_STREAM\_SECRET  
PORT=5050  

The \.env\ file is where you save your Stream credentials, which consists of your Stream key and your Stream secret key found on your account dashboard, as seen in the image below.

Create a new file called \server.js\ in the main project folder. Then, please copy and paste the following code into it.

import 'dotenv/config';
import express from 'express';
import cors from 'cors';
import { StreamChat } from 'stream-chat';

const app = express();

app.use(cors());
app.use(express.json());

const STREAM_KEY = process.env.STREAM_KEY;
const STREAM_SECRET = process.env.STREAM_SECRET;
const PORT = process.env.PORT || 5050;

if (!STREAM_KEY || !STREAM_SECRET) {
  console.error('Missing STREAM_KEY or STREAM_SECRET in .env');
  process.exit(1);
}

const serverClient = StreamChat.getInstance(STREAM_KEY, STREAM_SECRET);

app.post('/token', async (req, res) => {
  try {
    const { user_id } = req.body || {};

    if (!user_id) {
      return res.status(400).json({ error: 'user_id required' });
    }

    const token = serverClient.createToken(user_id);
    await serverClient.upsertUser({ id: user_id, name: user_id });

    res.json({
      token,
      user: { id: user_id, name: user_id },
      api_key: STREAM_KEY
    });
  } catch (err) {
    console.error(err);
    res.status(500).json({ error: 'failed_to_issue_token' });
  }
});

app.listen(PORT, () => {
  console.log(`Token server running at http://localhost:${PORT}`);
});

Create another file called \package.json\ in the main project folder. Then, copy and paste the following code into it.

{

  "name": "chat-token-server",

  "private": true,

  "type": "module",

  "dependencies": {

    "cors": "^2.8.5",

    "dotenv": "^16.4.5",

    "express": "^4.19.2",

    "stream-chat": "^9.18.1"

  },

  "scripts": {

    "start": "node index.js"

  }

}

Creating a New React Native Project

After setting up your \server.js\ and \.env\ files, you’ll need to create the group chat's user interface.

To build the UI, go to your main project folder and create a new React Native project. You can do this by running the following commands in your terminal from the project’s root folder.

npx create-expo-app rn-stream-chat

Installing Necessary Dependencies

Next, you’ll install the Stream Chat React Native SDK and its dependencies.

cd rn-stream-chat

npm i stream-chat stream-chat-react-native react-native-gesture-handler react-native-reanimated react-native-safe-area-context react-native-svg

​​Note that the RN SDK wraps the JS client and keeps versions compatible. The Stream Docs warn against manually installing stream-chat to avoid version skew, but if you do, pin to a compatible version and don’t duplicate it. If you hit weird client issues, remove your explicit stream-chat dep and let the RN SDK supply it.

Create a file named babel.config.js and paste the code below into it.

module.exports \= function(api) {

  api.cache(true);

  return {

    presets: \['babel-preset-expo'\],

    plugins: \['react-native-reanimated/plugin'\],

  };

};

Create a new folder called src to store your project files. Inside src, add a subfolder named stream, then create a file called client.ts and paste the following code into it.

import { StreamChat } from 'stream-chat';

let client: StreamChat | null \= null;

export function getStreamClient(apiKey: string) {

  if (\!client) client \= StreamChat.getInstance(apiKey);

  return client;

}

Next, you will create a new folder inside the \src\ folder named \screens\, then create a file \/LoginScreen.tsx\, and paste the code below inside.

import React, { useState } from 'react';

import { View, Text, TextInput, Button, Alert, Platform } from 'react-native';

type Props \= {

  onLoggedIn: (args: { userId: string, token: string, apiKey: string }) \=\> void;

};

export default function LoginScreen({ onLoggedIn }: Props) {

  const \[userId, setUserId\] \= useState('joel');

  const \[serverUrl, setServerUrl\] \= useState(

    Platform.select({ ios: 'http://127.0.0.1:5050', default: 'http://10.0.2.2:5050' })\!

  );

  const login \= async () \=\> {

    try {

      const res \= await fetch(\`${serverUrl}/token\`, {

        method: 'POST',

        headers: { 'Content-Type': 'application/json' },

        body: JSON.stringify({ user\_id: userId })

      });

      const data \= await res.json();

      if (\!res.ok) throw new Error(data?.error || 'Failed');

      onLoggedIn({ userId, token: data.token, apiKey: data.api\_key });

    } catch (e:any) {

      Alert.alert('Login failed', e.message);

    }

  };

  return (

    \<View style={{ flex: 1, padding: 20, gap: 12, justifyContent: 'center' }}\>

      \<Text style={{ fontSize: 22, fontWeight: '600' }}\>Sign in\</Text\>

      \<Text\>Pick a username (must match a seeded user or any string):\</Text\>

      \<TextInput

        value={userId}

        onChangeText={setUserId}

        placeholder="e.g. joel"

        style={{ borderWidth: 1, borderColor: '\#ccc', borderRadius: 8, padding: 12 }}

        autoCapitalize="none"

      /\>

      \<Text\>Token server URL\</Text\>

      \<TextInput

        value={serverUrl}

        onChangeText={setServerUrl}

        placeholder="http://10.0.2.2:5050 (Android Emulator)"

        style={{ borderWidth: 1, borderColor: '\#ccc', borderRadius: 8, padding: 12 }}

        autoCapitalize="none"

      /\>

      \<Button title="Continue" onPress={login} /\>

    \</View\>

  );

}

The next step is to create a new file named \ChannelScreen.tsx\ inside the \src/screens\ folder and paste the code below.

import React, { useEffect, useState } from 'react';

import { ActivityIndicator, View } from 'react-native';

import { Channel, MessageList, MessageInput, useChatContext } from 'stream-chat-expo';

import type { StackScreenProps } from '@react-navigation/native-stack';

type RootStackParamList \= {

  ChannelList: undefined;

  Channel: { channelId: string };

  NewGroup: undefined;

};

type Props \= StackScreenProps\<RootStackParamList, 'Channel'\>;

export default function ChannelScreen({ route }: Props) {

  const { client } \= useChatContext();

  const \[chan, setChan\] \= useState\<any\>(null);

  useEffect(() \=\> {

    let mounted \= true;

    (async () \=\> {

      const c \= client.channel('messaging', route.params.channelId);

      await c.watch();

      if (mounted) setChan(c);

    })();

    return () \=\> { mounted \= false; };

  }, \[client, route.params.channelId\]);

  if (\!chan) return \<View style={{ flex:1, justifyContent:'center', alignItems:'center' }}\>\<ActivityIndicator /\>\</View\>;

  return (

    \<Channel channel={chan}\>

      \<MessageList /\>

      \<MessageInput /\>

    \</Channel\>

  );

}

Finally, you’ll create a file named \app.tsx\ in the chat-group folder and paste in the code below. This code writes everything about the login and connects the Stream to the screens.

import React, { useEffect, useState } from 'react';

import { SafeAreaView, StatusBar, View, ActivityIndicator } from 'react-native';

import { NavigationContainer } from '@react-navigation/native';

import { createNativeStackNavigator } from '@react-navigation/native-stack';

import { OverlayProvider, Chat, useChatContext } from 'stream-chat-expo';

import { getStreamClient } from './src/stream/client';

import LoginScreen from './src/screens/LoginScreen';

import ChannelListScreen from './src/screens/ChannelListScreen';

import ChannelScreen from './src/screens/ChannelScreen';

import NewGroupScreen from './src/screens/NewGroupScreen';

type RootStackParamList \= {

  ChannelList: undefined;

  Channel: { channelId: string };

  NewGroup: undefined;

};

const Stack \= createNativeStackNavigator\<RootStackParamList\>();

function ChatStack() {

  return (

    \<Stack.Navigator\>

      \<Stack.Screen name="ChannelList" component={ChannelListScreen} options={{ title: 'Groups' }} /\>

      \<Stack.Screen name="NewGroup" component={NewGroupScreen} options={{ title: 'New Group' }} /\>

      \<Stack.Screen name="Channel" component={ChannelScreen} options={{ title: 'Chat' }} /\>

    \</Stack.Navigator\>

  );

}

function ConnectedChat({ apiKey, userId, token }: { apiKey: string; userId: string; token: string }) {

  const \[ready, setReady\] \= useState(false);

  const client \= getStreamClient(apiKey);

  useEffect(() \=\> {

    let mounted \= true;

    (async () \=\> {

      await client.connectUser({ id: userId, name: userId }, token);

      if (mounted) setReady(true);

    })();

    return () \=\> {

      mounted \= false;

      client.disconnectUser();

    };

  }, \[apiKey, userId, token\]);

  if (\!ready) return \<View style={{ flex:1, justifyContent:'center', alignItems:'center' }}\>\<ActivityIndicator /\>\</View\>;

  return (

    \<OverlayProvider\>

      \<Chat client={client}\>

        \<ChatStack /\>

      \</Chat\>

    \</OverlayProvider\>

  );

}

export default function App() {

  const \[session, setSession\] \= useState\<{ apiKey: string; userId: string; token: string } | null\>(null);

  return (

    \<SafeAreaView style={{ flex: 1 }}\>

      \<StatusBar /\>

      \<NavigationContainer\>

        {session ? (

          \<ConnectedChat apiKey={session.apiKey} userId={session.userId} token={session.token} /\>

        ) : (

          \<LoginScreen onLoggedIn={setSession} /\>

        )}

      \</NavigationContainer\>

    \</SafeAreaView\>

  );

}

Test the Application

Now that you’ve built the group chat, the next step is to run the application. To do so, you’ll start the token server following the steps below.

Navigate to the root folder where the \server.s\ is located and run the command below.

npm start

After running the command, your terminal should display something related to the screenshot below.

Finally, open a separate terminal, navigate to the \chat-group\ folder, and run the command below to start the iOS simulator.

npm start

To run the app, launch it on either an iOS Simulator by pressing “i” or an Android Emulator by pressing “a”. The token server URL is already set by default for both: http://10.0.2.2:5050 for Android and http://127.0.0.1:5050 for iOS.

Once in the app, log in as Alice or any of the other seeded users. To start a new chat, tap on a new group, select at least two users from the list, and then tap create. You can now begin chatting within the new group.

Below is a screenshot from the group chat before implementing the Stream AI moderator.

From the screenshot above, you can see several abusive slurs and profanity words being used, but none are flagged or blocked. That’s because you have not introduced the moderator yet.

Introduction to Group Chat Moderation with Stream

Stream’s AI-powered moderation API works natively with Stream’s Chat, Video, and Activity Feeds APIs, and it also works with inhouse experiences to automatically detect and act on a wide range of harmful content across text, images, live video, and audio.

Here’s a condensed list of some supported harms and categories:

• Spam: unsolicited or repetitive messages.
• Hatred: abusive or threatening language.
• Racism: discriminatory or prejudiced comments based on race or ethnicity.
• LGBTQIA+ phobia: negative or hostile comments towards LGBTQIA+ individuals.
• Misogyny: comments that express hatred or prejudice against women.
• Ableism: discriminatory or prejudiced comments based on physical or mental disabilities.
• Threat: statements expressing an intention to cause harm or violence to someone.
• Terrorism & violent extremism: support or promotion of terrorist activities or ideologies.
• Pedophilia: comments that express sexual interest in minors.
• Scam: fraudulent schemes or deceptive practices.
• Flood: excessive or repetitive posting of messages.
• Forbidden link: sharing links to prohibited or harmful websites.

Stream’s moderation system works by analyzing messages in real-time and taking action based on configurable policies. You can choose to:

• Flag messages: Mark messages for review by a human moderator.
• Block messages: Prevent other users from seeing your messages.
• Ban users: Temporarily or permanently block users from sending messages.
• Shadow block.
• Bounce and block.
• Bounce and flag.

Using Advanced AI Moderation Engines

While flagging, blocking, and banning are foundational tools in moderating group chat, they lack the nuance and adaptability needed for large and dynamic communities. Stream’s AI-powered moderation goes beyond these simple actions, offering sophisticated engines that understand intent, context, and subtlety.

Here’s what you gain:

AI Text Harm Detection (LLM)

The AI text harm detection uses Large Language Models to understand context and user intent, instead of only looking for profane words. It can catch subtle or indirect harmful content, like passive-aggressive remarks or harassment, that keyword filters often miss.

Here’s how it works:

As a developer, you can set up a few key parameters. First is the app context, which explains what your application is about. This helps the AI understand the background of the conversations. Next are the LLM rules, where you define harm labels and what they mean in your app. For example, you can explain what counts as “scam” or “hate speech” for your community. Stream then combines your app context, your rules, and the last few messages in a conversation.

With a bit of prompt engineering, it sends this to the LLM and asks it to classify the message based on the rules you provided.

Configuration:

Get started! Activate your free Stream account today and start prototyping your chat app.

You can set up the LLM moderation engine directly in the Stream dashboard. Follow these steps:

• Log in to your Stream dashboard.
• Go to Moderation in the sidebar.
• Open Policies.
• Select the policy you want to update.
• Click on AI (LLM) Text.

From here, you can choose harm categories, set actions like bounce and flag, bounce and block, and adjust sensitivity levels.

See the screenshot below for guidance on what it looks like.

AI Text Harm Detection (NLP) with Blocklists and Regex Filtering

A layered approach works best. Blocklists such as profanity_en_2020_v1 and regex rules give you fast and predictable filtering, while the NLP engine adds extra understanding of meaning. This makes it easier to tell the difference between harmless mentions and real violations.

Configuration:

To set up the AI Harm Engine, go to the AI Text section in your moderation policy. You will see a list of harm categories that the AI can detect, such as harassment, hate speech, spam, and explicit content.

For each category, you can choose what should happen: flag the message, block it, or shadow block it. You can also change the sensitivity level for different types of harmful content. This helps you fine-tune moderation to match your community.

It is usually best to start with lighter settings and then adjust as you learn more about how people in your community interact.

Severity Levels

The AI Text moderation engine can label harmful content as Low, Medium, High, or Critical. These levels are mainly used for toxic categories like hate speech or sexual harassment.

For each severity level, you can choose what action should happen. For example, flag, block, or shadow block.

See the screenshot below for how this looks in the dashboard.

Setting up Stream and Enabling Moderation

Now that your project is set up, the next step is configuring Stream and enabling the AI moderation features. Create your free account to log in to your Stream dashboard and create a new application. You’ll find your API key and secret in the app’s settings.

You can manage policies entirely in the dashboard with the lowest friction. Set rules (AI text/image engines, blocklists) and apply them to chat via config keys like \chat:messaging\, all messaging channels, and Stream. The dashboard lets you monitor the content flagged by your configured policies and provides tools to fine-tune those policies based on real data.

Setting up Moderation Policy

Within your Stream dashboard, you can configure your chat moderation settings. Begin by selecting the Chat section, then open the Moderation tab. From this screen, you’ll be able to enable Stream’s built-in moderation tools and apply the profanity_en_2020_v1 blocklist to filter out profanity.

For this tutorial, you will enable all the moderation categories and set the action to flag. This will automatically flag any harmful messages sent to the group chat.

Later, you can switch to the ‘block’ option by changing your Stream dashboard.

Implementing Moderation in the Group Chat

To add moderation to your group chat, you need to set up rules in your Stream dashboard. This allows you to take action against harmful content such as threats, self-harm, harassment, etc from the moderation/policies section.AI moderation policy and set the AI: Text to block

The Main App Component (Updated App.tsx)

The main \*App.tsx\* file will handle the initialization of the Stream client and the navigation between the channel list and the chat screen.

When you switch users, connecting and disconnecting should be clean. The server’s webhook will post a single notice so you won’t see duplicate messages. For a smooth user experience, you can use \additionalFlatListProps\ to help users scroll to older messages easily.

import React, { useEffect, useState, useCallback } from 'react';

import { SafeAreaView } from 'react-native';

import {

  Chat,

  ChannelList,

  Channel,

  MessageList,

  MessageInput,

  OverlayProvider,

} from 'stream-chat-react-native';

import { StreamChat } from 'stream-chat';

const backend \= 'http://localhost:5050'; 

const chatClient \= StreamChat.getInstance('dummy'); 

export default function App() {

  const \[ready, setReady\] \= useState(false);

  const \[activeChannel, setActiveChannel\] \= useState\<any\>(null);

  const userId \= 'alice';

  useEffect(() \=\> {

    (async () \=\> {

      const resp \= await fetch(\`${backend}/token?user\_id=${userId}\`).then(r \=\> r.json());

      const client \= StreamChat.getInstance(resp.apiKey);

      await client.connectUser(

        {

          id: userId,

          name: 'Alice',

          image: 'https://getstream.io/random\_png/?id=alice\&name=Alice',

        },

        resp.token

      );

      await fetch(\`${backend}/channel\`, {

        method: 'POST',

        headers: { 'Content-Type': 'application/json' },

        body: JSON.stringify({ channelId: 'demo-general', members: \['alice', 'bob'\] }),

      });

      (global as any).chatClient \= client; 

      setReady(true);

    })();

    return () \=\> {

      const c \= (global as any).chatClient;

      if (c) c.disconnectUser();

    };

  }, \[\]);

  const onSend \= useCallback(async (message: any) \=\> {

    try {

      await activeChannel.sendMessage(message);

    } catch (e: any) {

      alert(e?.message ?? 'Message rejected by moderation rules.');

    }

  }, \[activeChannel\]);

  if (\!ready) return null;

  return (

    \<OverlayProvider\>

      \<SafeAreaView style={{ flex: 1 }}\>

        \<Chat client={(global as any).chatClient}\>

          {activeChannel ? (

            \<Channel channel={activeChannel}\>

              \<MessageList /\>

              \<MessageInput onSend={onSend} /\>

            \</Channel\>

          ) : (

            \<ChannelList

              filters={{ type: 'messaging', members: { $in: \['alice'\] } }}

              sort={{ last\_message\_at: \-1 }}

              onSelect={(ch) \=\> setActiveChannel(ch)}

            /\>

          )}

        \</Chat\>

      \</SafeAreaView\>

    \</OverlayProvider\>

  );

}

Test the Application

To test the application, you’ll first navigate to the root folder or the backend folder where the \server.js\ is located, then open the terminal and paste the command below:

node server.js

After running the command below, you should see something related to the screenshot below:

Second, you have to open a new terminal different from the above to start Ngrok. Before you continue to run the application, you can copy the commands below to run the server.

ngrok http 5050

Copy the \https://e353facdfb71.ngrok-free.app/webhook\ URL and paste it in your Stream webhook event and configuration. Make sure you tick the box labeled “subscribe to all current and future events,” then click Submit to save the webhook, as shown in the screenshot below.

Then open another terminal and navigate to the Stream Chat folder from the root folder by pasting the command below:

cd rn-stream-chat

Copy the command below to start the emulator:

npx expo start \-c  

After running the command above, you will get a list of commands. Just reply with the letter ‘I’, and it will open the emulator window, where you will see the group chat members list.

The Chat List Screen

The chat list screen will display a list of available members. You can select a chat name or create a custom name to join the conversation in the group chat.

The Chat Screen

The chat screen is where the magic happens. This is where users can send and receive messages. Our AI moderator will be at work, removing spam messages before they’re delivered — as seen in the screenshot below.

Testing the AI Moderator

Once you have the application running, you can test the AI moderator by sending messages that violate the rules you’ve configured in your Stream dashboard. For example, you can try sending words from the \profanity\_2020\ list.

You should see that these messages are automatically blocked and not visible to other users in the chat, as shown in the short video below and in the Stream Moderation dashboard screenshot.

Below is a new test with two different iOS emulators.

Below is a screenshot from the chat explorer dashboard showing the allowed and blocked messages.

What’s Next?

You now have a fully functional React Native group chat that uses Stream to detect and block harmful or inappropriate messages in real time. The app securely connects users, allows seamless user switching without crashes, and loads older conversations smoothly as you scroll.

When building a chat app, it’s your responsibility to create a safe and respectful environment where your community can thrive. Without proper moderation, toxicity, spam, and harassment can quickly overwhelm online spaces, driving users away.

Stream’s AI Moderation API makes it easier for developers to scale communities responsibly by automatically detecting and filtering harmful content, reducing the burden on human moderators, and helping maintain a positive experience for all participants.

Check out the GitHub repository for the complete code.

In today’s world of data collection, intelligent AI agents are transforming how we approach data. Instead of using fixed scripts that stop working when a webpage changes, these agents can think, understand, and adjust, making data extraction stronger, more accurate, and smarter.

In this article, we examine how these ing paradigms and solutions, such as [OloStep](https://www.olostep.com/), are capitalizing on agents as they shift.

From Static Scrapers to Adaptive Agents

Traditional web scrapers are built on fixed rules and selectors. They expect pages to follow consistent HTML structures and often fail when sites update. Autonomous agents instead observe patterns, reason about structure, perform corrective actions, and recover from failures. They can decide when to click a “load more” button or when to scroll further. They can detect missing fields or empty responses and retry or choose an alternate path.
Because these agents act based on high-level goals rather than brittle scripts, they can survive changes in page layout and dynamic content more gracefully. They are more robust to JavaScript rendering, infinite scroll, asynchronous loading, and unexpected content shifts.

Why Web Scraping Matters for AI Agents

AI agents need fresh data. They cannot operate solely on training data or outdated snapshots. Scraped web data gives them the real-time sight and sound of the internet. It enables:

• Sensing changing prices, new product listings, or shifts in sentiment
• Validating assumptions with live examples
• Triggering downstream workflows or actions based on updated facts

Autonomous agents transform web scraping from a background utility into a primary component. The faster the data, the more intelligent the agent.

How OloStep Enables Smarter Agents

[OloStep](docs.olostep.com) presents itself as a unified web data API built for AI and research agents. It supports crawling, parsing, routing, automation, and workflows behind the scenes. It exposes features such as click and form fill, distributed infrastructure, and parsing engines. OloStep also maintains low latency, typically between two and six seconds, by combining optimized infrastructure and logic.

Additionally, OloStep enables agents to reason about which pages to crawl, which subpages to explore, and how to filter results. Agents using OloStep do not need to manage the plumbing of proxies, anti-blocking measures, or parser maintenance. The agent issues high-level commands, and OloStep handles extraction, cleaning, formatting, and delivery.

Because OloStep converts websites into structured outputs and enables agents to build higher-level workflows, it transforms the web into a real-time database usable by agents and applications. In effect, OloStep abstracts away the messy details, allowing agents to focus on reasoning and decision-making.

Key Capabilities of Autonomous Scraping Agents

Here are several capabilities that distinguish modern scraping agents:

1. Goal-driven workflows: Agents receive intents, such as “collect top 100 product listings for brand X,” and orchestrate multiple steps, including crawling, filtering, logging in, pagination, and error recovery.
2. Dynamic adaptation: When a page fails, when elements shift, agents can detect anomalies, adjust selectors, or pivot strategies rather than fail outright.
3. Self-improvement: Some agents can learn reusable “skills” (for example, how to navigate a site or extract tables) and refine their library over time. As agents accrue experience, they become faster and more accurate.
4. Collaborative human-agent control: In edge cases, agents may defer to human review or allow human override, improving reliability and trust.
5. Scalability across domains: Because agents reason rather than rely on hand-coded rules, they can scale across many websites, domains, languages, and content types.

Challenges and Ethical Considerations

Despite their promise, these agents face real challenges:

• Sites fight back. Measures like CAPTCHA, bot detection systems, and active blocking make scraping harder.
• Changes in site licensing, terms of service, or robot policies require agents to follow legal and ethical constraints.
• Agent hallucination or misbehavior risk exists when agents misinterpret content or take unintended actions.
• Infrastructure cost grows when agents operate at scale across many domains.
• Maintaining agent safety, monitoring, auditability and human oversight becomes essential.

Moreover, the arms race continues: as agents get smarter, websites will adopt more advanced defenses, including behavioral heuristics or dynamic content injection.

The Future Landscape

Looking ahead, autonomous agents will become a core layer of the web. The concept of a Web of Agents (where agents interact, collaborate, and request services from each other) is already being theorized in academia. In that world, web scraping capabilities will be internalized: agents will query agent services to fetch fresh data rather than building their own scrapers.

Hybrid systems will dominate. Agents will issue goals while robust scraping platforms like OloStep serve as the reliable data backbone. Agents will compose skills, reuse parsing modules, coordinate across domains, and even negotiate data contracts.

We will also see specialization. Some agents will focus on financial use cases, while others will concentrate on market monitoring, brand protection, compliance, or social listening. Each will lean on tailored scraping stacks behind the scenes.

As agents evolve, the boundary between crawling, searching, and reasoning will become increasingly blurred. The future is a world where agents not only read the web but also act on it.

How I picked these five

Selection criteria you care about:

1. Data quality (HTML → Markdown / clean text / JSONL)
2. LLM & vector-DB friendliness (embeddings pipeline, LangChain integrations, Pinecone/Weaviate exports)
3. Scalability & real-time (APIs, crawl scheduling, proxies)
4. Legal & ethical compliance (terms, takedown handling, human review where relevant)
5. Developer ergonomics (no-code UI, SDKs, output formats).

Here are five tools that make that dream real, and why Olostep should be the first thing you try.

Olostep — from URL to LLM-ready content, fast

If your priority is getting clean markdown or structured JSON directly from a URL, Olostep feels built for that last-mile moment before an LLM. It advertises simple endpoints that convert pages into Markdown, structured fields, and enables you to produce JSONL for embeddings without a full cleaning pipeline. For teams who treat web pages as content to be read (not raw HTML to parse), Olostep removes steps and friction. (Olostep)

Why use it: Minimal transformation work, output formats that match LLM ingestion patterns, and a developer-first API that can slot directly into an RAG or vector DB pipeline.

Spidra — no-code + LLM assistance for tricky pages

Spidra takes a different angle: a no-code interface that simulates real browser actions (click, scroll, fill), with LLM-driven extraction helpers for odd layouts. It’s great when you need human-like interaction to reveal content (lazy-loaded sections, gated lists) and want to export structured records without writing a spider. If you prefer GUI builds and occasional manual tuning, Spidra reduces the ramp. (spidra.io)

Why use it: Fast for one-off or low-code pipelines where human-style navigation matters.

Apify — the flexible platform for custom pipelines

Apify is the go-to when you need control: custom actors, scheduled crawls, and a marketplace of scrapers. It’s built to scale and to integrate with downstream systems — you can generate structured outputs, run post-processing, and push results to storage or an LLM pipeline. If your project grows from POC to product, Apify gives the building blocks. (Apify)

Why use it: Highly customizable, great for teams that expect to iterate on scraper logic and orchestration.

Bright Data — enterprise-grade scale and proxy coverage

When scale and reliability across multiple geo-targets matter, Bright Data’s integrated proxy and scraping offerings are hard to beat. They emphasize large proxy pools and managed scrapers, which help keep long-running crawls healthy and less likely to be blocked. For enterprise data collection to feed large LLMs, reliability matters. (Bright Data)

Why use it: Best choice for high-volume, geo-targeted extraction where anti-block and uptime are critical.

Zyte & ecosystem tools — LLM-friendly best practices

Zyte (and players in its orbit) have focused on integrating LLMs using AI-assisted selectors, QA, and RAG-style approaches for search-and-extract. If your scraping is research-heavy, tools from this space help with quality checks and smarter extraction logic — the kind that reduces garbage data entering your embeddings. (Zyte #1 Web Scraping Service)

Why use it: Strong on extraction quality, developer tooling, and best practices for LLM feeding.

How to choose (short checklist)

1. Output format: Need markdown or JSONL out-of-the-box? Prioritize Olostep or Apify.
2. Interaction: Do you need clicks or form fills? Spidra or Apify (browser automation).
3. Scale & geo: Big volume + geotargeting → Bright Data.
4. Quality & QA: If you need smart extraction and LLM-assisted checks, look at Zyte and Apify.
5. Compliance: Always respect robots.txt and site terms, rate-limit requests, and prefer ethically sourced proxies or managed platforms to reduce legal risk.

Why Olostep is the everyday choice for LLM builders

If your goal is “feed an LLM or a vector DB right now,” Olostep shines because it reduces transformation work: fewer cleaning steps, native markdown/JSON outputs, and a direct API path into embeddings or RAG pipelines. For fast experiments or production flows that demand consistent, LLM-ready records, it’s the pragmatic first stop.

Try Olostep for the initial ingest, and add Apify or Bright Data later as your scale or interaction needs grow. (Olostep)

However, as websites grew more dynamic, JavaScript-heavy, and protected by anti-bot firewalls, the simple approach began to fall apart. The modern web doesn’t just serve data anymore; it hides it behind authentication, React components, and captcha walls.

That’s why, in 2025, developers no longer build scrapers from scratch. They reach for scraping APIs, pre-built engines that handle rendering, rotation, and reliability. These services are the unsung heroes of the AI era, quietly feeding large language models, research dashboards, and e-commerce trackers with fresh, structured web data.

Let’s explore the Top 6 Web Scraping APIs shaping how developers extract, format, and scale web data today, and how each one compares on speed, formats, and developer experience.

Apify — The Automation Workhorse

If Olostep is elegance, Apify is endurance.

Apify has long been the backbone of web automation. Its “Actors” — prebuilt scraping scripts for popular sites — make it a dream for developers who want flexibility and scale. Need to crawl Amazon listings or extract real estate data? Someone’s already built that Actor.

Apify’s strength lies in community scripts, integrations, and workflow orchestration. You can connect crawlers to storage, queues, or even AI post-processors with minimal setup.

• Speed & Success Rate: Moderate to high, depending on the Actor setup.
• JS Handling: Uses headless browsers (Playwright, Puppeteer).
• Formats: JSON, CSV, HTML.
• Cost per 1,000 requests: Mid-range; pricing based on compute units.
• Developer Experience: Feature-rich, slightly steeper learning curve.

Apify remains the veteran choice for those who need customization over simplicity.

Spidra — The AI-Enhanced Scraper

If Olostep is precision, Spidra is instinct. Spidra is an AI-enhanced web crawler that dynamically crawls sites, rather than relying on hard-coded selectors. It combines a crawling, extraction, and summarization workflow, enabling you to access all products on the site,” a more efficient approach.

Its machine-learning-powered “auto-extract” feature identifies meaningful sections (titles, prices, content blocks) automatically. Developers love its “crawl intent” model; you tell it what you want, and it determines the best way to obtain it.

• Speed: 8.5/10 — adaptive crawling
• Formats: JSON, structured tables
• JS Handling: Yes, AI-controlled browser sessions
• Cost per 1,000 requests: Competitive mid-range
• Best for: AI-driven extraction, e-commerce, and dynamic sites

Spidra feels like the bridge between human intuition and automation — a scraper that thinks before it scrapes.

ZenRows — The Developer’s Sweet Spot

ZenRows balances flexibility and affordability. It handles JS rendering, proxy rotation, and retries, all behind a single API endpoint.
It’s ideal for developers who want predictable performance without the overhead of enterprise solutions.

• Speed: 8.5/10
• Formats: HTML, JSON
• JS Handling: Yes
• Cost per 1,000 requests: Mid-range
• Best for: Mid-scale scraping operations

ZenRows is the steady workhorse, not flashy, but dependable.

Olostep — The Markdown-First and AI-Ready Scraper

Imagine if you could turn any website into clean, AI-digestible text in seconds; that’s Olostep’s superpower.

Olostep is a newcomer that feels like it was built for 2025. Instead of just spitting out HTML, it provides structured Markdown, clean JSON, and even rendered HTML formats that plug directly into AI pipelines and content automation tools.

Where others focus on “scraping,” Olostep focuses on data readiness. Its APIs are designed to feed LLMs, not just databases. Developers can send a single URL or upload a batch of thousands, and within minutes, receive fully parsed, JS-rendered results.

• Speed & Success Rate: Near-instant for static pages, and surprisingly quick for JavaScript-heavy sites.
• JS Handling: Full browser rendering with bot-evasion techniques.
• Formats: HTML, Markdown, JSON, PDF.
• Cost per 1,000 requests: Low — with generous batch pricing and free tier credits.
• Developer Experience: Lightweight REST API, clean docs, no SDK bloat.

Olostep feels less like a scraping tool and more like a data pipeline engine, ready for anyone building AI agents, automation bots, or analytical dashboards.

ScraperAPI — Simplicity That Scales

ScraperAPI’s appeal is its plug-and-play simplicity. You make an API call, it returns a fully rendered page, no proxy management, no rotating IP headaches.
Perfect for teams that want something that “just works.”

• Speed: 9/10
• Formats: HTML, JSON
• JS Handling: Yes
• Cost per 1,000 requests: Low to moderate
• Best for: Quick data pulls and prototypes

ScraperAPI is the fast food of scraping — simple, reliable, and always open.

Bright Data — The Enterprise Giant

Bright Data operates on a different scale, think billions of requests per day across global proxy networks.
It’s built for enterprises that can’t afford downtime and need access to the full spectrum of the web.

It offers managed scraping, proxy rotation, data labeling, and even AI model integration. But it comes at a premium price and complexity level.

• Speed: 10/10
• Formats: JSON, CSV
• JS Handling: Industry-leading
• Cost per 1,000 requests: High (enterprise-grade)
• Best for: Large organizations & high-volume data feeds

Bright Data is the aircraft carrier of the scraping world — powerful but not agile.

The Future Belongs to Intelligence, Not Just Access

All these tools scrape the web, but only a few understand it; that’s the new line dividing old-school crawlers from AI-ready data platforms.

Olostep represents that new wave of scrapers built not just to fetch pages, but to interpret them, structure them, and feed them into intelligent systems.

So if 2023 was about scraping the web, 2025 is about teaching your scraper to think.

Then it hits you that the data is all out there, it’s just trapped inside messy webpages. That’s precisely what OloStep is built to fix. Instead of scraping, crawling, or building static indexes, OloStep flips the model. It treats the open web itself as a live, structured, and queryable database in real time.

The Web Wasn’t Meant to Be Queried

The internet is one of the most extensive and valuable datasets in history. However, it was not designed for structure, but it was created for publishing.

There are billions of pages, including blogs, product listings, pricing tables, and job boards, which are constantly being updated. Yet, querying this information in a traditional manner is nearly impossible. You can crawl, index, and scrape the web, but these methods are often brittle, slow, and quickly become outdated.

What if, instead, you could treat the open web as if it were your live backend?

What is Olostep?

Olostep is a powerful web scraping API that efficiently provides e data from any website. It meets the essential demands for fast, reliable, and cost-effective data acquisition, AI development and large-scale data aggregation. rtups, and established companies enhance their data-driven applications and automate workflows.

The Core Idea: Query the Open Web Like a Database

Think SQL, but the tables are web pages. Think APIs, but the endpoints are search queries. OloStep turns the open web into a real-time knowledge source structured, filterable, and directly useful for agents, workflows, and apps.

Let’s say you’re building an AI agent that finds pricing trends for B2B SaaS tools. Traditionally, you’d:

• Crawl pages
• Parse them manually
• Store them
• Run queries later

But with OloStep:

SELECT price, plan_name, vendor
FROM web
WHERE category = "CRM software" AND updated_within = "7 days"

It’s not a stretch. This is the direction we’re heading to obtain structured results directly from the public internet, without scraping or even relying on pipelines, just answers.

What Makes OloStep Different

Most tools that extract data from the web fall into one of two buckets:

1. Scrapers: Fast but fragile. Break often, need constant maintenance.
2. Knowledge graphs: Structured but stale. Require time-consuming ingestion pipelines.

OloStep is different because it utilizes AI-native tools to extract structured data pages in real-time. The model understands both layout and meaning. It can:

• Recognize product specs on landing pages
• Pull out job listings across multiple sites
• Extract facts from articles or documentation

It’s a web-native query engine. No scraping rules. No brittle XPath selectors. If you’re building anything that depends on real-world knowledge, you don’t want to be stuck waiting for someone else to scrape, ingest, and update it for you.

Why Now? Because the Stack Just Clicked

This wasn’t possible five years ago.

• LLMs were too weak.
• Scrapers were too brittle.
• Knowledge graphs were too slow.

However, now LLMs can comprehend, reason over tables, and infer meaning across web pages in real-time.

OloStep rides that wave, turning raw web content into structured, semantically rich data at the moment you need it. It’s not just search. It’s understanding.

Use Cases Already Emerging

Below are the benefits of implementing OloStep:

• AI Research Agents: agents that answer questions by citing live data from across the web, not just summaries from cached pages.
• Competitive Intelligence: pull pricing changes, product launches, or staffing shifts from public pages and career portals.
• Custom Tools & Dashboards: develop internal and public-facing information, such as public tenders, vendor updates, or compliance announcements.
• Programmatic Search: replace brittle Google scraping with structured queries and filters. No hacks. No captchas. Just answers.

The Future: Don’t Store the Web. Ask It.

OloStep flips the traditional methods so instead of trying to tame the chaos of the internet into a static database, it embraces the mess and makes it readable, understandable, and usable.

Think of it like this:

• Search gives you links.
• Scraping gives you fragments.
• OloStep provides you with facts, and it does so in real-time.

So instead of building brittle scraping tools or waiting for someone to publish a dataset, you can now ask the web a question and get structured, filtered, useful data in return.

The web is no longer just something you read; it’s something you query.

Try It Yourself

If you’re building agents, dashboards, or any product that depends on fresh, structured information from the real world, OloStep gives you the edge.

Check it out at https://olostep.com
Start asking the web real questions and get real answers.

With OloStep, you don’t have to.

Sometimes, you want to know immediately when a user logs into your app, whether it’s for security reasons, user activity monitoring, or simply keeping track of engagement. Real-time login alerts can help you detect unauthorized access attempts or provide useful analytics.

In this tutorial, we’ll show you how to send real-time SMS alerts using Twilio whenever a user signs in. You’ll build this system using [Next.js](https://nextjs.org/docs), [Strapi](https://strapi.io), and [Twilio](https://console.twilio.com/), and you can test it locally using ngrok.

* Next.js will serve as the frontend framework, offering a fast, modern, and highly customizable interface for your app’s login screen. It supports API routes out of the box, which simplifies handling backend logic in a unified codebase.

* Strapi, a powerful open-source headless CMS, will manage user authentication and act as the backend. It provides a ready-to-use user model and robust authentication system with minimal setup, saving you time and letting you focus on core functionality.

* Twilio will handle the real-time SMS delivery. Once a user logs in, your application will use Twilio’s programmable messaging API to send a notification that includes the user’s email, IP address, and login timestamp.

Prerequisites

- Node.js and npm installed
- Basic Next.js app set up
- Twilio account with verified phone number
- Ngrok installed (for optional webhook testing)
- Strapi instance running (locally or hosted)
- SendGrid: for sending the emails

Creating a new Next.js project

To get started with the article, let’s create a new Next.js project using the command below.

npx create-next-app@latest
```

Running the application above will prompt you to configure the project. Then configure it as shown below.

What is your project named? my-app
Would you like to use TypeScript? No / Yes
Would you like to use ESLint? No / Yes
Would you like to use Tailwind CSS? No / Yes
Would you like your code inside a `src/` directory? No / Yes
Would you like to use App Router? (recommended) No / Yes
Would you like to use Turbopack for `next dev`? No / Yes
Would you like to customize the import alias (`@/*` by default)? No / Yes
What import alias would you like configured? @/*
```

Creating the project structure

The project structure includes a `.env` file at the root, which will be used to store Twilio credentials securely. Inside the pages directory, there’s an api folder containing notify-login.js, which serves as the backend API route to send SMS alerts using the Twilio SDK. This function reads credentials from the environment and sends a login notification to a specified phone number.
The `test-login.js` page provides a simple frontend with a button that triggers this API, simulating a user login and prompting an SMS alert. The `index.js` file serves as the homepage, linking to the test login page.

This setup allows you to simulate and verify SMS alerts during user logins, laying the groundwork for integration with real authentication flows later. Running the app with. and visiting `/test-login` triggers the alert system, confirming server locally, up correctly.

Setting up Twilio

Next, log in to your Twilio Console to retrieve your Account SID, Auth Token, and Twilio phone number, as you can see in the screenshot below.

Then, use them to replace `<your_twilio_account_sid>`, `<your_twilio_auth_token>`, and `<your_twilio_phone_number>`, respectively, in `.env`.
This information is necessary because it authenticates your application with Twilio’s API, allowing it to send and receive SMS messages securely. The Twilio phone number serves as your application’s sender ID, allowing with real mobile eive message status updates viawebhooks.

Create the environment variables

To keep your API keys and other important information safe, use environment variables in a `.env file`. Start by creating a .env file in your your Strapi folder and adding the keys you need there.

APP_KEYS=key1,key2,key3,key4
API_TOKEN_SALT=replace_me
ADMIN_JWT_SECRET=replace_me
JWT_SECRET=replace_me
EMAIL_SMTP_HOST=smtp.sendgrid.net
EMAIL_SMTP_PORT=587
EMAIL_SMTP_SECURE=false
EMAIL_SMTP_USER=apikey
EMAIL_SMTP_PASS=your_sendgrid_api_key
EMAIL_DEFAULT_FROM=alerts@yourdomain.com
EMAIL_DEFAULT_REPLY_TO=alerts@yourdomain.com
ALERT_RECIPIENT_EMAIL=alerts.receiver@example.com
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_FROM=+1234567890
ALERT_RECIPIENT_PHONE=+234XXXXXXXXXX
HOST=0.0.0.0
PORT=1337

These environment variables help your app work with Twilio and Strapi. `TWILIO_ACCOUNT_SID` and `TWILIO_AUTH_TOKEN` are similar to your Twilio username and password; they enable your app to send messages through Twilio. `TWILIO_PHONE_NUMBER` is the Twilio number that sends the SMS, and `ALERT_RECIPIENT_NUMBER` is the phone number that will receive the login alert. `NEXT_PUBLIC_STRAPI_URL` is the web address of your Strapi backend, which your Next.js app uses to connect to Strapi and check login details.

Installing Twilio SDK

To enable the application to send SMS to users, we need to install the Twilio SDK into the project. To do that, run the command below.

npm install Twilio

Setting up Strapi to Send Email Alerts

To complement your SMS alerts, you’ll also send email notifications using Strapi. This is useful for backup notifications or sending alerts to multiple recipients.

Strapi Installation and Setup

Create a folder where both your Strapi and Next.js app will reside. Give it any name. Now navigate to the folder and run the command below in your terminal. Ensure you are in the directory where you want to keep both your Strapi backend and Next.js frontend.

npx create-strapi@latest

The command above will display the details below:

? What is the name of your project? strapi-backend
…
? Please log in or sign up. Skip
? Do you want to use the default database (sqlite) ? Yes
? Start with an example structure & data? No
? Start with Typescript? Yes
? Install dependencies with npm? Yes
? Initialize a git repository? No

Once you have successfully installed Strapi, cd into your strapi project and run the command below.

npm run develop

This should start up your Strapi application in the URL: http://localhost:1337/admin. Follow the instructions on the screen to continue.

This is what your Strapi dashboard should look like after registering the new admin:

Enable the Email Plugin in Strapi

Here, we are going to install the Strapi email plugin, run the command below in your terminal:

npm install @strapi/plugin-email

Setting up Strapi Email plugin with SendGrid

First, you need the email plugin and the SendGrid provider for Strapi.
In your Strapi project directory, run:

npm install @strapi/plugin-email @strapi/provider-email-sendgrid

Or if you use yarn, run:

yarn add @strapi/plugin-email @strapi/provider-email-sendgrid

For the sake of time and keeping the tutorial simple, click here for a full step-by-step guide to set up Strapi email plugin with SendGrid on the Strapi blog.

Creating the API route to send SMS

Now let’s create the part of your frontend project (built with Next.js) that sends the SMS. We’ll do this by adding a file called `notify-login.js` inside the `pages/api` folder.

This file will receive login details (like the user’s email, IP address, and time of login) through a `POST` request, and then use Twilio to send a text message with that info.

To do this, open your **Next.js project folder**, go to the `pages/api` folder, and create a new file named `notify-login.js`. This file will handle sending the SMS alert whenever a user logs in.

// pages/api/notify-login.js
import twilio from 'twilio';
const client = twilio(
 process.env.TWILIO_ACCOUNT_SID,
 process.env.TWILIO_AUTH_TOKEN
);
export default async function handler(req, res) {
 if (req.method !== 'POST') {
 return res.status(405).json({ message: 'Method Not Allowed' });
 }
const { email, ip } = req.body;
if (!email || !ip) {
 return res.status(400).json({ message: 'Missing email or IP address.' });
 }
try {
 await client.messages.create({
 body: `🔐 Login Alert: ${email} logged in from IP ${ip}`,
 from: process.env.TWILIO_PHONE_NUMBER,
 to: process.env.ALERT_RECIPIENT_NUMBER,
 });
res.status(200).json({ message: 'Alert sent!' });
 } catch (error) {
 console.error('Twilio Error:', error);
 res.status(500).json({ message: 'Failed to send alert.' });
 }
}

The code above sends a login alert via Twilio SMS when a POST request with `email` and `ip` is received. It uses environment variables for Twilio credentials and sends the alert to the user’s phone number, handling errors and invalid requests appropriately.

The Frontend design

Next, you will create `test-login.js` Page to simulate a login action by calling the API.

Copy the code below and paste it into your *test-login.js* file:

// pages/api/notify-login.js
import twilio from 'twilio';
const client = twilio(
 process.env.TWILIO_ACCOUNT_SID,
 process.env.TWILIO_AUTH_TOKEN
);
export default async function handler(req, res) {
 if (req.method !== 'POST') {
 return res.status(405).json({ message: 'Method Not Allowed' });
 }
const { email, ip } = req.body;
 if (!email || !ip) {
 return res.status(400).json({ message: 'Missing email or IP address.' });
 }
try {
 // Step 1: Send SMS via Twilio
 await client.messages.create({
 body: `🔐 Login Alert: ${email} logged in from IP ${ip}`,
 from: process.env.TWILIO_PHONE_NUMBER,
 to: process.env.ALERT_RECIPIENT_NUMBER,
 });
// Step 2: Send Email via Strapi
 await fetch(`${process.env.NEXT_PUBLIC_STRAPI_URL}/api/notify-login-email`, {
 method: 'POST',
 headers: { 'Content-Type': 'application/json' },
 body: JSON.stringify({ email, ip }),
 });
res.status(200).json({ message: 'Alert sent via SMS and Email!' });
 } catch (error) {
 console.error('Alert Error:', error);
 res.status(500).json({ message: 'Failed to send alert.' });
 }
}

The code above defines a simple Next.js page that simulates a user login and triggers an SMS alert using the `/api/notify-login` endpoint. It includes a form where users can enter their email, fetches the user’s public IP using the api-pify, and sends both the email and IP to the backend API via a POST request.

The next step is the main Page, known as the index.js page, where a placeholder or real login page code would go. You can copy the code below and paste it into your index.js file:

// pages/index.js
import { useState } from 'react';
import axios from 'axios';
export default function Home() {
 const [email, setEmail] = useState('');
 const [status, setStatus] = useState('');
 const [loading, setLoading] = useState(false);
const getUserIP = async () => {
 try {
 const res = await axios.get('https://api.ipify.org?format=json');
 return res.data.ip;
 } catch (error) {
 console.error('IP Fetch Error:', error);
 return 'Unknown IP';
 }
 };
const handleSubmit = async (e) => {
 e.preventDefault();
 setStatus('');
 setLoading(true);
const ip = await getUserIP();
try {
 const res = await axios.post('/api/notify-login', { email, ip });
 setStatus(res.data.message);
 } catch (err) {
 setStatus('Error sending alert.');
 }
setLoading(false);
 };
return (
 <main style={{ maxWidth: '400px', margin: '100px auto', textAlign: 'center' }}>
 <h1> Login Portal</h1>
 <form onSubmit={handleSubmit} style={{ display: 'flex', flexDirection: 'column', gap: '10px'}}>
 <input
 type= "email"
 placeholder= "Enter your email"
 value={email}
 onChange={(e) => setEmail(e.target.value)}
 required
 style={{ padding: '10px' }}
 />
 <button type="submit" disabled={loading} style={{ padding: '10px' }}>
 {loading ? 'Logging in…': 'Login'}
 </button>
 </form>
 {status && <p style={{ marginTop: '20px' }}>{status}</p>}
 </main>
 );
}

The code above defines a simple login interface in a Next.js page that collects a user’s email, fetches their public IP address using the ipify API, and sends this data to the /api/notify-login endpoint via a POST request using Axios. It includes basic form handling with loading and status states, updating the UI to reflect success or error messages after submission. The UI consists of a centered form with an email input and a submit button, styled inline for simplicity.

The External css

Next, go to your public folder and create a file named style.css, then copy the code below and paste it into the style.css file.

/* public/styles.css */
body {
 font-family: Arial, sans-serif;
 background: #f8f8f8;
 margin: 0;
 padding: 40px;
}
.container {
 max-width: 400px;
 margin: auto;
 background: white;
 padding: 30px;
 border-radius: 8px;
 box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
}
input[type="email"] {
 width: 100%;
 padding: 12px;
 margin-bottom: 15px;
 border-radius: 4px;
 border: 1px solid #ccc;
}
button {
 width: 100%;
 padding: 12px;
 background-color: #0070f3;
 color: white;
 border: none;
 border-radius: 4px;
 font-weight: bold;
 cursor: pointer;
}
button:hover {
 background-color: #005bb5;
}
.status {
 margin-top: 20px;
 font-weight: bold;
}

The CSS code above styles a clean, centered login form with modern input fields, a bold submit button, and a status message section.

Testing the application

The next step is to run the simple command below in your terminal to start the application:

npm run dev

After running the command above, you should see something like the image below displayed in your terminal.

Ngrok setup

Your Next.js app runs at http://localhost:3000, but Twilio can’t access that because it’s on your local machine. ngrok gives you a public URL that tunnels to your local server. This is essential for testing Twilio SMS alerts in real time.

In a new terminal tab, run the command below.

ngrok http 3000

This URL is now a secure tunnel to your local server, you’ll get output as shown in the screenshot below:

Add ngrok URL to Twilio Webhook

Once ngrok is running and you’ve obtained a public HTTPS URL (e.g., https://abcd1234.ngrok.io), you’ll need to link it to your Twilio number so that incoming messages or events from Twilio are forwarded to your local development server.

To link your ngrok tunnel to a Twilio SMS webhook, start by visiting the Twilio Console and navigating to the Phone Numbers section. Click on the phone number you want to configure. Once you’re on the settings page for that number, scroll down to the Messaging section. Look for the field labeled A Message Comes In. This is where you provide the URL that Twilio will call when a new message is received.

In that field, paste your ngrok URL pointing to your local API route, for example: `https://abcd1234.ngrok.io/api/notify-login`. Be sure to replace `abcd1234` with the actual subdomain that ngrok assigned to your session. Next, select HTTP POST as the request method. Once done, scroll down and click Save to apply the changes. Twilio will now send incoming SMS data to your local server via the ngrok tunnel.

Image description can be seen below:

Next, visit http://localhost:3000/. Then, type in your email address and click on the login button. The localhost page should be displayed as shown in the image below.

After you click on the login button, an sms with your information, which includes your IP address and email address, will be sent to your Twilio-registered number, just as shown in the screenshots below.

The screenshot above shows the sms alert being received from Twilio with the login information.

The image above shows that the login details were received in email from Strapi.

That is How to Build Real-Time User Login Alerts system with Twilio + Next.js

You’ve now built a complete real-time login alert system using Twilio and Next.js, all from scratch. You created a custom API route to send SMS notifications, designed a test login page, and used an external CSS file for styling. With ngrok, you expose your local server to the internet, allowing Twilio to send requests directly to your API. You also learned how to link your ngrok URL to a Twilio webhook for real-time testing. With everything connected and working, you now have a solid foundation for monitoring user activity and expanding into full authentication systems. You can find the complete source code and setup instructions on my [GitHub repository](https://github.com/Davidfagb/-User-Login-Alerts-system-with-Twilio-Next.js-and-Strapi).

As developers and teams build more complex applications, one of the toughest parts they deal with is managing access and permissions. With more use of cloud services, APIs, and mobile apps, ensuring only authorized personnel can access these systems has become a big challenge.

The need for secure and scalable solutions is growing, and users expect a smooth experience when logging into apps, while companies want to protect sensitive data and avoid security risks. This has led to a demand for better ways to handle authentication and authorization, especially as systems get more distributed and user access becomes more varied.

The Traditional Approach to Authentication

Authentication is the process of verifying a user’s identity before allowing them access to an application. In the past, developers used a few standard methods to handle authentication.

Let’s look at three of the traditional approaches, which are Basic Authentication, Session-based Authentication, and Cookie-based.

Basic Authentication

In this method, when the user is ready to perform an action that requires the server, they send their username and password every time. The server will receive the credentials, username, and password, check them, and if they are good, will allow the user access. This method has clear and significant security issues. The credentials are sent as plain text, and unauthorized parties can intercept the credentials with a basic sniffer, allowing them access to the account and perpetrating fraudulent activity with little effort.

In addition to the security vulnerabilities, this method does not scale for larger systems and provides a terrible user experience. Requiring the user to log in repeatedly is annoying and is an unnecessary barrier to authentication.

Session-based Authentication

When the user logs in, the system creates a session. The server stores the user’s session information and sends the session ID to the user’s browser.. Each request uses this ID to verify the user. While it’s more secure than Basic Authentication, it can put a strain on server resources as more users join, and it’s still vulnerable to session hijacking if a malicious actor steals the session ID. Additionally, sessions eventually expire, forcing users to log in again, which can be frustrating.

Cookie-based

This is similar to session-based, but instead of storing session data on the server, it stores a session ID or token in a cookie on the user’s browser. The browser sends the cookie with each authentication request. While it can reduce server load, it’s still vulnerable to attacks like cross-site scripting (XSS), where a malicious script steals the cookie. Cookies also expire or can be deleted, causing users to log in again unexpectedly.

OAuth: The Evolution of Authentication

OAuth is a solution to the problems of managing and securing user authentication. It is a protocol that allows users to grant applications access to their data without sharing their passwords. Instead of directly handing over sensitive login credentials, OAuth uses tokens to enable secure access.

Here’s a quick look at how OAuth works:

• Authorization Code Flow: This is the most common flow for web applications. The user logs in to an identity provider like Google, and the application gets an authorization code. This code is then exchanged for a token, which the app uses to access the user’s data.
• Implicit Flow: This flow is used for client-side applications, like mobile apps or single-page web apps. The user logs in, and instead of a code, the application directly receives a token. This flow is faster but less secure than the Authorization Code Flow.
• Client Credentials Flow: This flow is used when applications need to access their own data, not on behalf of a user. The application itself authenticates using its client ID and secret, and gets a token to access resources.

Why OAuth?

OAuth simplifies the authentication process by replacing passwords with tokens, making it more secure and easier to scale across different services and devices.

Below are the reasons OAuth is a better alternative compared to the traditional methods:

• Security: OAuth minimizes risk by keeping credentials private. Instead of sharing passwords, it uses tokens like JWT to authenticate users, making it scalable and secure.
• User Experience: OAuth makes logging in easier by allowing users to authenticate without passwords, such as using their Google or Facebook account.
• Scalability: OAuth supports scalable systems, especially for APIs and microservices. It ensures secure access even as systems grow and become more complex.

OAuth vs. Traditional Authentication

Traditional authentication methods like session-based or cookie-based authentication often come with several drawbacks. OAuth, on the other hand, was created to address these issues, especially in scenarios where secure, scalable, and seamless user authentication is needed.

Below is a comparison between OAuth and traditional authentication methods, highlighting how OAuth solves everyday problems and the trade-offs involved.

How OAuth Fits into the Modern Architecture

OAuth plays a key role in modern architectures, particularly in microservices, by managing secure access between services without exposing sensitive credentials. It allows each service to request specific permissions, ensuring only authorized interactions.

For third-party API integrations, OAuth enables apps to securely access external services, like Google or Facebook, without needing users’ passwords.

In mobile app development, OAuth allows users to log in using trusted providers, such as Google or Apple, which enhances the user experience and eliminates the need for users to create new accounts. OAuth also works with Identity Providers (IDPs) to authenticate users and issue access tokens. OpenID Connect, an identity layer built on top of OAuth, adds functionality by providing user identity information, allowing apps to verify both the user’s identity and their access permissions.

Limitations and Considerations

While OAuth is a powerful solution, it does come with some challenges that developers need to consider, which include:

Complexity for Small Teams

Implementing OAuth can be complicated for smaller teams or solo developers. The setup often requires integrating with multiple services, configuring redirect URIs, and handling token management, which can be time-consuming and error-prone.

Possible Challenges

Token expiration means that tokens have a limited lifespan, requiring them to be refreshed periodically, which adds extra steps to the authentication process. Token theft can occur if tokens are not stored or transmitted securely, allowing attackers to steal and misuse them.

Additionally, OAuth has potential attack vectors like Cross-Site Request Forgery (CSRF) and replay attacks, which can happen if proper security measures are not in place.

Learning Curve

OAuth involves a steep learning curve, especially when managing different types of token access, refresh, ID tokens and dealing with various flows. Understanding and handling token expiration, revocation, and securing token storage can be overwhelming for developers new to OAuth.

Conclusion

Using OAuth can be complicated. It requires managing the duration of tokens and keeping them safe. Developers must also work with different OAuth methods, which can be tricky, especially for smaller teams or those new to OAuth. Even with these challenges, OAuth is still a popular choice for modern apps because it offers secure and scalable authentication.

To stay ahead in securing applications, developers and teams should adopt OAuth and keep up with its evolving capabilities.

Most developers prefer Python over other programming languages due to its functionality and add-ons. Thanks to advancements in the gaming development industry, Python programming is an excellent choice for rapid prototyping video games.

This article will teach you about the top Python frameworks for your next game development adventure.

Python frameworks for game development

Below are the top Python frameworks you need to know to hasten and help you in the production of your Python games:

PyGame

PyGame is a Python open-source library for creating applications and products, such as games.sing the excellent library created using C, Python, Native, and OpenGL. Pygame enables users to create games and multimedia apps. This library is portable and can operate on any operating system and platform.

PyGame Basic Features:

Below are features that make PyGame a sound library for game development

• This library readily accommodates multicore CPUs.
• For key functionality, it employs efficient C and Assembles code.
• This library is lightweight and portable.

PyKyra

The PyKyra framework, which is used for game development, is the quickest Python framework. It is built on software and documentation localization (SDL) and functionalities. This Python language framework supports MPEG video and audio formats V, Ogg Vorbis, and many more.

PyKyra Basic Features:

Below are simple PyKyra features:

• Pykyra is a framework designed to handle both side symmetric and top-down translations.
• In Pykyra, the user may divide the screen into sub-window views, each with its object transformation.
• The Kyra engine is quick and competent, with specially designed algorithms for rendering updates.

PyGlet

Pyglet is a robust Python library that serves as an open-source, cross-platform windowing and multimedia framework. This library is used for game creation and many other aesthetically pleasing apps and software for Windows, Linux, and Mac OS X. Pyglet is a Python language module that enables user interfaces for event handling, OpenGL graphics, joysticks, loading films and images, windowing, and playing sounds and music.

PyGlet Features:

Below are some incredible reasons to use the PyGlet library for your game development.

• It has native portable windowing.
• Load practically any picture, sound, audio, or video format.
• The Pyglet library does not require any installation and has no external dependencies.

PyOpenGL

PyOpenGL is the most widely used cross-platform Python library. This framework connects Python to OpenGL and other APIs. The binding in the PyOpenGL framework was created by utilizing the standard c —types library. This is a Python-standardized framework that contains a large variety of external graphical user interfaces and audio libraries, such as PyGame, Raw Extended Library(XLib), Python binding for Qt( PyQt), and others.

PyOpenGL Basic Features

It has array handling, which includes:

• Integers (functioning as pointers-to-single-value arrays) (acting as pointers-to-single-value arrays).
• arrays of ctypes
• parameters for ctypes

Python Kivy

The Kivy library is built with new user interfaces, such as multi-touch apps. The Kivy Library is used for quick application and software development. This framework is compatible with Windows, Android, Raspberry Pi, Linux, Mac OS X, and iOS. Many inputs, devices, and protocols, such as the Mac OS X Trackpad and Magic Mouse, WM Touch, WM Pen, Linux kernel HID, and TUIO, are automatically supported. Kivy is a graphics processing unit (GPU)-accelerated library. It also includes over 20 widgets, all of which are very expandable.

The Kivy library is free under an MIT license beginning with version 1.7.2 and LGPL 3 for older versions. This reliable framework comes with a well-documented API and a programming guide to help users get started.

Kivy Features

Kivy provides access to mobile APIs for manipulating things like a phone’s camera, GPS tracking, and vibrator, and it also includes numerous components for developing applications, such as:

• A graphics library based on OpenGL ES 2.
• A wide variety of widgets that enable multi-touch.

Python-Ogre

Python — Orge, sometimes known as PyOrge, is a game development framework. It is developed in C++ with Python bindings for the Orge 3D engine. This framework is cross-platform, with flexibility and speed. It offers an excellent feature set that is leveraged to create engaging games. The PyOrge framework comprises two libraries: Orge3D and Crazy Eddie Graphical User Interface (CEGUI) systems.

The Orge is used to design scenery and any graphics imported into the game; for versions built before Orge v1.05, the PyOrge framework is utilized. Following that, numerous additional modules become accessible.

Python-Orge Basic Features

• Graphics user interface library that is compact and lightweight.
• A cross-platform audio API that is mainly used with the Ogre3D.
• OGRE has an input library that supports buffering and object-oriented programming.

Soya3D

Soya 3D is a Python high-level framework for creating 3D video games. This framework is free and licensed under the GNU General Public License ( GPL ). According to its framework, it is a multi-platform game creation engine that can operate on Linux, Microsoft Windows, and Oya 2Dis. It is entirely focused on speedy creation. It is pretty simple to use. The primary purpose of this framework is to enable novice users to create complex 3D games entirely in Python.

Soya3D Basic Features

• It supports sound rendering, input, and physical simulation.
• All networking is done using basic Python APIs.
• Soya has a comprehensive user guide detailing its design, functionality, and interfaces.

Conclusion

Currently, gaming has become an active part of our everyday lives, and improvements in the gaming sector have driven developers to create high-quality, visually appealing games. In this article, you will see the top Python frameworks for game development and their distinct capabilities that are beneficial for generating various functions.

This article explores how Kestra is not just keeping pace with the rapid changes in workflow orchestration but is leading the way. We’ll examine the key innovations that set Kestra apart from other tools, discuss how it addresses the current challenges in the field, and look ahead at the future trends in orchestration that Kestra is poised to influence.

By the end of this article, you’ll clearly understand why Kestra is positioned as a leader in the future of workflow orchestration.

Overview of Workflow Orchestration

Workflow orchestration has become a critical component in modern software development, enabling the seamless automation and management of complex processes across various systems. As businesses continue to scale and diversify their operations, the need for robust orchestration tools has never been greater. These tools ensure that tasks are executed in the correct order, dependencies are managed, and errors are handled efficiently, all while maintaining high levels of performance and reliability.

Getting Started with Kestra

Kestra is an innovative open-source orchestration tool designed to meet the evolving demands of today’s complex workflows. Unlike traditional tools that often struggle with scalability and real-time processing, Kestra offers a modern solution that is both powerful and user-friendly. With its ability to integrate with a wide range of technologies, Kestra provides a flexible platform for automating and orchestrating workflows at any scale. Whether you’re managing simple tasks or orchestrating intricate, multi-step processes, Kestra is engineered to handle the challenges of modern infrastructure.

Kestra Installation

Kestra is open source, which means anyone can run it on their machine for free. To set it up, ensure that you have Docker installed and running on your machine, then run the following command to start up your instance!”

```bash

docker run — pull=always — rm -it -p 8080:8080 — user=root \

-v /var/run/docker.sock:/var/run/docker.sock \

-v local server at /tmp:/tmp kestra/kestra:latest

```

Once you’ve run the above code, you can open your browser and navigate to https://localhost:8080 to see the interface and begin developing workflows!

Create Your First Flow

Head to the Flows section on the left side, then click the ‘Create’ button. Copy the lines of code below and paste them to create your initial flow.

id: myflow
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello World! 🚀

Make sure to click “Save” and then “Execute” to start your first execution.

The Current Landscape of Workflow Orchestration

The landscape of workflow orchestration has been shaped by several widely-used tools, each with its strengths and limitations. Apache Airflow, one of the most popular orchestration tools, offers a mature and extensible platform that has been adopted by many organizations for managing complex workflows. However, Airflow’s reliance on Python and its somewhat cumbersome setup can pose challenges for teams lacking deep expertise in programming or DevOps. Additionally, Airflow struggles with real-time processing, as its scheduling model is more suited to batch-oriented tasks.

These limitations highlight the need for more advanced solutions that can handle the growing complexity and scale of modern workflows being accessible to a broader range of users.

Emerging Needs in Workflow Orchestration

As organizations embrace digital transformation, the demand for sophisticated workflow orchestration tools has surged. Modern applications often require orchestrating workflows that span multiple systems, cloud environments, and data sources. These workflows must be flexible and scalable, capable of adapting to dynamic business needs while maintaining high levels of reliability and performance.

Real-time processing has also become increasingly important, particularly in finance, healthcare, and e-commerce industries, where timely data processing can be critical. Traditional batch-processing models are no longer sufficient in these contexts, driving the need for orchestration tools to handle real-time events and data streams.

These emerging needs underscore the importance of innovation in workflow orchestration. Tools that can meet these demands will be crucial in helping organizations maintain a competitive edge in an increasingly complex and fast-paced digital landscape, and that is where Kestra comes in to save the day.

How Kestra Is Addressing Modern Orchestration Challenges

Kestra is designed to overcome the limitations of traditional workflow orchestration tools by offering a suite of advanced features tailored to the demands of modern applications. One of Kestra’s standout capabilities is its scalability. Built with a distributed architecture, Kestra can quickly scale across multiple nodes, efficiently handling large, complex workflows. This scalability is crucial for enterprises managing workflows across vast datasets, numerous microservices, or geographically distributed systems.

Kestra also excels in real-time processing. Integration is another area where Kestra shines. It supports various plugins and connectors, allowing seamless integration with various data sources, cloud services, and third-party applications. This flexibility enables organizations to create workflows that interact with their entire tech stack, from databases and APIs to cloud storage and machine learning models.

User-Friendly Design and Usability

One of the barriers to adopting powerful orchestration tools is often their complexity. Kestra addresses this challenge by offering a user-friendly interface and a design that prioritizes ease of use. The platform is built with a declarative approach, where users can define workflows using a simple, YAML-based syntax. This reduces the need for deep programming knowledge, allowing a broader range of users, from data engineers to operations teams, to create and manage workflows efficiently.

Open-Source Flexibility and Community Support

As an open-source tool, Kestra offers flexibility that proprietary solutions often cannot match. Organizations can customize and extend Kestra to fit their needs by adding new features, integrating with custom tools, or optimizing performance for unique workloads. Kestra's open-source nature also means that it benefits from continuous improvement driven by a global community of developers and users.

Future Trends in Workflow Orchestration

One of the most significant trends shaping the future of workflow orchestration is the rise of low-code and no-code platforms. As businesses strive to become more agile, there is a growing demand for tools that allow users with little or no programming experience to create and manage complex workflows. Low-code/no-code orchestration platforms empower non-technical users to automate processes, reducing the reliance on specialized development teams and speeding up the time to market for new solutions.

Kestra is well-positioned to capitalize on this trend by potentially offering low-code/no-code features that complement its existing capabilities, such as providing a more visual, drag-and-drop interface or simplifying the workflow creation process.

AI and Machine Learning in Orchestration

Another emerging trend is integrating artificial intelligence (AI) and machine learning (ML) into workflow orchestration. AI and ML can enhance automation by enabling predictive analytics, intelligent decision-making, and error resolution. For instance, AI-driven orchestration tools could predict potential workflow bottlenecks and automatically adjust resource allocation to prevent delays. Machine learning models could analyze workflow patterns and suggest optimizations that improve efficiency and reduce costs.

Kestra has the potential to incorporate AI and ML capabilities to enhance its orchestration engine.

Cloud-Native Orchestration

As more organizations migrate their infrastructure to the cloud, the demand for cloud-native orchestration solutions grows. Cloud-native orchestration tools are designed to operate seamlessly in cloud environments, leveraging the cloud's elasticity, scalability, and capability of managing distributed systems, microservices, and applications while maintaining high performance and availability.

Greater Focus on Security and Compliance

With workflows becoming increasingly complex and the regulatory landscape growing, there is a heightened focus on security and compliance in workflow orchestration. Organizations need tools that automate processes and ensure that workflows comply with industry standards and regulations, such as GDPR, HIPAA, or SOC 2. Security features such as encryption, access controls, audit trails, and automated compliance checks are becoming essential.

Enhanced Observability and Monitoring

As workflows become more complex, enhanced observability and monitoring are critical. Organizations require detailed insights into their workflows' performance and status, including metrics, logs, and real-time alerts. Enhanced observability allows teams to quickly identify and resolve issues, optimize workflows, and meet service level agreements (SLAs).

Conclusion

In summary, Kestra addresses the modern challenges of workflow orchestration by offering a highly scalable, real-time processing platform that is both powerful and accessible. Its user-friendly design, extensive integration capabilities, and the flexibility of open-source software make it an ideal choice for organizations looking to streamline their operations and stay ahead in a competitive landscape.

Data retrieval is a crucial aspect of web development. It allows for the delivery of dynamic, real-time content, enhancing the user experience. It enables applications to obtain and display information quickly, seamlessly integrate with external services, and efficiently manage states without needing full page reloads.

This article details the evolution of data fetching methods in [Next.js](https://nextjs.org/docs) from the early days to the present with version 14.

Early Data Fetching Using getInitialProps

Before Next.js version 9.3, [getInitialProps](https://nextjs.org/docs/pages/api-reference/functions/get-initial-props) was the only way to retrieve server-side data. This function allows developers to collect data for a page before rendering; the server sends down a fully hydrated [HTML](https://developer.mozilla.org/en-US/docs/Web/HTML) page to the client. This approach has a markedly positive impact on SEO and initial page load times.

The code below will create a simple blog page in Next.js 13 that fetches five blog posts from an external API and displays them. The `getPosts` function retrieves the posts, and the `HomePage` component renders the content with each post’s title and body in a list. The posts are fetched and rendered on the server using server-side components before being sent to the browser.

```javascript
async function getBreweries() {
const res = await fetch(‘https://api.openbrewerydb.org/breweries?per_page=4');
if (!res.ok) {
throw new Error(‘Failed to fetch breweries’);
}
return res.json(); // Return the breweries array
}

const HomePage = async () => {
const breweries = await getBreweries();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’ }}>
<center><h1 style={{ fontSize: ‘35px’, color: ‘#333’ }}>Brewery Data Fetching Example</h1></center>
<ul>
{breweries.map((brewery) => (
<li key={brewery.id} style={{ marginBottom: ‘25px’ }}>
<h2 style={{ fontSize: ‘28px’, color: ‘#333’ }}>{brewery.name}</h2>
<p style={{ fontSize: ‘22px’, color: ‘#555’ }}>{brewery.city}, {brewery.state}</p>
</li>
))}
</ul>
</div>
);
};

export default HomePage;
```

The code above defines an `asynchronous` function, `getBreweries`, that fetches data from the open brewery API, specifically retrieving a list of four breweries. If the fetch request fails, it throws an error. Once the data is successfully fetched, the function returns the breweries as a `JSON` array.

The `HomePage` component is an asynchronous React component that calls the `getBreweries` function to fetch the data. It then renders a webpage with a yellow background (`#FFFF99`) and a minimum height of 100% of the viewport. Each brewery is displayed with its name, city, and state, styled with distinct font sizes and colors. The component maps through the breweries array and displays each brewery’s information inside list items. The layout emphasizes a clean, simple presentation of the fetched brewery data.

The data is fetched once before the page is rendered and sent to the client. The server fetches the data during the initial page load, and the client receives the fully rendered page with the data.

Advancements with getServerSideProps and getStaticProps

Next.js 9.3 added [getServerSideProps](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-server-side-props) and [getStaticProps](https://nextjs.org/docs/pages/building-your-application/data-fetching/get-static-props), two key enhancements that improve data fetching capabilities and online application efficiency and flexibility. Other `getServerSideProps` and `getStaticProps` benefits include the following:

* `getServerSideProps` fetches data on the server for each request, making it suitable for dynamic content. Key advancements include improved performance through optimized server-side rendering, support for streaming HTML content, built-in caching to avoid unnecessary data fetching, and the ability to leverage Incremental Static Regeneration (ISR) for a hybrid approach.

* `getStaticProps` fetches data at build time, generating static pages for quick delivery. Enhancements include better Static Site Generation (SSG), revalidation options for automatic page regeneration, hybrid rendering capabilities to mix static and dynamic content, and improved developer experience with error handling and TypeScript support.

* Choose `getServerSideProps` for fresh data on every request and `getStaticProps` for infrequent updates and better performance. Combining these methods allows for flexible and optimized application development.

Example of Fetching Data on Every Request

Here’s a simple example of how `getServerSideProps` can retrieve data for each request.

The code below creates a simple blog page in Next.js where the `homepage` displays a list of blog posts, where the `homepage` component receives an array of posts and shows them with each post’s title and body. The `getServerSideProps` function fetches five posts from an external API (`https://jsonplaceholder.typicode.com/posts?_limit=3`) whenever a user requests the page. The posts are passed as props to the component. The expected output is a page showing three blog posts with titles and descriptions, dynamically fetched for each page load.

Successful Data Fetch on Fetching Data on Every Request with `getServerSideProps`

```javascript
const fetchPosts = async () => {
const res = await fetch(‘https://jsonplaceholder.typicode.com/posts?_limit=5');

// Check if the response is successful
if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

return res.json();
};

const HomePage = async () => {
try {
const posts = await fetchPosts();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Example of Fetching Data on Every Request (Success)</h1>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ marginBottom: ‘20px’ }}>
<h2>{post.title}</h2>
<p>{post.body}</p>
</li>
))}
</ul>
</div>
);
} catch (error) {
return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Error: {error.message}</h1>
</div>
);
}
};

export default HomePage;
```

In the above code example, `getServerSideProps` retrieves blog entries from an API with each request. This ensures that the data on the page is constantly current, which is especially useful for content that changes often or varies depending on the user’s context. Image output can be seen below.

Second example of Failed Data Fetch with `getServerSideProps`

This code demonstrates how to handle both successful and failed data fetching in Next.js using the App Router. Instead of `getServerSideProps`, the `fetch` API is used directly within the server component. On success, it displays blog posts; on failure, it catches the error and shows an error message.

```javascript
export async function getServerSideProps() {
try {
const res = await fetch(‘https://invalid-api-url.com/posts?_limit=5');

// Check if the response is successful
if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

const posts = await res.json();

return {
props: { posts },
};
} catch (error) {
return {
props: { error: error.message },
};
}
}

const HomePage = ({ posts, error }) => {
if (error) {
return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Error: {error}</h1>
</div>
);
}

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Blog Posts</h1>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ marginBottom: ‘20px’ }}>
<h2>{post.title}</h2>
<p>{post.body}</p>
</li>
))}
</ul>
</div>
);
};

export default HomePage;
```
This code above attempts to fetch data from an API. If the fetch is successful, the posts are displayed as a list. If the fetch fails due to an invalid URL, an error message is shown. The page has a yellow background and black text, with posts or error messages styled appropriately. The fetch operation is wrapped in a `try-catch` block to handle errors and display them when they occur.

## Server-Side Rendering with getServerSideProps

The `getServerSideProps` function retrieves data from the server at the request time. This technique is appropriate for pages that need to display frequently updated information.

The code below will create a simple blog page in Next.js that fetches and displays the latest four blog posts. The `getPosts` function fetches the posts from an external API (`https://jsonplaceholder.typicode.com/posts?_limit=4`), ensuring that the data is not cached (`cache: no-store`), meaning it fetches fresh data on every request. The `homepage` component then renders a list of the post titles. The expected output is a page showing a list of four blog post titles, updated every time the page is loaded.

```javascript
async function getPosts() {
const res = await fetch(‘https://jsonplaceholder.typicode.com/posts?_limit=4', {
cache: ‘no-store’,
});
return res.json();
}

const HomePage = async () => {
const posts = await getPosts();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’ }}>
<h1 style={{ color: ‘#333’}}>Server-Side Rendering with getServerSideProps</h1>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ color: ‘#555’, fontSize: ‘22px’, margin: ‘10px 0’ }}>
{post.title}
</li>
))}
</ul>
</div>
);
};

export default HomePage;
```

This code above fetches a list of three blog posts from an API on every render but, due to the `no-store` cache setting, dynamically displays them as a list on the home page. The `getPosts` function handles the API call, and the posts are displayed using `map()` to generate `<li>` elements.

When the server renders the page, [Javascript](https://devdocs.io/javascript/) calls the async `getServerSideProps` method. It retrieves the data and then returns an object containing the props key. The data retrieved is subsequently supplied to the page.

The image produced above indicates that the data was properly fetched; otherwise, an error message is returned.

Rendering with Fresh Data (API Calls on Every Request)

Using the Fetch API directly in a server component can allow you to always fetch fresh data from the API on each request.

```javascript
// app/page.js

const fetchFreshPosts = async () => {
console.log(‘Fetching fresh data on every request’);
const res = await fetch(‘https://jsonplaceholder.typicode.com/posts?_limit=3', {
cache: ‘no-store’, // No caching, fetches fresh data each time
});

if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

return res.json();
};

const FreshDataPage = async () => {
const posts = await fetchFreshPosts();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’ }}>
<h1 style={{ fontSize: ‘32px’, color: ‘#333’ }}>Latest Blog Posts (Fresh Data)</h1>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ marginBottom: ‘20px’ }}>
<h2 style={{ fontSize: ‘28px’, color: ‘#333’ }}>{post.title}</h2>
<p style={{ fontSize: ‘22px’, color: ‘#555’ }}>{post.body}</p>
</li>
))}
</ul>
</div>
);
};

export default FreshDataPage;
```

The `FreshDataPage` component fetches the latest blog posts from an API using the `fetchFreshPosts` function, which ensures that fresh data is retrieved on every request by setting the cache option to `’no-store’`. It handles potential fetch errors by throwing an exception if the response is not okay.

Rendering with Cached Data (API Calls on Build)

In contrast, if you want to cache the data and only fetch it on build time or based on a revalidation strategy, you can use the code below.

```javascript
// app/static-page.js

const fetchCachedPosts = async () => {
console.log(‘Fetching data and caching it for 30 minutes’);
const res = await fetch(‘https://jsonplaceholder.typicode.com/posts?_limit=3', {
next: { revalidate: 1800 }, // Cache the data and revalidate every hour
});

if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

return res.json();
};

const CachedDataPage = async () => {
const posts = await fetchCachedPosts();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’ }}>
<h1 style={{ fontSize: ‘32px’, color: ‘#333’ }}>Rendering with Cached Data (API Calls on Build)</h1>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ marginBottom: ‘20px’ }}>
<h2 style={{ fontSize: ‘28px’, color: ‘#333’ }}>{post.title}</h2>
<p style={{ fontSize: ‘22px’, color: ‘#555’ }}>{post.body}</p>
</li>
))}
</ul>
</div>
);
};

export default CachedDataPage;
```

The `CachedDataPage` component fetches data from an API using the `fetchCachedPosts` function, which implements a caching mechanism to store the data for 30 minutes (1800 seconds). When the component is rendered, it displays a list of posts with their titles and bodies in a styled layout with a yellow background. If the data fetch fails, an error is thrown, ensuring that only valid responses are processed and displayed.

Static Site Generation with getStaticProps

`GetStaticProps` allows you to retrieve data during construction. This strategy works best for pages pre-rendered with data that does not change frequently. The code sample is below.

The code below will create a simple blog page in a Next.js app that fetches and displays the latest six blog posts. The `HomePage` function fetches data from an external API (`https://jsonplaceholder.typicode.com/posts?_limit=6`) on the server side, which means the data is retrieved when the page is requested. The page then renders a list of the post titles. The expected output is a webpage displaying six blog post titles under the heading Latest Blog Posts, showing updated content each time the page is loaded.

```jsx
export default async function HomePage() {
// Fetch data directly in the component (server-side)
const res = await fetch(“https://jsonplaceholder.typicode.com/posts?_limit=6");
const posts = await res.json();

return (
<div style={{ backgroundColor: ‘#FFFF99’, padding: ‘45px’ }}>
<h1>Static Site Generation with getStaticProps</h1>
<ul>
{posts.map((post) => (
<li key={post.id}>{post.title}</li>
))}
</ul>
</div>
);
}
```

This code above renders a list of blog posts fetched from an API at build time using `getStaticProps`, displaying the titles of the first six posts. The post `prop` is passed to the component and mapped to render each post’s title in an unordered list `<ul>`.

The output image above indicates that the data has been correctly fetched; if not, an error message will be returned.

Unlike `getServerSideProps`, which fetches the list of posts on each request, `getStaticProps` fetches it during the build time. This strategy improves performance by serving the page as static [HTML](https://developer.mozilla.org/en-US/docs/Web/HTML), faster than server-side-produced pages for each request.

When using `getStaticProps` and needing to update the data regularly, there was an option to use incremental static regeneration ([ISR](https://nextjs.org/docs/pages/building-your-application/data-fetching/incremental-static-regeneration)) by including a revalidate time in the return of `getStaticProps`.

Evolution of Next.js 14: Beyond getServerSideProps and getStaticProps

With the introduction of Server components in Next.js 13, much has changed in Next.js. For example, you no longer require methods such as `getServerSideProps` and `getStaticProps` to handle data fetching.

You can now utilize the fetch web API to conduct fetch requests in your components using `async/await` with the code below.

```jsx
‘use client’; // This marks the component as a Client Component

import React, { useEffect, useState } from ‘react’;

const HomePage = () => {
const [posts, setPosts] = useState([]);
const [error, setError] = useState(null);

useEffect(() => {
async function fetchPosts() {
try {
const res = await fetch(‘https://jsonplaceholder.typicode.com/posts?_limit=7', {
next: { revalidate: 10 }, // Revalidate every 10 seconds (ISR behavior)
});

if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

const data = await res.json();
setPosts(data);
} catch (err) {
setError(err.message);
}
}

fetchPosts();
}, []);

if (error) {
return <div>Error: {error}</div>;
}

return (
<div style={{ backgroundColor: ‘#FFFF99’, padding: ‘20px’, color: ‘black’ }}>
<h1 style={{ fontSize: ‘36px’}}>Evolution of Next.js 14: Beyond getServerSideProps and getStaticProps</h1>
<h2 style={{ fontSize: ‘28px’}}>openreplay</h2>
<ul>
{posts.map((post) => (
<li key={post.id} style={{ fontSize: ‘24px’}}>{post.title}</li>
))}
</ul>
</div>
);
}

export default HomePage;
```

The code above helps create a simple webpage that shows a list of posts like short messages from a website. In the first part, we have a function called `fetchPosts` that goes to the linked website to get seven posts. It uses `fetch`, like sending a request asking for data. If the request works, it returns the posts in a way the webpage can understand. If the request fails, it gives an error. The cool part is that it rechecks the website every ten seconds to see if any posts have changed or new ones have been added.

The second part of the code is the `Homepage` function. This function uses the posts from the other website and displays them in a list `<li>`.

According to the image output above, it shows that the data is fetched successfully if not, it will return an error message as an output.

This solution simplifies data retrieval by leveraging conventional web APIs and `async/await` for more understandable asynchronous code, dramatically shifting how data is handled in Next.js apps.

Caching in Next.js 14

Caching improves the retrieved Web API by allowing for server-side caching and revalidation. By default, Next.js caches the returned fetch values in the server’s data cache. Data can be gathered during the build or request process, cached, and utilized for subsequent requests.

```javascript
//’force-cache’ is the default caching strategy
fetch(‘https://...', { cache: ‘force-cache’ });
```

The `force-cache` strategy in the fetch API forces the browser to use a cached response if available, even if the resource has expired, before making a network request.

For cases requiring new data on each request, the cache option can be set to `no-store` to bypass the cache entirely:

```javascript
fetch(‘https://api.example.com/posts', { cache: ‘no-store’ });
```
Furthermore, [Route Segment Config](https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config) settings allow developers to turn off caching for specified route segments, affecting all data requests inside that segment.

```javascript
/ Disable caching for this route segment
export const dynamic = ‘force-dynamic’;
```

The code command above disables caching for a specific route segment in Next.js by forcing the content to be dynamically generated on every request, ensuring the latest data is served.

View all the available cache options in the [fetch](https://nextjs.org/docs/app/api-reference/functions/fetch) API reference.

Example of Caching with the Fetch API in Next.js 14

In this example, we’ll fetch blog posts directly in a React component and leverage the built-in caching capabilities of the Fetch API. This approach allows you to cache the fetched data automatically.

```javascript
const fetchUsers = async () => {
const res = await fetch(‘https://reqres.in/api/users?page=1', {
next: { revalidate: 3600 }, // Cache data for 1 hour
});

if (!res.ok) {
throw new Error(‘Failed to fetch data’);
}

const data = await res.json();
return data.data; // Return the users array
};

const UsersPage = async () => {
try {
const users = await fetchUsers();

return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Example of Caching with the Fetch API in Next.js 14; Cached Users</h1>
<ul>
{users.map((user) => (
<li key={user.id} style={{ marginBottom: ‘20px’ }}>
<h2>{`${user.first_name} ${user.last_name}`}</h2>
<p>Email: {user.email}</p>
</li>
))}
</ul>
</div>
);
} catch (error) {
return (
<div style={{ backgroundColor: ‘#FFFF99’, minHeight: ‘100vh’, padding: ‘20px’, color: ‘black’ }}>
<h1>Error: {error.message}</h1>
</div>
);
}
};

export default UsersPage;
```

`fetchPosts` function uses the `Fetch` API to retrieve blog posts. By setting `next: { revalidate: 3600 }`, we specify that the data should be cached for one hour. After that time, the next request will fetch fresh data from the API. The posts are then rendered in the component, maintaining a consistent yellow background for styling, along with readable text colors and sizes.

After entering the code above in your `app/page.js` file and you click on save, the image above is what will display in your terminal to show you are doing it right. Then you navigate to your browser to see the output on your localhost as shown below.

The output image above indicates that the data has been correctly fetched; if not, an error message will be returned.

Revalidating data in Next.js 14

Revalidation involves clearing the data cache and re-fetching the most recent data. This is particularly handy when your data is dynamic and you must present the most up-to-date information.

The following codes can revalidate data at a specific time interval. Use the fetch revalidate option to set a resource’s cache lifespan in seconds.

```javascript
fetch(‘https://api.example.com/posts', { next: { revalidate: 3600 } });
```

Route segment `config` options also allow for `route-wide` revalidation settings:

```javascript
export const revalidate = 3600; // revalidate at most every hour
```

This code sets a revalidation interval of 3600 seconds (1 hour) for the export, meaning the data will be revalidated at most once every hour.

If you have many fetch queries in a statically rendered route, each with a unique revalidation frequency, all inquiries will be served at the fastest possible speed. However, each fetch request will be verified separately to render routes dynamically.

Data fetching in client components in Next.js 14

Next.js 14 allows you to use [Route handlers](https://nextjs.org/docs/app/building-your-application/routing/route-handlers) or third-party libraries like [SWR](https://swr.vercel.app/docs/with-nextjs) or [TanStack Query](https://www.npmjs.com/package/@tanstack/react-query) to fetch data on the client side.

Let’s look at how to fetch data with route handlers.

* Route handlers run on the server and return data to the client, which is useful when you don’t want to share sensitive information with the client, such as API tokens.

* Route handlers use the [Web Request](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest) and [Response API](https://developer.mozilla.org/en-US/docs/Web/API/Response) to create custom request handlers for specific routes, they are defined as a [routes.js](https://www.npmjs.com/package/routes-js) file within the app directory.

This code below will create a client component in a Next.js app that fetches five blog posts from an external API, manages loading states and errors with React hooks, and displays either a loading message, an error message, or a list of the post titles once the data is successfully retrieved.

```javascript
“use client”; // Ensure this is present at the very top

import { useState, useEffect } from “react”;

const HomePage = () => {
const [posts, setPosts] = useState([]);
const [loading, setLoading] = useState(true);
const [error, setError] = useState(null);

const fetchPosts = async () => {
setLoading(true);
try {
const response = await fetch(“https://jsonplaceholder.typicode.com/posts?_limit=5");
if (!response.ok) {
throw new Error(“Failed to fetch posts”);
}
const data = await response.json();
setPosts(data);
} catch (error) {
setError(error.message);
} finally {
setLoading(false);
}
};

useEffect(() => {
fetchPosts();
}, []);

if (loading) {
return <div>Loading…</div>;
}

if (error) {
return <div>Error: {error}</div>;
}

return (
<div>
<h1>Data Fetching in Client Component Nextjs 14</h1>
<ul>
{posts.map((post) => (
<li key={post.id}>{post.title}</li>
))}
</ul>
</div>
);
};

export default HomePage;
```

The code above helps make a webpage that shows a list of 5 posts (short messages) from the linked webpage. First, it has a part called `useState` that keeps track of the posts, when the page is still loading, and if there’s any problem. But when the webpage starts, it runs a special function called `fetchPosts` to get the posts from the linked webpage. If the posts come back okay, it saves them to display. If something goes wrong, it shows an error message.

The `HomePage` function controls what you see on the screen. While the posts are being loaded, it shows loading…, and if there’s an error, it shows a message with what went wrong. Once the posts are fetched, they get displayed in an unordered list `<ul>`, each with its spot. This way, the webpage can grab information from the internet and show it neatly. Output can be seen in the image below.

Based on the image output above, it appears that the data was successfully fetched. If not, an error message will be returned as the output.

Conclusion

In Next.js 14, data fetching has evolved from the early use of `getInitialProps` to more sophisticated methods like `getServerSideProps` and `getStaticProps`, introduced in Next.js 9.3, which improved efficiency by fetching data on every request or during build time. With the advent of Server Components and Fetch API in Next.js 13, developers now fetch data directly within components using `async/await`, simplifying the process and eliminating the need for earlier methods. Client-side data fetching is also enhanced, utilizing React hooks and libraries like [SWR](https://swr.vercel.app/docs/with-nextjs) and [TanStack Query](https://tanstack.com/query/latest/docs/framework/react/overview) for dynamic data management. The caching and revalidation features introduced in Next.js 14 enable control over data freshness and improve performance by optimizing how and when data is retrieved or updated.

In summary, Next.Js 14 offers an advanced and flexible data-fetching ecosystem, blending server and client-side strategies to efficiently handle dynamic content, caching, and real-time data updates.