We use this methodology at the time of writing for our feeds on Pinkary, and to combat the issue, we limit our not-so-infinite scrolling to 100 posts; hence, I set out to find if an alternate approach exists.

In this article, I'll describe a way to do this more efficiently: fetching only the records we need to keep the page growing and appending those records to the existing DOM. Aka, how you'd normally do it when using a typical JS framework + api approach.

Traditional Livewire Infinite Scrolling Recap

Let's quickly go over the traditional setup to establish a baseline. We're going to infinitely load Posts into a feed (at least, until our server would run out of memory).

Here's an example Livewire feed component:

<?php

namespace App\Livewire;

use App\Models\Posts;
use Livewire\Attributes\Locked;
use Livewire\Component;

final class Feed extends Component
{
    /**
     * The number of posts per "load more" cycle.
     */
    #[Locked]
    public int $perPage = 5;

    /**
     * The total number of posts to load.
     */
    #[Locked]
    public int $limit = 5;

    /**
     * Load more questions.
     */
    public function loadMore()
    {
        $this->limit += $this->perPage;
    }

    public function render()
    {
        $posts = Posts::query()
            ->orderByDesc('created_at')
            ->simplePaginate($this->limit);

        return view('livewire.feed', [
            'posts' => $posts,
        ]);
    }
}

And that livewire.feed view can be something like:

<section>
    <!-- Render all the posts -->
    @foreach ($posts as $post)
        <livewire:posts.show
            :postId="$post->id"
            :key="'post-' . $post->id"
        />
    @endforeach

    <!-- Load more intersection -->
    <div x-intersect.margin.50%="$wire.loadMore()">
        <div class="text-center text-slate-400" wire:loading wire:target="loadMore">
            <x-heroicon-o-arrow-path class="w-5 h-5 animate-spin" />
        </div>
    </div>
</section>

And we can assume that's wired together by a simple web route that does little more than return the view (which gets Livewire rolling).

As mentioned above, the problem with this setup is that the infinite scrolling works by increasing the limit on our query for each "load more" cycle, grabbing all the records (which includes the records from the prior loading cycle), creating new html for the entire <section>, sending that over the internet, and then morph-replacing the existing DOM with our new stuff. So, when we want the 91-95th posts, we just grab all 95 from the database and render + send that html. When the 96-100th posts are needed, we grab all 100, etc.

Each load more cycle means a larger query, larger html payload over the internet, and more DOM stuff to morph.

What Would Be More Efficient?

Doesn't take rocket science here... We've been doing this effectively in our api + JS apps for years now. Something like:

1. Make a JS feed component that has a data prop for an array of posts.
2. Fetch an initial amount of posts from the server and hydrate those into the array, rendering them into the DOM (possibly as a SSR procedure).
3. When it's time to load more, ask the api to give you the limited subset of posts you need (ie. 6-10th posts, 11-15th, etc.).
4. Importantly, append those new posts to the array, which re-renders the DOM and appends the new post elements to our feed.
5. Rinse and repeat.

Since we're not working with a JS frontend like Vue/React/etc., we'll need to approach this in a similar but slightly different way with Livewire.

What if we could set up a Livewire component to render a specific subset of posts (what we'd normally think of as a "page") as html and then append that html into the right location within the DOM? That'd be almost identical to the JS app methodology, except we get the server to convert an array of posts into html rather than doing that in the browser. Appending those new DOM elements, however, is the same concept.

Sounds like a plan.

The Efficient Approach

Let's do this.

First, we'll need a way to append html elements to the DOM in our Livewire app over replacing. Now, I'm a fan of the least-dependencies-possible approach, but for the sake of this article, I'm going to utilize a cool package by Christian Taylor (@imacrayon) called Alpine AJAX.

With that installed, we can update our Livewire Feed component and view. Here's the best part: the changes are minimal and overall, we get to reduce our code.

Note: the code examples are just rough examples, not meant to be copied verbatim; they don't represent the true source code of Pinkary.

<?php

namespace App\Livewire;

use App\Models\Posts;
use Livewire\Component;

final class Feed extends Component
{
    /**
     * The number of posts per "load more" cycle.
     */
    #[Locked]
    public int $perPage = 5;

    public function render()
    {
        $posts = Posts::query()
            ->orderByDesc('created_at')
            ->cursorPaginate($this->perPage); // yay! cursor pagination!

        return view('livewire.feed', [
            'posts' => $posts,
        ]);
    }
}

To update the view, we just need to tell Alpine AJAX what to do ($ajax('{{ $posts->nextPageUrl() }}', { target: 'feed pagination' })), what to do with the html payload (x-merge="append"), and to which DOM elements (id="feed' and id="pagination").

<section id="feed" x-merge="append">
    @foreach ($posts as $post)
        <livewire:posts.show
            :postId="$post->id"
            :key="'post-' . $post->id"
        />
    @endforeach

    @if($posts->hasMorePages())
        <div
            id="pagination"
            x-intersect.margin.600px="$ajax('{{ $posts->nextPageUrl() }}', { target: 'feed pagination' })"
        ></div>
    @endif
</section>

Last piece is to update our web route handling for the feed so that the Alpine AJAX call is handled correctly.

This can be done in a controller like so:

<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Illuminate\View\View;
use Livewire\Livewire;

final class FeedController
{
    public function __invoke(Request $request): View|string
    {
        // Determine if the request is a first page load or an infinite scroll "load more" request.
        $isInfiniteScrollRequest = $request->hasHeader('X-Alpine-Request');

        return $isInfiniteScrollRequest
            ? Livewire::mount('feed') // render the component and return its html string fragment 👀
            : view('feed'); // do a regular full page load
    }
}

That's it. Let's break it down.

First of all, if we're dealing with a full page load (someone hitting our feed route and not attempting to "load more"), we can just return the view('feed') like normal.

If we're working with a "load more" request from Alpine AJAX, we get a little hacky (here's where the hack in the title comes from). We ask Livewire to mount our Feed component, and the returned value from that Livewire::mount() function call is the html string returned from the render() method of our Feed component after Livewire has performed some lifecycle events.

public function render()
{
    $posts = Posts::query()
        ->orderByDesc('created_at')
        ->cursorPaginate($this->perPage);

    return view('livewire.feed', [
        'posts' => $posts,
    ]);
}

Through the ease provided to us from Laravel's cursor pagination implementation, our AJAX request will have the query params needed for Laravel to know which cursor we're working with and what page we want to query (you could manually handle this if you needed to for some reason).

Thus, on our infinte scrolling "load more" requests from Alpine AJAX, we will fetch only a subset of posts for the next cursor page (->cursorPaginate(...)), render a fragment of html with those posts (return value from render()), and respond with that html string back to the browser. Alpine AJAX handles that response, and because of our configuration above, it will append that new html of posts into our <section id="feed"...> element.

Easy peasy.

With this in place, we do a normal full page load (like any other Livewire app) and we get efficient infinite scrolling (appending a limited subset of new html elements to our feed). All of the appended elements retain all of their normal Livewire component functionality.

It's a little hacky, because we're no longer using Livewire exclusively (we now have that FeedController that handles our special request scenarios) and we're asking Livewire to render a component as a raw html string that we send back to the browser. But hey - it's not the worst hack I've seen for a production app. Plus, this concept of appending html (vs. replacing) from a Livewire request->response will be built into the package one day in the future.

Bonus Example Using Blade Fragments

If it seems silly to make the feed a Livewire component (our Feed component in this article), since it doesn't have any extra methods that are called for client-triggered async functionality (one of the big benefits of using Livewire), I'll show you how to accomplish this using Laravel's @fragment feature of its Blade engine.

Since you now know the concept, I'm just going to quickly dump the code.

<?php

namespace App\Http\Controllers;

use App\Models\Post;
use Illuminate\Http\Request;
use Illuminate\View\View;
use Livewire\Livewire;

final class FeedController
{
    public function __invoke(Request $request): View|string
    {
        $isInfiniteScrollRequest = $request->hasHeader('X-Alpine-Request');

        // Look familiar? 😉
        $posts = Post::query()
            ->orderByDesc('created_at')
            ->cursorPaginate($perPage = 5);

        return view('feed', ['posts' => $posts])
            ->fragmentIf($isInfiniteScrollRequest, 'posts-list'); // fragment magic! 👀
    }
}

Making a posts.list Blade component:

@props(['posts'])
@fragment('posts-list')
    <div x-data>
        <section id="feed" x-merge="append">
            @foreach ($posts as $post)
                <livewire:posts.show
                    :postId="$post->id"
                    :key="'post-' . $post->id"
                />
            @endforeach
        </section>
        @if($posts->hasMorePages())
            <div
                id="pagination"
                x-intersect.margin.600px="$ajax('{{ $posts->nextPageUrl() }}', { target: 'feed pagination' })"
            ></div>
        @endif
    </div>
@endfragment

And that posts.list component can be called from a feed view.

<x-app-layout>
    <x-home-menu></x-home-menu>

    ...

    <!-- This would have been the livewire:feed component earlier in the article -->
    <x-posts.list :posts="$posts"></x-posts.list>
</x-app-layout>

* Actually, these limitations likely extend beyond PHP, as the root cause may be in an underlying C library used by a number of other programming languages.

Here's what I learned, the rabbit holes I went down, and the surprisingly simple solutions that gave me massive performance improvements.

Context

• The XML files I was importing contained a small intro/header section of data followed by an array of ~3 million nodes that all shared a known schema.
• When importing, each element could be handled in isolation.

1. The Speed Problem

I initially wrote a simple artisan command that would truncate the database table, take in the source xml file as a command argument, use the XMLReader class to iterate through the array's sibling nodes, and then bulk import the objects 5,000 records at a time.

public function handle()
{
    // truncate, handle file argument...

    $reader = \XMLReader::open($this->file);

    $batch = [];
    $count = 0;
    $batchSize = 5000;

    foreach($this->iterateRecords($reader, $elementName) as $xmlRecord) {
        $dto = $this->createDto($xmlRecord, $type);
        $batch[] = [
            ...$dto->toModelAttributes(),
            // other static attributes
        ];

        // Bulk insert batch
        if (count($batch) >= $batchSize) {
            $modelClass::query()->insert($batch);
            $count += count($batch);
            $batch = [];
        }
    }

    // Insert remaining batch
    if (! empty($batch)) {
        $modelClass::query()->insert($batch);
        $count += count($batch);
    }

    // clean up...
}

private function iterateRecords(XMLReader $reader, string $elementName): Generator
{
    // This isn't verbatim code but gets the point across...

    // Get to the first instance of our element to import
    do {
        if ($elementName === $reader->localName) {
            break;
        }
    } while ($reader->read());

    // Yield the outer xml for each element
    do {
        yield $reader->readOuterXml();
    } while ($this->reader->next());
}

With this in place, my import process would start strong but degrade in speed over time:

Starting to import records...
Imported 5000 records. Current rate: 3557 records/s
Imported 10000 records. Current rate: 2273 records/s
Imported 15000 records. Current rate: 1931 records/s
...
Imported 125000 records. Current rate: 320 records/s
Imported 130000 records. Current rate: 307 records/s

Starting at ~3,500 records/second and dropping to ~300/s after just 130,000 records. At that rate, importing 3 million records would take forever—assuming PHP didn't run out of memory first.

That wasn't going to do.

2. Noticing the Memory Leak

To begin my investigation of the speed issue, I opened Activity Monitor and noticed two things: the PHP process was consuming an ever-growing amount of memory, and 2) the mysqld process was barely touching the cpu.

So I now had two symptoms to investigate:

1. Performance degradation - throughput dropping from 3,500/s to 300/s
2. Memory leak - unbounded memory growth of the PHP process

I assumed these were related. Spoiler: they weren't.

3. Investigating the Leak

My first instinct was that the memory leak was related to XML parsing or query logs. We were streaming gigabytes of data after all. All it would take is some distant reference messing up the garbage collector or keeping a hold of the xml strings and kaboom-big leak. So, I tried some of the simple and obvious options:

Forced garbage collection:

if (count($batch) >= $batchSize) {
    $modelClass::query()->insert($batch);
    unset($batch);
    gc_collect_cycles();
    gc_mem_caches(); // PHP 8.1+ - releases string interning cache
}

Disabling Laravel's query log:

DB::disableQueryLog();

Neither of these attempts made a significant difference. The performance continued to degrade, and memory continued to grow.

I wrote a series of tinker tests to isolate the issue, testing raw XMLReader, XMLReader with DTO creation, batch building, and various combinations. The tests revealed something surprising: XMLReader itself wasn't leaking memory in isolation, but performance was degrading severely. We were onto something, at least for the speed issue...

4. The Real Performance Culprit: XMLReader's Traversal

After systematic testing, I discovered the root cause of the performance degradation. It wasn't memory, garbage collection, or the database—it was XMLReader's fundamental traversal behavior.

I ran performance tests comparing different approaches:

===
  Raw fread() baseline
  How quickly can we buffer through the file?
===
Duration: 0.05s for 50000 records
Batch 1: 5000 records, 777586/s
Batch 2: 10000 records, 1099885/s
Batch 3: 15000 records, 1182893/s
Batch 4: 20000 records, 988756/s
Batch 5: 25000 records, 1094604/s
Batch 6: 30000 records, 948594/s
Batch 7: 35000 records, 1110367/s
Batch 8: 40000 records, 993863/s
Batch 9: 45000 records, 1118839/s
Batch 10: 50000 records, 951261/s
First batch rate: 777586/s
Last batch rate: 951261/s


===
  XMLReader::next()
  How quickly can the XMLReader traverse elements with no addt'l processing?
===
Duration: 36.29s for 50000 records
Batch 1: 5000 records, 10303/s
Batch 2: 10000 records, 3627/s
Batch 3: 15000 records, 2683/s
Batch 4: 20000 records, 1848/s
Batch 5: 25000 records, 1578/s
Batch 6: 30000 records, 1320/s
Batch 7: 35000 records, 1000/s
Batch 8: 40000 records, 950/s
Batch 9: 45000 records, 839/s
Batch 10: 50000 records, 746/s
First batch rate: 10303/s
Last batch rate: 746/s

The results were damning:

• Raw fread(): Consistent ~1 million records/s with no degradation; disk reading & buffering was plenty fast.
• XMLReader::next(): Starts at 10,303/s but degrades to 746/s—and this was without even reading any content

The degradation appears to be roughly quadratic as the number of nodes grows. Just calling next() to traverse to the next sibling gets progressively slower the further you are into the document.

Why This Happens: libxml2?

PHP's XMLReader is a thin wrapper around libxml2. Therefore, I believe the performance degradation originates in libxml2, not PHP itself. There are known performance issues in libxml2, which may be causing this specific behavior seen during my imports. Here's a specific issue from someone using a XML library in Ruby that showed a similar performance problem.

If I can find some spare time, I may write a C program to test libxml2 in a few ways to ensure that's truly the cause and not something else related to PHP's wrapper of the library.

My Pragmatic Solution: String-Based Parsing

For predictable XML structures where you know the element names you're parsing, simple string operations massively outperform XMLReader:

public function iterate(): Generator
{
    $handle = fopen($this->xmlPath, 'r');
    $buffer = '';
    $bufferSize = 1024 * 1024; // 1MB chunks

    while (! feof($handle)) {
        $buffer .= fread($handle, $bufferSize);

        while (($startPos = $this->findStartTag($buffer)) !== false) {
            $endPos = strpos($buffer, $this->endTag, $startPos);

            if ($endPos === false) {
                break; // Need more data
            }

            $endPos += strlen($this->endTag);

            yield substr($buffer, $startPos, $endPos - $startPos);

            $buffer = substr($buffer, $endPos);
        }
    }

    fclose($handle);
}

This approach:

• Reads the file in 1MB chunks with fread()
• Uses strpos() to find record boundaries
• Uses substr() to extract each record
• Yields records one at a time for memory efficiency

I ran a performance test like the ones above for this implementation:

=== String-based (fread + strpos) ===
Duration: 0.64s for 50000 records
Batch 1: 5000 records, 80299/s
Batch 2: 10000 records, 76330/s
Batch 3: 15000 records, 79961/s
Batch 4: 20000 records, 75922/s
Batch 5: 25000 records, 76749/s
Batch 6: 30000 records, 76267/s
Batch 7: 35000 records, 79428/s
Batch 8: 40000 records, 79196/s
Batch 9: 45000 records, 81635/s
Batch 10: 50000 records, 80697/s
First batch rate: 80299/s
Last batch rate: 80697/s

Now we're talkin'. Obviously, it's much slower than just reading through the file, but the key is that the performance remains constant and that it's fast enough for my needs.

5. The Memory Leak: Sentry's Query Tracking

After switching to string-based parsing, throughput was excellent and consistent. But the memory leak persisted. The import would run beautifully at ~17,000 records/s with consistent performance, but eventually crash:

Allowed memory size of 2147483648 bytes exhausted
at vendor/laravel/framework/src/Illuminate/Database/MySqlConnection.php:53

I initially suspected something in Laravel's query builder was holding a reference to the inserted data. Alas, I tested raw PDO statements (bypassing the query builder) and it had stable memory throughout the import. So something about using Laravel's query building and DB connection was causing the issue.

Since the source code between my raw PDO statements and Laravel's connection were so similar, my immediate next suspicion was the QueryExecuted event that is dispatched when running a statement in Laravel. Bazinga. I discovered the culprit: Sentry.

Sentry's Laravel integration listens to QueryExecuted events to capture queries for breadcrumbs and performance tracing. Each query's data was being retained in memory. With thousands of batch inserts, this accumulated until PHP ran out of memory.

The fix is simple—disable Sentry's breadcrumbs & tracing via config settings, or, as I chose to do, disable the QueryExecuted event listeners during bulk imports:

use Illuminate\Database\Events\QueryExecuted;
use Illuminate\Support\Facades\Event;

// Disable query event listeners during bulk import
Event::forget(QueryExecuted::class);

// Now Laravel's standard insert() works with stable memory
foreach ($batches as $batch) {
    Model::query()->insert($batch);
}

6. Index Optimization

For even more processing throughput, I removed indexes from the database tables before the import and recreated them afterwards. This is a recommended approach by MySQL for bulk data loading.

This reduces how much processing the database has to do per insert, which was fairly significant in my case as some of the tables I was importing to had multiple unique indexes. Thus, the database had to not only update the B-tree structure but also validate uniqueness per insert.

7. Overall Results

After all optimizations:

Starting to import records...
Imported 5000 records. Current rate: 19593 records/s
Imported 10000 records. Current rate: 19549 records/s
Imported 15000 records. Current rate: 19980 records/s
...
Imported 95000 records. Current rate: 20877 records/s
Imported 100000 records. Current rate: 21021 records/s

Consistent ~20,000 records/second throughout the entire import, with stable 80-100MB memory usage.

Key Takeaways

1. Don't trust XMLReader for very large files. Despite being marketed as a streaming parser, it has fundamental performance issues with multi-gigabyte files. The degradation is in traversal itself—even next() without reading content degrades ~14x over 50,000 records. This appears to be a libxml2 issue. Simple string operations can be 100x faster with zero degradation.

2. Separate your symptoms. Performance degradation and memory leaks can have completely different causes.

3. Watch your CPU distribution. When PHP is at 100% and MySQL is at 2%, your bottleneck isn't the database.

4. APM tools can cause memory leaks in bulk operations. Sentry (and likely other APM tools) listen to query events and retain data for tracing. For bulk imports, disable query tracking with Event::forget(QueryExecuted::class).

5. Drop and recreate indexes for bulk imports. MySQL recommends this approach for good reason—the time complexity math works in your favor.

6. Profile before optimizing. Systematic isolation testing found XMLReader and Sentry faster than guessing would have.

That title is a lot to take in. Let's break it down, cause this is actually a simple concept.

Example Scenario

Let's imagine we have 2 Eloquent models - a Question and an Answer. Those models are linked by a simple BelongsTo and HasMany relationship (a Question->hasMany(Answer) and an Answer->belongsTo(Question)).

Now let's imagine our business logic requires that someone can set a maximum number of answers permitted for each question. Ok, so our Question model will get an attribute called max_answers. If a particular Question has a count of Answers that matches the integer stored in the max_answers attribute, we will consider that Question "full". Perhaps in our imaginary app, we would only reveal answers to "full" questions and so we need to account for this in our API responses for the /answers endpoint.

There are other scenarios where this could prove useful: polls with special logic around a user-configurable number of votes, an ecommerce product that requires a user-configurable number of orders before it goes into production, etc. The key here is "user-configurable". At the very least, this exercise will help you explore some advanced Eloquent concepts.

Here's where the title of this article comes in...

Rephrasing the question

What if you wanted to find a list of Answers that belong to a "full" Question?

You may know, from the Laravel docs, that we can query for records based on the existence of a relation and include a count in that query. It looks like:

Model::query()->has('relation', '=', :integer:)->...;

Or, if you need to specify a constraint on the relation:

Model::query()->whereHas(
    'relation',
    function(Builder $relationQuery) {
        $relationQuery->where('foo', 'bar');
        ...
    }
    '=',
    :integer:
)->...;

Using our Question and Answer example, we could find a list of questions that have exactly 5 answers like so:

Question::query()->has('answers', '=', 5)->get();

Don't forget the operator (= above) could be any database compliant operator (ex. <, >, <>, etc).

However, all of these examples use a hard-coded count value (:integer:).

How do we do this with a count value that is stored in the database so that we can look for "full" Questions?

Unlocking additional functionality

It turns out it's possible to leverage the has() or whereHas() methods to accomplish this. At first, this will seem impossible because the Laravel has() method specifies the third parameter must be of integer type via docBlocks. Nonetheless, the following does work, but it's important to wrap our special count in a Query\Expression, otherwise Eloquent will use a literal string in the final query to the database (like 'questions.max_answers' vs. questions.max_answers), which would cause a useless integer to string comparison by the database engine.

Let's first describe the query for a list of "full" Questions directly, since that's conceptually easier to grasp.

Question::query()
    ->has('answers', '=', new \Illuminate\Database\Query\Expression('questions.max_answers'))
    ->get();
// or, using the DB Facade...
Question::query()
    ->has('answers', '=', \Illuminate\Support\Facades\DB::raw('questions.max_answers'))
    ->get();

Side note: you can use (new Question)->qualifyColumn('max_answers') vs. 'questions.max_answers' if you want Laravel to qualify the columns for you. You will see a version of this shortly, just using an existing query Builder instance instead of a new Model instance...

Technically, one could perform a relation load on the Questions there to resolve the Answers to our original question (Question::query()->with('answers')->has(...)->get()), but the point of this article is to show you how to accomplish the Eloquent setup by looking up the Answers directly.

Final Solution

Ok. We're ready to take this one step further. Let's find Answers that belong to a "full" Question.

The final solution conceptually uses a combination of what we discussed above:

Answer::query()
    ->whereHas(
        'question',
        fn (Builder $whereQuestion) => $whereQuestion
            ->has('answers', '=', $whereQuestion->qualifyColumn('max_answers'))
    )
    ->get();

As you can see, it's possible to use the whereHas() method to easily say "give me answers that belong to a 'full' question as saved in the db."

Hint: under the hood, the whereHas() method on the \Illuminate\Database\Eloquent\Concerns\QueriesRelationships trait is nothing more than an alias to the has() method. The whereHas() definition simply reorders the parameters so that the sub query Closure is in the second slot.

Bonus Round

Want to see two other ways of doing this? See if you can figure out what's going on and how the database may handle each query (use of indexes, etc).

Answer::query()
    ->whereIn('question_id', fn($questionIdQuery) => $questionIdQuery
        ->select('id')
        ->from('questions')
        ->where('questions.max_answers', '=', fn($subQuery) => $subQuery
            ->selectRaw('count(*)')
            ->from('answers')
            ->whereColumn('questions.id', 'answers.question_id')
        )
    )
    ...

The query above is essentially an alternate version of a WHERE EXISTS (...) query and will perform similarly due to foreign key indexes.

Answer::query()
    ->whereHas('question', fn($query) => $query
	    ->withCount('answers')
	    ->groupBy('id')
	    ->havingRaw('answers_count = questions.max_answers')
    )
    ...

Just in case you want to confuse someone by using a HAVING clause... 😅

Want to see it in action? Jump straight to the playground →

In an AI conversation (and for documentation), Markdown is the status quo, and for good reason: it's cheap to generate, it streams cleanly, and you can read the raw text with no renderer at all. It's also a language with a long habit of being extended when people want more. GitHub wanted more than base Markdown offered, so GitHub-Flavored Markdown added tables, task lists, strikethrough, and bare-URL autolinks, and the extensions were useful enough that GFM became the default dialect; models emit it without being asked. GitHub kept going with alerts, which gave Markdown a callout syntax. Mermaid solved diagrams with a small DSL inside a code fence, and enough renderers support it now that it's practically part of the dialect too. Stripe's Markdoc and MyST push further still, adding whole component systems through named tags and directives, though both are aimed at documentation sites and their tags are wordy enough that you wouldn't want to stream them into a chat. So the appetite is proven, and so is the pattern. What's missing is the conversation-shaped version: the moment a response wants to show a metric, a small chart, a status badge, or two stats side by side, there's still little to no way to do so, and you get a wall of prose where a couple of stat cards would have landed the point at a glance.

There's already a halfway answer to this, and it's spec'd into Markdown itself: when Markdown can't express something, you can drop into raw HTML and (some) renderers pass it through. And in certain cases, that solves the richness problem. But there's a few issues with that, in order of my concerns when it comes to AI conversations & workflows:

1. Most GUIs (and definitely the terminal) don't render inlined html from an agent.
2. Html is token hungry.
3. It's miserable to read raw html when any meaningful styling is applied.
4. It's a security threat.

Plus, the power of html + css + js is overkill for a conversation and simple documentation. It doesn't need animated, interactive, on-brand interfaces for every response, and when that is needed, well... that's when I have AI write a full html page (or generate a pdf, powerpoint, etc.).

I just want something a little richer than plain Markdown, at close to plain Markdown's cost.

So the question became: why stop where GFM stopped? Do what GitHub did for tables and alerts, and keep going: give Markdown components for basic layout, information display, and input, and you get the richer response at a negligible token and legibility cost. In Supermark, that three-column layout is three characters: |||. The rule I kept coming back to, and eventually wrote at the top of the spec: if a feature can't be expressed in a handful of characters, it probably doesn't belong. A typical Supermark document lands around 2–6× smaller than the equivalent HTML for the same interface (there's a counter in the playground), and when you're streaming to users or paying per token, that compounds fast.

That sketch turned into Supermark. It isn't trying to replace real UI (it would be bad at that). It's trying to make a slightly richer Markdown response possible: keep everything that makes Markdown good in a conversation, and add just enough vocabulary to show the handful of shapes a reply actually wants.

One caveat up front: Supermark is an idea, not a proposal, and certainly not a product. A real proposal would come with the follow-through: parsers in more than one language, renderers people can actually adopt, plugins for the tools where Markdown already lives. This is earlier than that. It's something I've noticed, something I've wanted, and a sketch of what it could look like, with a playground so you can poke at it. I'm writing it up because it already makes my own reading a little better, and because I'd like to know whether it makes sense to anyone else, or whether richer AI responses should come from a different direction entirely, before deciding it deserves the follow-through.

The one-sentence version

Supermark is a superset of Markdown (GFM included) where layout, data, and interactivity are first-class, expressed in the fewest characters that still read as plain text. Every valid Markdown document is already valid Supermark. You only reach for the extra syntax when you want something Markdown can't do (a metric, a chart, a card, a callout), and when you do, it stays terse and legible.

Here's the whole idea in one example. Ask an analytics agent how the quarter's going, and this is the reply you'd want back. On the left is what the model writes. On the right is what my renderer turns it into.

# Q3 at a glance
@crumbs Home / Reports / Q3

|||
@stat $128k "MRR" +18%
|||
@stat 2,410 "Active teams" +6%
|||

@bar Enterprise 62000
@bar Growth 41000
@bar Starter 25000

[!ok] Net revenue retention crossed 100% this quarter.

That whole interface is a handful of lines of text a model can generate as reliably as it generates a bulleted list, for about the token cost of writing it out as prose. That's the premise: richer responses, negligible cost. From there, the interesting decisions are in the syntax itself. Given what Supermark is for, I had four priorities, and then a payoff that fell out of meeting them.

Why the syntax looks the way it does

1. It has to stream

Supermark is meant to be read as it's written, token by token, because that's how models produce it. So the grammar is line-oriented: with very few exceptions, the meaning of a line is decided by how it starts, and no line needs to know about the lines after it. A parser never has to look ahead, which means it can render a document as it arrives. The stat card appears the instant its line is complete; you're not staring at a spinner waiting for a closing tag five hundred tokens away.

2. It has to read well raw

Every construct is designed so that, stripped of a renderer, it still reads as sensible plain text. @stat $128k "MRR" +18% isn't a cryptic macro; it's a line you can understand in a raw log, a git diff, or a terminal. This is the Markdown lineage showing through: the source is never hostage to the renderer.

That also happens to smooth the only realistic path to adoption. Nothing supports Supermark on day one, so most of the time a document will land in a plain Markdown renderer (GitHub, an editor, someone's inbox), and there it degrades to legible text instead of breaking. It reads as lightly-annotated Markdown, which is a far gentler failure than what a Markdown renderer does to a page of HTML.

3. It has to be easy to remember

Most Supermark will be written by models, with a person typing the odd line by hand, and the same property serves both: ambiguous, awkward syntax is syntax that gets produced wrong. The ergonomics of the characters themselves matter.

So the sigils are deliberate. Block-level things that own a whole line start with @: @stat, @bar, @chart. Inline things you drop into a sentence use square brackets, which your eye (and a model's training data) already associates with Markdown links: [badge.green]Live[/]. Variants chain with dots, borrowed straight from CSS and Pug ([badge.green.lg.outline]), because that's a pattern models have seen a million times.

Status: [badge.green]Passing[/] [badge.amber.outline]Flaky[/] [badge.red]Down[/]

Press [kbd]⌘K[/] to search. Rated [stars 4.5] by the team.

Deploying [spinner] please wait…

None of that needs a manual. It reads as what it is.

4. It has to be easy to render, and the renderer decides how it looks

Supermark markup is semantic, not pixel-precise. A @stat says "this is a KPI with a label and a delta." It does not say what colour, font, or corner radius to use. That's the renderer's call.

I care about this one because it's why the examples on this page look like my site and not like a generic component kit. The exact same Supermark, handed to a different renderer, would wear that renderer's design language. The document describes intent; the host owns the aesthetics. It also keeps a renderer small: a line-oriented grammar with a fixed vocabulary is a weekend of parsing work, not a browser engine.

5. The payoff: removing HTML entirely

This is the part I'm proudest of, and it's a consequence rather than a goal I started with. Markdown supports raw HTML passthrough for a reason: the language is thin, so inline HTML is its designed escape hatch for everything it can't express. That escape hatch is also where the trouble lives, and it nags at me more now that a model is usually in the loop. Renderers need a sanitizer standing guard, and Markdown lets you hide things: an HTML comment, a display:none span, a zero-width character. A reader never sees them, but anything parsing the raw text does. That gap between what a person reads and what a machine reads is exactly where prompt injection lives, since a document that looks innocent to you can carry a second set of instructions to the next model that reads it. And models now read documents, skills, and plugins written by strangers all day.

But follow the chain back: the only justification for the escape hatch was the richness Markdown couldn't express, and for the shapes a conversation actually wants, Supermark covers that. So the escape hatch can simply go. Supermark has no raw-HTML passthrough, no comment syntax, no hidden-metadata block. Type <!-- … --> or <script> and it renders as its own literal text, in plain sight. And the language is logic-less by design: no variables, no expressions, no conditionals, no loops. Every value in a document is a literal the model already computed, so there's no machinery for untrusted output to smuggle behaviour through and nothing for a sanitizer to catch. (My renderer builds real DOM nodes and lets the framework escape text, so even a <script> you type into the playground is just... the text <script>.)

To be clear about the scope of the claim: none of this fixes the Markdown pipelines that already exist. GitHub will keep rendering HTML for compatibility forever. The claim is that a new surface (an agent UI, a chat renderer, a harness) that speaks Supermark never has to implement HTML handling at all, because the reason to implement it is gone. Source and rendered output carry the same information, so what you read is what the model actually said. That doesn't make any given document trustworthy, but it takes away the place you'd hide the untrustworthy part.

A quick tour

Beyond the basics, there's a component for most of the shapes a real interface needs. A few favourites:

Callouts, for the four things you're always trying to say. These are GitHub's alerts in spirit, minus the blockquote wrapper:

[!info] Your export is being prepared.
[!ok] Payment received, you're all set.
[!warn] This action can't be undone.
[!err] We couldn't reach the server.

Charts from plain CSV, no charting library, just the shape of the data:

@chart:area title="Signups this week"
Mon,120
Tue,180
Wed,165
Thu,240
Fri,310

Cards and layout, for composing the above into something that feels designed:

>>>
@avatar "RA" teal
**Review agent** found a race condition [badge.red.sm]publisher.ts[/]

Two workers can claim the same event before either marks it
in-flight, so the dedupe check reads stale state under load.

[btn.teal]Show the diff[/]  [btn.gray.outline]Dismiss[/]
<<<

And two flavours of control. UI controls (tabs and toggle groups) hold local view state, so they're useful the instant they render. Input controls (text and date fields, selects, sliders, switches) render as real widgets that an agent harness can wire up to collect a structured answer from the user; on their own they're presentational, and either way the document stays logic-less:

@toggle-group [Chart] Table Raw

@slider "Confidence" 0 100 80
@switch "Email me the results" on

There's more (timelines, kanban boards, calendars, trees, diffs, progress bars), but the fastest way to get a feel for it is to type.

What I changed along the way

Building a real renderer is the fastest way to find the soft spots in a spec, and I hit a few worth calling out.

The original grammar had no way to escape a sigil, which meant you literally couldn't write an article about Supermark, because [badge] always tried to become one. So I added backslash escaping: \[badge] renders as the literal text. Small thing, but you're reading it work right now.

I also rounded out the Markdown superset claim so it's actually true. A first pass had the extras but was missing pieces of plain GitHub-Flavored Markdown a model emits without thinking: images, _underscore_ emphasis, bare-URL autolinks, hard line breaks, and per-column table alignment. Those all work now, so a document that's just Markdown renders exactly as you'd expect, and you only notice Supermark when you reach for something Markdown can't do.

I also loosened the container blocks (cards, collapsibles, and columns), which used to terminate on the first blank line and so couldn't hold more than one paragraph. Now each closes on a matching tag (<<< for a card, +++ for a collapsible) or, for columns, its final ||| fence, so a card or a column can hold real, multi-paragraph content. I decided that mattered more than the tidiness of auto-closing on a blank line.

That same choice, distinct open and close markers, is also what lets containers nest: a card inside a card, an accordion inside an accordion. The parser just depth-counts matched pairs, which is unambiguous and cheap in a line-oriented grammar (it's why every bracket in every language is mirrored rather than symmetric). Lists nest the ordinary Markdown way, by indentation. Flat fences that never nest, like code blocks and dividers, stay symmetric on purpose.

A couple of honest limitations remain. The input controls are display-only in this playground: by design they're meant for a host harness to read values from, and a bare renderer like this one has nothing wired up to collect them. And "streaming-first" is a property of the grammar today more than a feature of this particular renderer. The design supports it; I haven't wired incremental rendering into the playground yet. I'd rather tell you that than pretend.

Prior art

The Markdown-extension lineage at the top of this article is half the picture; the other half comes from the app side. A2UI (Google), Adaptive Cards (Microsoft), and Slack's Block Kit let an agent drive real UI by emitting JSON against a component catalog. They're solving application interfaces, and JSON is a fine call there, but it's expensive per token, unreadable raw, and useless the moment the output needs to live as text in a doc, a diff, or an inbox.

Nothing in either lineage sits where a conversation needs: Markdown-compatible, terse enough to stream cheaply, a fixed vocabulary a model can already half-guess, and readable with no renderer at all. That's the gap Supermark is sketching at. It also composes with what already works: Mermaid lives in a fenced code block, Supermark keeps fenced code blocks, so a renderer that wants rich diagrams can hand a mermaid fence to a Mermaid renderer unchanged.

Where this could go

I'd rather be straight about the state of this. Supermark is a syntax and one reference implementation, nothing more. For it to be useful to anyone but me, the interesting work is all still ahead: parser and renderer libraries in the languages people actually generate text from, plugins for the tools where Markdown already lives (Obsidian, VS Code, the GitHub pipeline), and enough agreement on the syntax that a document written for one renderer looks right in another.

None of that happens without adoption, and adoption isn't something I can will into being from a personal site. GFM proved Markdown grows by small, adoptable increments; it also had GitHub's distribution behind it, which is the part I can't replicate. But if something like this did settle into a convention, the Markdown-heavy surfaces we already stare at all day (agent UIs, docs, issues, chat) could carry a little more structure for a lot less effort, with nobody hand-writing HTML or trusting a model to. This is my sketch of that. If the idea is right and my particular syntax is wrong, I'd be glad to see a better one win.

Try it

The best way to understand Supermark is to break it. There's a live playground where you can type Supermark on one side and watch it render on the other, load a few presets, and watch the token counter tick as you go.

Open the Supermark playground →

I'd love to know what you build with it, where it falls short, or whether you'd solve this problem a different way entirely.

The full syntax

Everything Supermark understands, in one place: terse by design, so a person or a model can write compliant Supermark without hunting through the prose above. Since Supermark is a superset of Markdown, the ordinary Markdown you already know (including GitHub-Flavored extras like tables, task lists, strikethrough, and autolinks) works as-is; what follows adds the layout, data, and component syntax on top.

• I wrote a package that provides validation for Eloquent models.
• It allows you to define the foundational rules, messages, etc. for each model directly within your model classes.
• It has an opt-in feature to perform validation when saving.
• If you need to perform special validation at some point, for example when handling a particular request payload, it's easy to retrieve, extend, or replace the validation logic for a model.
• As of this moment, it's in an alpha release state. Feel free to give it a try, but note that the api may change slightly before it's v1.0 release.

Why use this package?

For a few main reasons:

1. To reduce code duplication
2. To reduce validation whoopsies
3. To ensure data integrity when you can't control all of the times when your model will be filled/saved

Shouldn't validation be handled in Controllers or FormRequests?

Indeed, I think there are plenty of situations where validation needs to be performed in Controllers, FormRequests, and elsewhere (like Nova/Filament fields, console input, etc). However, after working with large applications containing over a hundred models and a ton of input vectors, I have found it's efficient and less buggy to define a foundational set of validation rules for each model and then handle extra validation concerns on a case by case basis wherever necessary.

Here are a few examples of putting that into practice.

• Sometimes, I will write a route to update a model record but I only want to permit the updating of certain attributes. In these situations, I will add the validation logic to my Controller or FormRequest class, but I will do so by grabbing the relevant rules from the model itself ($requestRules = $model->validationRules('attribute_1', 'attribute_2', ...)). By doing so, I can ensure that my $request->validated() array only contains the attributes I want to update and that the rules for each attribute is correct (at least, as correct as I had previously defined within the model).
• There are times when model attributes get a little messy and it can be easy to mistakenly fill a model in a database-valid-yet-logically-invalid way. Consider a hypothetical Order model with shipment_status, tracking_number, carrier, shipped_at, and delivered_at attributes (ok ok - we'd almost certainly want to break this out into related models but go with me for a second). As you can imagine, we may need various implementations to integrate with UPS, USPS, FedEx, DHL, etc and each would fill those variables. Note that our code has nothing to do with user input. Now, would it be possible to write one of these implementations and forget to fill an attribute or parse a date from the carrier's API incorrectly? Sure. Remember: handling validation for user input is essential, but programmers make mistakes too. In an ideal world, you'd have a test to cover yourself, but perhaps you had to rush out a feature and 100% code/state coverage wasn't possible. By defining some model validation rules, you could ensure that you never save a record when the delivered_at is before the shipped_at, or have a record where the tracking_number is filled but the carrier was left null, or have a cancelled shipment_status that left the shipped_at filled.
• When writing a package for others to use, you can't always control how the data will be filled for your packages' models. Thus, having validation at the model level helps to eliminate a footgun for the consumer of your package. If they write a custom admin form that deals with your models, you can be sure that they won't be able to (easily) save records that represent an invalid state.

Get started

Check out the repo and if you like the features & api described in the README, give it a try in your app.

composer require steven-fox/laravel-model-validation

I'm open to feedback and improvements.

Heads Up

With so many variables at play, one must develop some "tweaking judgement" through experience. Durations, ingredient amounts, folding counts of the dough, etc. are constantly being adjusted to suit the air temp, time constraints, flour, freshness of yeast, yada yada. So, I can’t provide a “this is guaranteed to work if you follow this to the minute/gram” but it should still serve as a helpful framework towards good pizza dough.

Notes

• Although this is intended for wood-fired pizzas, I also use this dough for artisan and ciabatta bread frequently.
• I make ~280g dough balls to produce ~12” pizzas.
• Target hydration is 70-75% (mass of water / mass of flour).
  • 75% is softer, fluffier, but HARD to work with (sticking, ripping, & overstretching comes easily).
  • 70% yields similar results but a little more forgiving.
  • If you struggle when working with the dough during pizza prep, you can try to go lower in hydration but anything below like 65% won’t give the epic “soft and crunchy” results.
• Times are very dependent on ambient temp. The poolish will ferment at widly different rates at 66F vs. 76F.
• This recipe uses a poolish preferment. There is classic variation that uses a biga preferment instead, but unless you have a commercial style spiral dough mixer, you'll struggle to mix the biga and final dough.

Recipe

Ingredients

The type of flour makes a notable difference. My go to is Caputo Pizzeria, but I’ve recently started to experiment with Caputo Nuvola Super (some people like this over the Pizzeria flour but I have found it to be stickier thus far) and Caputo Manitoba Oro (very high strength flour that helps bring structure to the dough when using Pizzeria flour poolish preferments).

I use Caputo Semolina flour on my work surface/paddle when spreading/prepping the pizzas.

I use Caputo Lievito yeast but any good instant yeast will work fine.

Use good water - we have a reverse osmosis setup at home and use that.

Poolish Preferment

Target hydration: 100%

• 2 strategies:
  • Very little yeast with 12-16 hour rest at room temp (68-72*F) overnight.
  • Much more yeast with 1hr at room temp and then 24 hours in fridge.

Example ingredient amounts PER PIZZA

• 81.3g room temp water
• 81.3g Pizzeria flour
• Yeast
  • 1/12 teaspoon for room temp / 12hr version
  • 0.5g for 1hr room temp + 24hr fridge version
• 1.6g honey
• No salt

Steps

1. Combine ingredients in a container that has an airtight lid.
2. Mix by hand/spatula until smooth (1 min), but you are not trying to develop gluten structure.
3. Follow resting strategy based on the amount of yeast you used.
4. The poolish is ready when:
   1. Room temp rest: it has doubled/tripled in volume and there are bubbles popping every few seconds at the surface.
   2. Fridge rest: 24 hrs have passed (with this strategy, it will not have tripled in volume but you should still see some gluten development and bubbles throughout).

Mixing the Final Dough

• Target hydration: my go to is 70%, but as described above, somewhere between 65% and 75% will do depending on your level of dedication.
• Flour used: either more Pizzeria or this is where I’m experimenting with the Manitoba flour to have more dough strength… most of the time, I just use more Pizzeria flour.

Example ingredient amounts PER PIZZA

• 32.5g room temp water
• 81.3g flour
• 3.3g fine sea salt
• Ideally, NO yeast. The poolish activity will develop the final dough. However, if you struggle with rise or are in a time crunch, you can add a little bit of yeast (like 1/12tsp per pizza) to the final dough to speed things along.

Steps

1. Pour the water into the poolish to help it separate from the container.
2. Add the flour.
3. (Autolyse 20 min) Mix until combined but don’t worry about developing gluten yet. Cover and rest for 20 min.
4. (Develop) Work the dough. Add the salt during this process (if by hand, immediately; if by mixer, at the halfway point). I have 2 strategies for this:
   1. By hand. Use the pinch-pull-fold technique.
   2. Mixer. Start on a slower speed (2-3?) until the dough is kinda uniform in texture (3-4min). Then speed up (4-6?) to get the dough separating and slapping around the bowl.
5. After the first round of development (maybe 8min), the dough should hold a shape momentarily (when tensioned), by smooth in texture, and have a good amount of strength.
6. Rest for 20 mins.
7. Assess the dough and if needed, work the dough again (common if doing this by hand). Doesn’t have to be long, but you are looking to add tension to the dough so that it will hold a ball longer and longer.
8. Depending on how the dough is forming, you may need to rest and then develop again, especially when working by hand. If using a mixer, you’re probably good.

Shaping and Proofing

At this point, we once again have a couple strategies which differ in time:

1. You intend to make the pizza same day.
2. You intend to wait 24-48hrs before making pizza.

For same day:

1. Rest the final dough for ~1 hr.
2. Divide into ~280g balls.
3. Allow to proof until doubled in size. Time will vary significantly depending on air temp and whether you added additional yeast for the final dough. At 75*F and no addt’l yeast, maybe 4-6hrs.

For long proof

1. Rest the final dough for ~1 hr.
2. Divide into ~280g balls.
3. Immediately put into fridge and allow to rest for 24-48hrs.
4. On the day of pizzas, remove from the fridge about 1-2hrs ahead of time.

This week felt like a good time to give it an update.

Fun fact: I don't know React or Next.js. So, this particular project seemed like a great opportunity to learn something new. After a few days learning the basics of Next.js & React (still feeling partial to Vue) and spending what felt like forever to get the logos sorted out for the home page, I'm ready to publish the first version. I hope you like it!

Along with this update, I also hope to begin publishing a few articles from time to time. Not entirely sure what my focus will be just yet, but that's something we can hone in the coming months.

For those interested, here are a few facts about the new site:

• Based on the Spotlight template from TailwindUI
• Next.js v14
• Tailwind v3
• A few inspirations from Flowbite