How Jev is different from an LLM

Jev is made by TypeSafe. An LLM generates text: you ask it something, it writes an answer, and if you want structured data back you have to ask for it and hope. Jev doesn't generate anything. You give it some state and a question, and it gives you back a number.

TypeSafe calls these System One models. They read natural language like an LLM does, but instead of writing a reply they pick between answers that you define up front. The probabilities are calibrated, which means they're trained against real outcomes, so across a batch of answers a 0.9 should be right about nine times out of ten.

In practice that means no prose to parse, and no prompt asking the model to please respond with valid JSON.

The Syntax folks made a video explaining it:

Noul, Choice and Score

You define what the answers can be using one of three question types.

A Noul is a yes or no question. The answer is a single number: the probability that the answer is yes. Here's how you ask one:

use Laravel\Ai\Classification;
use Laravel\Ai\Classification\Boolean;

$result = Classification::of('I have asked three times now. Can I please talk to a real person?')
    ->question('urgent', new Boolean('Does this request need an immediate response?'))
    ->classify();

$result['urgent']->probability;            // 0.94
$result['urgent']->isTrue(threshold: 0.8); // true

Notice that you get back 0.94 instead of true. You decide where the cutoff is, which means you can set a different one per question.

A Choice picks one option out of a set that you name. Here's an example:

use Laravel\Ai\Classification\Choice;

$result = Classification::of('My card was charged twice for order A-104. Please refund the duplicate.')
    ->question('department', new Choice('Which team should handle this request?', [
        'billing' => 'Payments, invoices, and refunds',
        'technical' => 'Bugs, outages, and integrations',
        'sales' => 'Pricing, plans, and upgrades',
    ]))
    ->classify();

$result['department']->choice;                   // 'billing'
$result['department']->probabilityOf('billing'); // 0.87
$result['department']->confidence;               // 0.82

Next to the option it picked, you also get a probability for every option, and a confidence score that tells you how concentrated those probabilities were.

A Score rates something against levels that you describe yourself. You could ask how frustrated a customer is, where 0 is calm, 1 is frustrated and 2 is very angry. The answer can land between two levels, so a 1.4 is a perfectly good answer.

The state doesn't have to be a string. When a decision depends on more than one thing, you can pass an array so every part has a name:

Classification::of([
    'subject' => 'Duplicate charge',
    'message' => 'My card was charged twice for order A-104.',
    'order' => ['id' => 'A-104', 'charges' => [49, 49]],
    'refund_policy' => 'Duplicate charges are eligible for a refund.',
])->question('refund_due', new Boolean('The policy entitles this customer to a refund.'))
    ->classify();

That's still one state, even though it holds a message, an order and a policy.

You configure Jev like any other provider in config/ai.php, with a TYPESAFE_API_KEY in your env file.

The problem we wanted to solve

You might have noticed that we launched There There yesterday as well, our new helpdesk. A lot of the mail that arrives in a helpdesk isn't from a customer. Out-of-office replies, bounces, subscription confirmations, DMARC reports. You don't want those in your inbox, and you don't want to pay an LLM to write a title and a summary for each one.

Some of that mail says what it is in the headers. Auto-Submitted, an empty return path, a sender called mailer-daemon. Checking those costs nothing, so we do that first.

Plenty of mail servers don't set those headers. For those we had a list of subject prefixes: Automatische Antwort, Réponse automatique, Out of office, in fifteen languages. Nine more for bounces. On top of that, rules so that a customer asking a question about auto-replies didn't get treated as one.

Every time we found mail that the list missed, we added another string to it.

Replacing the list

The header checks still run first. Everything they can't answer goes to Jev.

We describe each question once, as a case on an enum that also carries its own threshold. That way adding a fourth question is a single case instead of an edit in four files.

enum InboundJudgement: string
{
    case IsAutoResponse = 'is_auto_response';
    case IsBounce = 'is_bounce';
    case IsSpam = 'is_spam';

    public function question(): Boolean
    {
        return match ($this) {
            self::IsAutoResponse => new Boolean(
                'A system sent this mail on its own, rather than a person choosing to write to us.',
                [
                    'true' => 'Sent on a trigger with no human involved at send time: out-of-office
                        notices, delivery reports, ticket acknowledgements, subscription
                        confirmations, digests and alerts. Wording composed in advance still counts',
                    'false' => 'A person sat down and sent this. Still false when a contact form or
                        chat widget wrapped their words in a template and added lines such as Name,
                        E-mail or Subject',
                ],
            ),
            self::IsSpam => new Boolean(
                'This mail is unsolicited bulk mail, a scam, or phishing rather than a genuine
                    message from a customer.',
                [
                    'true' => 'Cold sales outreach, marketing blasts, scams, phishing, or anything
                        the recipient never asked for',
                    'false' => 'A real person writing about the product, their account, or their own
                        support request, however brief or badly written',
                ],
            ),
            // ...
        };
    }

    public function threshold(): float
    {
        return match ($this) {
            self::IsAutoResponse => 0.75,
            self::IsBounce, self::IsSpam => 0.9,
        };
    }
}

Those true and false descriptions are optional, but I'd recommend writing them. They do more work than the question above them.

All the questions the headers couldn't answer go out in one request. Jev reads the state once and answers them in parallel, and you only pay for input tokens, so asking three questions costs the same as asking one. Here's the action that does it:

public function execute(Message $message, Ticket $ticket, Workspace $workspace): void
{
    $judgements = array_filter(
        InboundJudgement::cases(),
        fn (InboundJudgement $judgement) => ! $judgement->settledByHeaders($message),
    );

    if ($judgements === []) {
        return;
    }

    try {
        $response = Classification::of([
            'subject' => $ticket->subject,
            'from_name' => $message->author_name,
            'from_email' => $message->author_email ?? $ticket->contact?->email,
            'message' => Str::limit($message->body_text, 10_000),
        ])
            ->questions($this->questionsFor($judgements))
            ->timeout(10)
            ->classify();

        $verdicts = $this->verdicts($judgements, $response);
    } catch (Throwable $exception) {
        Log::warning('Could not classify an inbound message.', [
            'message_id' => $message->id,
            'error' => $exception->getMessage(),
        ]);

        return;
    }

    $message->updateQuietly([...$verdicts, 'classification' => $response->answers]);
}

There are two things in there I'd suggest copying. The whole call sits in a try block that logs the problem and moves on, because a classification is a nice to have and it shouldn't be able to break the mail pipeline it's helping. And we truncate the message, because an inbound mail can be megabytes long and nothing past the first part of it changes what the mail is.

Turning the answers into booleans is where each question's own threshold is applied. We store the raw probabilities next to them, so we can change a threshold later and see what it would have done:

private function verdicts(array $judgements, ClassificationResponse $response): array
{
    $verdicts = [];

    foreach ($judgements as $judgement) {
        $answer = $response->answer($judgement->value);

        $verdicts[$judgement->value] = $answer->isTrue($judgement->threshold());
    }

    return $verdicts;
}

Those verdicts are stored on the message. When a customer builds a workflow in There There with an "Is spam" condition, checking that condition reads a single column and doesn't call Jev at all.

In closing

I like that Jev does one small thing. It gives you a number and leaves the rest of the decisions in your own code, where you can read them and write tests for them.

It's also fast and cheap enough that you don't really have to think about it. Classifying a mail with three questions at once takes 639ms, and we get around 48 per second when we run them in parallel. We didn't spend any time tuning that, so I'm sure you could get more out of it, but for what we're doing it's fast enough. Jev costs $0.042 per million input tokens and output tokens are free, which for us comes down to four hundredths of a cent per mail, or about 36 cents a month.

We have a list of other places where we want to use this in There There, and in our other products. Expect more Jev powered features soon.

If you want to read more, there are the TypeSafe docs and the Laravel AI SDK. And if you'd like to see the spam detection at work, you can try There There.