pijev — Permutation Invariant Jev

Better Jev probabilities with theoretical guarantees—all with a one-line import change.

pijev turns Jev's option-order variations into a single averaged prediction, with Brier score and log loss guaranteed no worse than the average across the included orderings. All permutations run in one system_one request. Keep the same calls and official response types; pijev handles the rest.

The same options can receive different probabilities when reordered. This official SDK call sends two Choices with identical instructions and descriptions, but opposite option orders:

from typesafe_sdk import Choice, TypeSafeClient

instructions = "Which team should handle this support ticket? Choose the best primary team."

with TypeSafeClient() as client:
    response = client.system_one(
        state={
            "ticket": "Since upgrading my plan yesterday, I cannot access the "
                      "dashboard. The payment went through, but the page says "
                      "my subscription is inactive. Please fix this."
        },
        questions={
            "forward": Choice(
                instructions=instructions,
                criteria={
                    "billing": "Payments, charges, invoices, and refunds.",
                    "technical": "Product failures, errors, and troubleshooting.",
                    "account": "Login, account access, and account settings.",
                },
            ),
            "reverse": Choice(
                instructions=instructions,
                criteria={
                    "account": "Login, account access, and account settings.",
                    "technical": "Product failures, errors, and troubleshooting.",
                    "billing": "Payments, charges, invoices, and refunds.",
                },
            ),
        },
    )

print(response.choices["forward"].probabilities)
print(response.choices["reverse"].probabilities)

One request, two questions, one shared state. A live jev-1.13.0 run returned:

Both selected billing, but its probability changed by 20 percentage points. This run also changes question IDs and does not control model randomness; see the experiment details.

pijev treats option order as a nuisance variable: evaluate different orderings, align predictions by label, and average their probabilities. The highest mean probability determines the returned choice.

For three options, there are six permutations. Here is a separate live jev-1.13.0 run with all six in one request. A = account, B = billing, T = technical.

The pijev mean column averages all six predictions: billing receives 0.555, compared with 0.46–0.64 across individual orderings. Reproduce this run · Recorded results

Ask once using the same SDK interface; pijev handles the six-question batch and returns one aggregated answer:

from pijev import Choice, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state={
            "ticket": "Since upgrading my plan yesterday, I cannot access the "
                      "dashboard. The payment went through, but the page says "
                      "my subscription is inactive. Please fix this."
        },
        questions={
            "team": Choice(
                instructions="Which team should handle this support ticket? "
                             "Choose the best primary team.",
                criteria={
                    "billing": "Payments, charges, invoices, and refunds.",
                    "technical": "Product failures, errors, and troubleshooting.",
                    "account": "Login, account access, and account settings.",
                },
            ),
        },
    )

answer = response.choices["team"]
print(answer.choice)
print(answer.probabilities)

The result is an official SystemOneResponse with your original question names. Live results may differ from the recorded run.

Install from PyPI:

pip install pijev

Set your API key:

export TYPESAFE_API_KEY=your-key

Change only the import:

-from typesafe_sdk import TypeSafeClient, Choice, Noul, Score
+from pijev import TypeSafeClient, Choice, Noul, Score

Set an optional permutation budget:

client = TypeSafeClient(n_permutations=16, seed=42)

The default is up to 8 distinct permutations per Choice. Smaller sets use all permutations; larger sets use uniform sampling without replacement. Set n_permutations="all" for full enumeration. The seed controls local sampling only. Batches are capped at 720 expanded questions and remain subject to server limits.

Async usage works the same way: import AsyncTypeSafeClient from pijev, then use async with and await client.system_one(...) as usual.

Permutation averaging has a proper-scoring-loss guarantee, not an unconditional calibration guarantee. Let $p^{(m)}$ be the label-aligned probability vector for permutation $m$, and let $y$ be the same correct answer under every ordering. Define

$$ \bar p = \frac{1}{M}\sum_{m=1}^{M}p^{(m)}. $$

For any loss convex in the predicted probabilities, Jensen's inequality gives

$$ L(\bar p,y) \leq \frac{1}{M}\sum_{m=1}^{M}L(p^{(m)},y). $$

This holds for the actual set of predictions being averaged, including a sampled subset of permutations. Two useful cases are:

• Log loss: $-\log\bar p_y \leq \frac{1}{M}\sum_m -\log p_y^{(m)}$.
• Brier score: writing $e_y$ for the one-hot correct answer, the improvement equals the disagreement removed by averaging:

$$ \frac{1}{M}\sum_m|p^{(m)}-e_y|_2^2 - |\bar p-e_y|_2^2 = \frac{1}{M}\sum_m|p^{(m)}-\bar p|_2^2 \geq 0. $$

The guarantee compares the average prediction with randomly choosing one of the included predictions. Brier improvement is strict when those predictions differ. For uniform sampling and a fixed predictor, the same inequality holds in expectation against a uniformly random ordering. Full enumeration gives the group average; see Chen, Dobriban, and Lee for related symmetry-averaging theory.

For $K$ options and budget $B$, pijev evaluates $M=\min(B,K!)$ permutations in one HTTP request. Multiple Choices contribute their own permutations; Noul and Score each add one unchanged question.

The state is sent once. For state size $S$ and question size $Q$, request content is approximately $S+MQ$, versus $M(S+Q)$ for separate calls, excluding protocol overhead. This is a payload estimate; use response.usage for reported tokens.

At TypeSafe's published rate of $0.042 per million input tokens, with free output tokens (checked 2026-09-22):

$$ C = \frac{T}{10^6}\times 0.042 $$

Here, $C$ is the estimated cost in USD and $T$ is response.usage.input_tokens.

These estimates use reported usage and the published rate, not invoice charges. One request still contains multiple decisions; latency and billing depend on the backend, and retries may add requests.

• Only Choice answers are averaged. Noul and Score pass through unchanged.
• confidence is the selected label's mean probability, not Jev's native confidence score.
• Sampling approximates full permutation averaging; it does not guarantee exact order invariance or better calibration. The live examples demonstrate probability differences, not accuracy or calibration gains.
• Official interface: TypeSafe Python SDK. This is a third-party extension, not an official TypeSafe project.

Apache License 2.0.