feat(jev): add TypeSafe evaluation models to Agent - #8103

waleedlatif1 · 2026-09-21T08:07:18Z

Summary

Add TypeSafe Jev as a native evaluation provider in the Agent model selector, using State and typed Choice, Score, and Noul questions.
Support hosted key rotation, normal model billing, and workspace/organization BYOK overrides. Default to jev-latest while retaining pinned and preview options.
Expose typed answer fields through the shared reference picker and preserve whole-object selection.
Reuse Agent permissions, secret projection, streaming, logs, and usage tracking. Include the official TypeSafe icon, brand color, bounded transport, retries, and documentation.

Type of Change

New feature

Testing

3,441 tests passed across provider, credential, configuration, Agent, output-discovery, variable-resolution, and BYOK settings suites; five existing tests skipped.
Six authenticated live checks passed through the real provider and key resolver: all three Jev IDs through serialized workflow execution with hosted keys, streaming billing, stored-BYOK precedence and zero model charges, and invalid credentials. Database/auth infrastructure was mocked in this harness.
Five signed-in local browser workflow runs passed, including nested answers consumed by a downstream Function.
App, deployment-config, and setup type-checks; cleanup; lint; all 47 audits; block registry; and generated documentation checks passed.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-21T08:07:22Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 22, 2026 7:34pm UTC

waleedlatif1 · 2026-09-21T08:07:53Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T08:07:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

greptile-apps · 2026-09-21T08:11:45Z

Confidence Score: 5/5

The PR appears safe to merge; no actionable issue remains in the changes made since the previous review.

Summary

This PR adds TypeSafe Jev as a native Agent evaluation provider, including typed evaluation inputs and outputs, hosted and BYOK credential resolution, billing and usage integration, bounded provider transport, model registration, documentation, and regression coverage. Since the previous review, the provider credential subblocks were moved directly after the model selector so the API key appears before Jev’s State and Questions fields, with the corresponding test expectation updated.

Registers pinned, latest, and preview Jev models for Agent evaluation workflows.
Adds typed Choice, Score, and Noul question handling with structured answer discovery.
Integrates hosted key rotation, workspace and organization BYOK precedence, usage accounting, and provider branding.
Separates evaluation execution from conversational Agent settings and rejects evaluation models as chat fallbacks.
Adds provider, handler, output-discovery, credential, transport, and configuration tests.

Diagram

sequenceDiagram
  participant W as Workflow Agent
  participant H as Agent Handler
  participant K as Key Resolver
  participant T as TypeSafe Provider
  participant J as Jev API

  W->>H: model + state + typed questions
  H->>K: Resolve TypeSafe credential
  K-->>H: Workspace/org BYOK or hosted key
  H->>T: Native evaluation request
  T->>J: Bounded request with retries
  J-->>T: Typed answers and usage
  T-->>H: content + answers + tokens
  H-->>W: Structured Agent outputs

_{Reviews (14) · Last reviewed commit: "improvement(agent): place credentials be..."}

cubic-dev-ai

All reported issues were addressed across 25 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-21T08:22:20Z

@greptile

waleedlatif1 · 2026-09-21T08:22:21Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T08:22:27Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 25 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-09-21T16:01:17Z

@greptile

waleedlatif1 · 2026-09-21T16:01:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T16:01:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 25 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-09-21T16:09:54Z

@greptile

waleedlatif1 · 2026-09-21T16:09:55Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T16:10:03Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 25 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-21T16:17:56Z

@greptile

waleedlatif1 · 2026-09-21T16:17:57Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-21T16:18:04Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 25 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-22T19:23:22Z

@greptile

waleedlatif1 · 2026-09-22T19:23:24Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T19:23:29Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 43 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

waleedlatif1 · 2026-09-22T19:34:45Z

@greptile

waleedlatif1 · 2026-09-22T19:34:46Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T19:34:53Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 43 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic}

vercel Bot deployed to Preview September 21, 2026 08:09 View deployment

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/tools/jev/evaluate.ts Outdated

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/tools/jev/choice.ts Outdated

Comment thread apps/sim/tools/jev/evaluate.ts Outdated

Comment thread apps/sim/tools/jev/shared.ts Outdated

Comment thread apps/sim/providers/typesafe/schema.ts

vercel Bot temporarily deployed to Preview September 21, 2026 08:21 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 21, 2026 16:01 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/blocks/blocks/jev.ts Outdated

vercel Bot temporarily deployed to Preview September 21, 2026 16:09 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/blocks/blocks/jev.ts Outdated

vercel Bot temporarily deployed to Preview September 21, 2026 16:17 Inactive

cubic-dev-ai Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread apps/sim/blocks/blocks/jev.ts Outdated

waleedlatif1 mentioned this pull request Sep 21, 2026

feat(typesafe): add Jev decision integration #8119

Closed

10 tasks

waleedlatif1 force-pushed the codex/jev-integration branch from b3fc80d to 9cce6a8 Compare September 22, 2026 06:39

vercel Bot temporarily deployed to Preview September 22, 2026 19:11 Inactive

waleedlatif1 added 13 commits September 22, 2026 12:11

feat(jev): add TypeSafe decision integration

057bef1

fix(jev): validate batch answers against requested questions

9d51953

improvement(jev): refine criteria helpers and suggested skills

5b473f5

fix(jev): sanitize generated JSON inputs

1209fed

fix(jev): clarify empty Noul canvas sentence

f592d2d

fix(jev): preserve Agent tool criteria and rubric schema

8cc956b

fix(jev): expose text inputs to Agent models

e654323

feat(jev): integrate native evaluation into Agent providers

8b6b9a4

fix(jev): harden evaluation transport and output discovery

cc5e8f6

fix(jev): bound evaluation payloads before serialization

0c179f7

chore(jev): use the standard hoisted test mock

a8b48e1

fix(jev): expose typed answer fields in reference pickers

5d8a181

feat(jev): support hosted TypeSafe keys and default to latest

5ee7e31

waleedlatif1 force-pushed the codex/jev-integration branch from 4bc50f2 to 5ee7e31 Compare September 22, 2026 19:23

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

vercel Bot deployed to Preview September 22, 2026 19:25 View deployment

greptile-apps Bot reviewed Sep 22, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/blocks/block-outputs.ts

improvement(agent): place credentials beside the model selector

b6c725e

vercel Bot temporarily deployed to Preview September 22, 2026 19:34 Inactive

cubic-dev-ai Bot reviewed Sep 22, 2026

View reviewed changes

waleedlatif1 merged commit 3e1e1fc into staging Sep 22, 2026
25 checks passed

waleedlatif1 deleted the codex/jev-integration branch September 22, 2026 19:37

waleedlatif1 mentioned this pull request Sep 22, 2026

v0.8.56: opus 5.5, jev, gpt 6 sol, ui consolidation, usage entry #8150

Merged

Conversation

waleedlatif1 commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

greptile-apps Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

waleedlatif1 commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 21, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

waleedlatif1 commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 22, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Sep 21, 2026 •

edited

Loading

vercel Bot commented Sep 21, 2026 •

edited

Loading

greptile-apps Bot commented Sep 21, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading