fix(knowledge): make connector syncs resilient to removed credentials, partial outcomes and slow completion counts - #8126

waleedlatif1 · 2026-09-22T04:52:32Z

Summary

A content-engine connector whose credential was removed is left unscheduled in error with a reconnect message instead of being re-dispatched every interval and walking the consecutive-failure ladder to disabled. The sync engine short-circuits with a credential_missing skip before taking the lock, and credential deletion writes the same state up front for workspace/admin connectors that are left with no API key either; a members-mode connector only drops its optional dedicated content credential. The connector settings page offers Reconnect for that state
One buildSyncUnscheduledUpdate builder owns the terminal "unscheduled with error, lock released" write used by credential removal, both deleted-knowledge-base skips, the deleted-connector lock release and the capacity rejection
A partial sync with source or dispatch failures is returned from the Trigger task as a normal result (with a warn log) instead of AbortTaskRunError, so a connector that fails a few documents stops paging as a failed run; the engine already persists the partial outcome on the sync log and the connector row
completeSuccessfulSync counts the connector's live documents before taking the completion locks; a database failure of the count keeps the previous document count and corpus size instead of failing a sync whose documents already landed, and a partial doc_connector_live_idx (connector_id where not excluded, archived or deleted) makes the count an index-only scan (migration 0375, concurrent, expand-only)
The permission hold, the "listing incomplete" notice and the content hold are all reported on lastSyncError, one per line, instead of the first masking the rest
Workflow execution admission, child-workflow admission, custom-block child execution, table column enrichment and guardrail model calls read the usage gate through the existing usage-gate-cache (cached admissions, refusals re-read) instead of re-summing the payer's whole billing period on every call; the public v1 knowledge search uses the search entry point like the internal search already does
Integration fixtures give their connector a credential row, since a content-engine connector without one is now skipped by design

Type of Change

Bug fix

Testing

New unit tests for each behaviour (credential-less connector, count ordering, count fallback and non-database rethrow, partial outcome and warn gate, joined holds, deletion scoping, reconnect row, execution gate cache); each new test was mutation-checked by reverting its fix and watching it fail
CI-gated PostgreSQL integration suites for knowledge connectors run locally against a migrated disposable database
bun run type-check, bun run lint, bun run check:audits, bun run check:api-validation, bun run check:migrations origin/staging all pass

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…duled instead of walking the failure ladder to disabled

… of failing the run

…ocks and index the live set

…tead of re-summing the billing period per run

…al removal to content-engine modes, and keep both sync holds visible

vercel · 2026-09-22T04:52:38Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 22, 2026 5:48am UTC

greptile-apps · 2026-09-22T04:57:06Z

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issue remains after the latest recovery-state correction.

Summary

This PR makes connector synchronization more resilient when credentials disappear, individual documents fail, or completion counts are slow. It also consolidates terminal unscheduling behavior, preserves independent synchronization hold notices, adds an index for live-document counting, and routes high-volume usage checks through the existing admission cache.

Credential-less content connectors are left unscheduled with a reconnectable error state.
Partial synchronization outcomes are returned normally while terminal failures still fail the task.
Successful completion counts live documents before acquiring completion locks and tolerates database count failures.
Usage-gate caching is reused across execution, guardrail, enrichment, and public knowledge-search paths.
Integration fixtures and UI recovery behavior are updated for the new credential requirements.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Connector sync requested] --> B{Authentication source present?}
  B -- No --> C[Conditionally mark connector error]
  C --> D[Clear schedule and sync lock]
  D --> E[Expose Reconnect action]
  B -- Yes --> F[Acquire sync lock]
  F --> G[Run listing and content pass]
  G --> H{Outcome}
  H -- Failed --> I[Persist failure and fail task]
  H -- Partial --> J[Persist partial result and warn]
  H -- Successful --> K[Count live documents]
  K --> L{Database count available?}
  L -- Yes --> M[Persist count and successful state]
  L -- No --> N[Keep previous count and persist successful state]

_{Reviews (6) · Last reviewed commit: "fix(knowledge): stop offering reconnect ..."}

cubic-dev-ai

1 issue found across 28 files

Confidence score: 3/5

In apps/sim/lib/knowledge/connectors/sync-engine.ts, a stale sync task can race with reconnection, mark the newly connected row unscheduled, and clear its next run, potentially preventing future synchronization. Recheck the credential-less state in the update predicate before applying the stale-task update.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/knowledge/connectors/sync-engine.ts">

<violation number="1" location="apps/sim/lib/knowledge/connectors/sync-engine.ts:852">
P2: If a connector is reconnected after the initial read but before this update, the stale sync task marks the newly connected row unscheduled and clears its next run. Recheck the credential-less state in the update predicate or re-read before applying the terminal write.</violation>
</file>

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…schedule only connectors left without an auth source, and give fixture connectors a credential

waleedlatif1 · 2026-09-22T05:18:18Z

@greptile

waleedlatif1 · 2026-09-22T05:18:19Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T05:18:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 42 files

_{Requires human review: Auto-approval blocked because this review re-detected 2 unresolved issues already reported by Cubic.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

… the unscheduled state only over the row the run observed

waleedlatif1 · 2026-09-22T05:25:03Z

@greptile

waleedlatif1 · 2026-09-22T05:25:05Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T05:25:11Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 43 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…tial is missing

waleedlatif1 · 2026-09-22T05:32:28Z

@greptile

waleedlatif1 · 2026-09-22T05:32:29Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T05:32:33Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 44 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…g terminal write

waleedlatif1 · 2026-09-22T05:37:40Z

@greptile

waleedlatif1 · 2026-09-22T05:37:41Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T05:37:46Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 44 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…ential again

waleedlatif1 · 2026-09-22T05:48:51Z

@greptile

waleedlatif1 · 2026-09-22T05:48:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-22T05:48:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 44 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 added 5 commits September 21, 2026 21:24

fix(knowledge): leave a connector whose credential was removed unsche…

3548b25

…duled instead of walking the failure ladder to disabled

fix(knowledge): report a partial connector sync as an outcome instead…

0e6fd98

… of failing the run

fix(knowledge): count a connector's documents before the completion l…

35a1c58

…ocks and index the live set

fix(billing): serve execution admission from the usage gate cache ins…

6a0469e

…tead of re-summing the billing period per run

fix(knowledge): share the unscheduled connector write, scope credenti…

81d1ae9

…al removal to content-engine modes, and keep both sync holds visible

vercel Bot deployed to Preview September 22, 2026 04:54 View deployment