fix(file-search): avoid foreground GIN cleanup stalls

[icecrasher321]

Summary

• Avoid foreground GIN cleanup stalls by writing directly to the existing chunk index in 128 KiB batches. File coverage, regex matching, and line numbers stay the same.
• Disable deferred index writes and drain the existing pending list in a replayable migration, without rebuilding the index or deleting content.
• Give indexing separate statement and transaction deadlines so slow statements cancel and roll back before PostgreSQL terminates the connection.

Type of Change

• Bug fix

Testing

• 55 integration checks passed on each of PostgreSQL 16 and 17, covering migration replay, pending-list draining, connection survival after cancellation, complete-file publication, regex/line numbers, and dispatch.
• 152 targeted unit tests; app and database type checks; full lint; all 47 repository audits; migration safety check.
• Full migration runner on an empty disposable PostgreSQL 17 database, followed by replay; schema generation reports no drift.
• Synthetic ten-writer runs passed 160 prose batches and 160 dense-text batches on the smaller fixture. On the larger dense fixture with verified cache settings, all 80 batches passed (p95 9.34 s, max 9.54 s on two CPUs), leaving zero pending pages. Direct GIN writes trade bulk throughput for avoiding shared pending-list cleanup. The deliberately cache-constrained dense-text fixture still hits statement timeouts; this change does not remove CPU/I/O capacity limits, and deployment needs insert-latency verification under the real workload.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 19, 2026 12:55am UTC

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no actionable regression or outstanding blocking issue was established.

Summary

This PR changes workspace-file indexing to write directly to the existing GIN index, reduces write-batch size, gives indexing transactions separate statement and transaction deadlines, and adds replayable pending-list drainage. Changes since the previous review also harden knowledge-document recovery and add bounded provider diagnostics.

• Disables GIN deferred updates and drains existing pending entries without rebuilding the index.
• Limits direct-index insert batches to 128 KiB and preserves atomic publication.
• Adds liveness-aware, generation-fenced document-processing recovery.
• Projects bounded OCR and OAuth failure diagnostics without retaining provider body content.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Apply ALTER INDEX fastupdate off] --> B[Commit metadata change]
  B --> C[Drain existing GIN pending list]
  C --> D[Workers insert 128 KiB batches]
  D --> E{Statement completes within 10s?}
  E -- Yes --> F[Commit batch]
  E -- No --> G[Cancel and roll back batch]
  F --> H{All chunks stored?}
  H -- Yes --> I[Publish complete file revision]
  H -- No --> D
  G --> J[Retry through a fresh fenced build]

Loading

_{Reviews (2) · Last reviewed commit: "fix(file-search): create chunk GIN index..."}

[cubic-dev-ai]

All reported issues were addressed

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic</sub>

[icecrasher321]

@greptile

[icecrasher321]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@icecrasher321 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 8 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>