improvement(knowledge): rank organization keyword search with Tin - #8007

waleedlatif1 · 2026-09-19T07:56:55Z

Summary

A member whose reach exceeds the permitted-set limit searched keywords through the GIN projection, which scores every chunk matching the term before access is checked. On a production-sized copy a common word cost 5–6.5s of server time and sometimes timed out empty
Where the database provides the tin extension, a BM25 projection of organization-search-index chunks (embedding_keyword_tin) is kept by an embedding trigger. The keyword leg ranks with Tin, checks access on the top 2,000 ranked chunks, widens to 10,000 / 50,000 while too few are readable, and leaves the page to GIN if the widest window cannot fill it
Queries are analyzed by the same websearch_to_tsquery('english') as the GIN path and translated to TINQL (AND/OR/AND NOT/phrases/THEN/N); shapes TINQL cannot express (pure negation, negation inside OR, prefix matches) keep GIN
Gated by the knowledge-tin-keyword flag (default off), a valid Tin index (built only after the backfill), every base being a search index, and no tag filters. Bounded members keep the permitted-set direct scan
Script migration 0019_tin_keyword_projection: installs the extension, functions, and trigger; backfills in keyset pages; builds the index CONCURRENTLY (repairing an invalid one). A no-op where tin is unavailable, and it skips instead of failing the deploy if the database refuses CREATE EXTENSION
New diagnostics: keyword.tin stage, keywordRanking: 'tin' | 'gin', keywordTinWindow

Measured on a production-sized copy (same cluster size, production JIT/planner settings)

Server critical path (permitted-set resolution + the slower retrieval leg), p50 over 4 queries each, a member reaching ~36% of the index:

	before planner	staging	this PR
common term	5.4s	6.5s	0.35s
medium term	2.0s	0.63s	0.29s
rare term	2.1s	0.31s	0.30s

Keyword leg for common terms: 4.9–6.4s → 67–185ms; every query filled 20 results
Translator vs real Tin, 24 queries per shape: 0 errors; identical match counts for single/AND/NOT, 23/24 for OR; top-20 ranking 10–100× faster (single-word p50/max 2/5ms vs 40/504ms)
Tin phrase matching is slightly looser: dropped stopwords leave no gap, so "data pipeline" also matches "data for the pipeline"
Write cost (warm, alternating, 200-row batches): insert median 2.7 → 3.5 ms/row; enabled updates median 2.8 → 5.2 ms/row with occasional 7–12 ms spikes
Migration on the copy: backfill ~60 min, concurrent index build 70s (the migrations job allows 300 min)

Rollout

Deploy; the script migration installs and backfills the projection
Confirm the index is valid, then enable knowledge-tin-keyword in AppConfig

Migration regenerated as 0367 on top of #8005's 0366.

Type of Change

Improvement

Testing

Translator (15), readiness gates (6), keyword planner (window widening, exhaustion, GIN fallback, bounded/tag-filter/not-ready bypass, parenthesized operator) — each checked to fail when its behavior is reverted
lib/knowledge + lib/core/config + connectors (5,668 passing), app and db type-check, bun run lint, check:audits, check:migrations origin/staging
Real-Tin validation and profiling on the production-sized copy (above)

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-09-19T07:57:01Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 19, 2026 5:03pm UTC

cubic-dev-ai

All reported issues were addressed across 18 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

greptile-apps · 2026-09-19T08:01:29Z

Confidence Score: 5/5

The PR appears safe to merge; the previous projection-consistency issue is resolved and no new actionable defect was established.

Summary

This PR introduces feature-gated Tin/BM25 ranking for large-scope organization keyword searches, with bounded candidate-window widening and a complete GIN fallback path.

Since the previous review, the migration was strengthened to:

acquire per-base membership locks before each backfill page is reread and projected;
avoid restoring projection rows when a base is demoted during backfill;
remove demoted-base projections through the existing embedding base index;
add PostgreSQL coverage for demotion during backfill and index-backed cleanup.

The previously reported promoted-base projection gap is no longer outstanding: the current knowledge-base trigger projects existing chunks on promotion and removes them on demotion, with advisory locking coordinating marker changes, embedding writes, and backfill pages.

Diagram

sequenceDiagram
  participant Search
  participant Readiness
  participant Tin as Tin projection/index
  participant Access as Document access checks
  participant GIN as GIN keyword search

  Search->>Readiness: Check flag, valid Tin index, and base eligibility
  alt Tin unavailable or query unsupported
    Readiness-->>Search: Use GIN
    Search->>GIN: Rank all matching chunks
  else Tin ready
    Readiness-->>Search: Return translated TINQL
    loop Windows 2,000 / 10,000 / 50,000
      Search->>Tin: Rank top window
      Search->>Access: Filter ranked documents by visibility
      Access-->>Search: Readable page
    end
    alt Page filled or all matches exhausted
      Search-->>Search: Return Tin-ranked page
    else Widest window cannot fill page
      Search->>GIN: Fall back to complete GIN ranking
    end
  end

_{Reviews (3) · Last reviewed commit: "fix(knowledge): serialize the Tin backfi..."}

waleedlatif1 · 2026-09-19T08:13:51Z

@greptile

waleedlatif1 · 2026-09-19T08:13:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T08:13:59Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 24 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-19T08:23:05Z

@greptile

waleedlatif1 · 2026-09-19T08:23:06Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-19T08:23:11Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 24 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

A member who reaches more of an organization search index than an exact ranking can afford searched keywords through the GIN projection, which scores every chunk matching the term before access is checked, so a common word cost seconds. Where the database provides the tin extension, a BM25 projection of search-index chunks is kept by embedding and knowledge-base triggers, and the keyword leg ranks with Tin first and checks access only on the top of that ranking, widening the window while too few ranked chunks are readable and leaving a page to GIN if the widest window cannot fill it. The query is analyzed by the same websearch_to_tsquery as the GIN path and translated to TINQL; shapes TINQL cannot express keep GIN. The path is gated by the knowledge-tin-keyword flag, a valid Tin index (built only after the backfill completes), and every base being a search index. Script migration 0019 installs the projection only where tin is available and creatable, so self-hosted databases keep an empty table and the GIN path.

vercel Bot deployed to Preview September 19, 2026 07:58 View deployment

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread apps/sim/lib/knowledge/search/tin-query.ts

Comment thread packages/db/script-migrations/0019_tin_keyword_projection.ts

Comment thread apps/sim/lib/knowledge/search/queries.ts Outdated

Comment thread packages/db/script-migrations/index.ts

Comment thread packages/db/schema.ts

greptile-apps Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread packages/db/script-migrations/0019_tin_keyword_projection.ts

vercel Bot deployed to Preview September 19, 2026 08:15 View deployment

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread packages/db/schema.ts

Comment thread packages/db/script-migrations/0019_tin_keyword_projection.ts Outdated

Comment thread apps/sim/lib/knowledge/search/queries.ts

vercel Bot deployed to Preview September 19, 2026 08:24 View deployment

cubic-dev-ai Bot reviewed Sep 19, 2026

View reviewed changes

waleedlatif1 force-pushed the improvement/search-tin-keyword branch from 4119ea6 to 844e40e Compare September 19, 2026 17:02

vercel Bot deployed to Preview September 19, 2026 17:03 View deployment

waleedlatif1 merged commit eaa9ff0 into staging Sep 19, 2026
33 of 34 checks passed

waleedlatif1 mentioned this pull request Sep 19, 2026

v0.8.48: search improvements, file versions #8008

Closed

waleedlatif1 deleted the improvement/search-tin-keyword branch September 20, 2026 00:00

waleedlatif1 mentioned this pull request Sep 21, 2026

v0.8.48: prewarm, tin pg extensions, knowledge search speedups, ui consolidation, keyword search speedups #8096

Merged

Conversation

waleedlatif1 commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Measured on a production-sized copy (same cluster size, production JIT/planner settings)

Rollout

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

waleedlatif1 commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 19, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 19, 2026 •

edited

Loading

vercel Bot commented Sep 19, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 19, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading