# Omni-search groundwork

Status: implemented portable lexical and opt-in semantic services, with hosted Workers AI and four Vectorize bindings; no production sign-off. Reviewer/date/outcome belong in DEVLOG.md. This specification does not authorize publication or change the security release gates.

## Identity and authority

A unified item document is `{id,owner,source:{kind,id},revision,type,title,text,tags,classification,createdAt,acl:{readers},quarantined,deleted,embeddingAllowed}`. IDs are stable within an owner; namespaced IDs such as `message:<source-id>` avoid collisions across item kinds. Revisions are positive, monotonically increasing integers retained through deletion and recreation. Owner is mandatory. An explicit owner entry in `acl.readers` is required for search; ownership alone does not grant access. Current search is an authenticated owner's view; cross-account sharing is not implemented by treating a reader as another owner.

The authoritative source holds item state, permissions and source revision. Search documents, FTS rows, vectors and pagination sessions are derived. A cached/indexed ACL never grants access. Each candidate must have the requested owner before authoritative lookup. Before returning title, tags, classification or a snippet, the service loads that item from the authority, requires its exact revision, checks owner and readers, rejects deletion/quarantine and reapplies filters. Snippets come only from that authorized live document. This applies equally after restart and to later cursor pages. An index containing revoked content can lose recall until reindexing; it cannot grant access to that content.

The runtime holds its per-owner read/mutation gate throughout a search response so authorization remains consistent across asynchronous candidate checks. A missing authoritative record must never fall back to a derived inbox catalog. Null means the authoritative item is absent; temporarily unavailable or shelved authority must throw a retryable error, or ingestion must be deferred until source restoration. Pending upserts must not be converted into tombstones merely because cold metadata is unavailable. Cold shelving must preserve/recover pending work without granting authority to cached indexes.

`authority.get(owner,id,{purpose,revision})` must use indexed/direct source lookup and return a unified document or null. The purpose is `index` during extraction, `search` during final result authorization, and `authorize` after an embedding await. For search, first verify live source metadata/ACL/quarantine and exact revision, then the adapter may use `index.readContent(owner,id,revision)` instead of rereading an immutable source body. It must never combine a newer source revision with an older text snapshot. This permits zero blob reads during queries; otherwise the authority must explicitly budget candidate-count multiplied by extraction bytes. `authority.scan(owner,{limit,cursor})` must perform an indexed, owner-scoped, bounded range query before materializing results and return `{items,cursor}`. It must not call `list` over a full inbox then slice. Every change affecting text, tags, type, date, owner, ACL, quarantine or deletion must increase the source revision and publish search work atomically.

## Transactional ingestion

Call `enqueueSearchEvent(tx,{owner,id,source,operation,revision?},{maxQueue?})` inside the same serializable transaction as source mutation. Operations are `upsert` and `tombstone`; no source plaintext appears in events. Omitting revision increments a durable `search-revision` record. Use the returned revision in the source record before committing. Never erase the revision record on source deletion. A queue event for the same item is coalesced to the latest revision. The queue has an independent per-owner bound; creating a new event at capacity throws `search_queue_full`, causing the source transaction to roll back. A runtime must drain/retry work rather than silently drop events.

`MetadataSearchOutbox` uses `metadata.list({prefix,limit})`, and acknowledges inside a transaction only if the stored revision still matches. The metadata adapter must enforce LIMIT in storage, not after full enumeration. Both transaction `get/set/delete` and `get/put/delete` conventions are supported. Processing batches are at most 100 events, normally 50. Failure leaves work pending. No external embedding call occurs inside the source transaction.

`SearchService.ingest(event)` loads authoritative source state. It copies only the exact event revision; delayed work does not copy old event payloads or overwrite newer documents. Missing/unauthorized/quarantined sources produce tombstones. Lexical/vector indexes retain latest revision markers and reject old or repeated mutations. Embedding receives only explicitly approved content (`embeddingAllowed:true`), and source revision/authorization is rechecked after embedding completes before its output is retained. Runtime owner gates must serialize index mutations.

`rebuild(owner,{limit,cursor})` backfills pages exclusively from authority. With lexical indexes, `{reset:true}` starts a fresh rebuild by deleting that owner's derived rows and revision markers; use only on the first page under the runtime owner gate. It then uses normal bounded authority pages. Concurrent authoritative mutations still publish outbox events, and all returned data still passes live checks. During rebuild search may be incomplete. Rebuilding a remote vector index needs a new index/namespace generation and reconciliation; reset is explicitly rejected when a vector port is attached. No global item enumeration is required. Source revision/tombstone records remain authoritative even when an index is discarded.

The local AccountController separates legacy inbox catalog migration from search backfill, with independent persistent cursors and at most 200 source metadata rows per pass. The catalog can finish even when extraction work is at capacity. Backfill uses the active plan's queue budget and reserves the smaller of 100 items or 10% of capacity for new source mutations; exhausted capacity records a blocked state and schedules retry without failing owner reads. Alarms advance migration/backfill and drain at most 10 extraction jobs per pass; touch's longer lifecycle alarm must not overwrite pending migration work. Search reports backfill incomplete until its source cursor finishes, and pending/limited state until indexing work is handled. Cold shelving is deferred during pending work. `tests/search-runtime.test.mjs` checks native SQLite warm/restart deletion safety, shelved queues, legacy indexing, queue exhaustion, 451-message progress driven by alarms without owner retries, and the reserved capacity bound.

## Query and pagination

`search(owner,{text,tags:{all,any},types,from,to,mode,limit,cursor})` supports exact case-sensitive tags. Every `all` tag is required; at least one `any` tag is required when present. Tag groups combine with AND. Types are ORed; all other groups combine with AND. Dates are inclusive epoch milliseconds. Lexical text is normalized into Unicode word tokens with AND matching. It accepts no FTS query language, SQL, regular expression or wildcard expression from the client. Query text is limited to 500 characters, each tag/type group to 32 entries, and pages to 100 items (default 20).

Modes are lexical, semantic and hybrid (default). Semantic requires a supplied embedding and vector provider; without them it throws `semantic_search_unavailable`. Hybrid falls back to lexical and reports `semanticAvailable:false`. No hash or string similarity is presented as an embedding. Hybrid ranks bounded lexical and vector lists using reciprocal rank fusion, `sum(1/(60+rank+1))`, with stable ID tie breaks. Each list and final candidate set is bounded (default 200). There is no full-inbox result sort. Search deliberately does not report a total count or claim exhaustive retrieval. A stale candidate can be removed by authority checks, so a page may be short.

Cursors are random opaque tokens stored in bounded service sessions (default 1,000 sessions, five-minute lifetime). They bind exact owner, normalized query, filters, mode and page size. Sessions hold candidate identifiers/revisions/scores, never document content. Every page repeats live authorization and source-revision checks. Cursors are consumed after use, expire, and become invalid after service restart. Cursor pages expose at most the bounded initial candidate set; they are not full-archive pagination. Storage browse/export needs its own indexed cursor contract.

## Adapters and deployment status

`MemorySearchIndex` is a bounded reference/test adapter; it evaluates the capped owner partition and retains only bounded best candidates. `MemoryVectorIndex` performs exact cosine scoring over a capped owner partition using caller-supplied vectors. These are not scale benchmarks or substitutes for durable production indexes.

`SqliteSearchIndex({sql,transaction,budgets})` uses FTS5, indexed owner/date/type columns, exact indexed tag rows and maintained per-owner token/document usage. Its SQL applies owner and filters before ORDER/LIMIT. Lexical parameters are bound and token expressions are quoted. Mutating source-revision markers, document rows, tags, usage and FTS rows is one synchronous atomic transaction. Each runtime account should have its own database to isolate FTS corpus statistics as well as data. `sql.exec(statement,...bindings)` returns a synchronous row iterable. `nodeSqlPort(db)` adapts an injected Node DatabaseSync with rollback-capable savepoints; it imports no Node modules. DO integration supplies its `storage.sql` and `fn => storage.transactionSync(fn)`.

[SQLite-backed Durable Object storage officially supports FTS5](https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/). [Node's SQLite API supplies DatabaseSync and prepared statements](https://nodejs.org/api/sqlite.html). This is a verified API capability; runtime wiring and local tests are separate from production verification.

`WorkersAIEmbeddingAdapter` requires an actual binding, a configured model and exact dimensions; it validates returned embedding shape. `VectorizeSearchIndex` requires a real binding, durable revision metadata, a single-owner mutation gate and explicit dimensions/budget. Owner-derived namespaces restrict ANN candidate generation before topK. Remote records contain opaque vector IDs and owner/item/revision metadata, no plaintext or snippets. Immutable revision-specific vector IDs prevent late work overwriting a newer revision. Tombstones mark local state before remote deletion; remote mutation completion is eventual, so authority checks remain mandatory. Provider deletion failure, orphan-vector reconciliation, model/version migration and durable usage reconciliation remain deployment gates; adapters alone do not prove removal from a provider or production readiness.

Runtime ingestion currently covers messages and email metadata with supported message bodies. The unified schema and authority/index ports support additional item kinds, but storage-provider object bodies, general documents and RFC822/MIME email parsing are not wired or claimed searchable by this runtime. A new source must provide authoritative live ACL/revision checks and transactional ingestion events before appearing in search.

[Vectorize namespace/metadata filtering precedes candidate topK](https://developers.cloudflare.com/vectorize/reference/metadata-filtering/). [Vectorize limits full-metadata queries to 50 results](https://developers.cloudflare.com/vectorize/reference/client-api/), so this adapter caps topK to 50 and live filters may further reduce recall. Exact tag/date/type filters are reapplied by the service; remote prefilter optimization needs configured metadata indexes and is not claimed here. [Workers AI embedding integration](https://developers.cloudflare.com/vectorize/get-started/embeddings/) documents actual model output. The hosted configuration now binds Workers AI to the fixed BGE-small English model and four account-routed Vectorize indexes. Local defaults leave remote semantic work off. These bindings are provisioned; deployment, throughput and provider billing verification remain separate gates.

## Content and independent resource limits

`extractSearchText` accepts at most 1 MiB input by default and emits at most 32 KiB UTF-8 text. Plain text and HTML are supported. HTML is treated as inert text; tags are discarded, script/style/template/noscript blocks are omitted, and no links or media are fetched. A validated web archive uses only its text entrypoint. It does not recurse into archives, execute scripts, extract files, OCR images, transcribe audio/video or parse office/PDF documents. Unsupported media returns empty content with `supported:false`. Metadata remains searchable. The runtime can impose tighter input limits or defer larger media to separately bounded, approved extraction jobs. Extracted HTML snippets are text and must be displayed with textContent, never innerHTML.

Credentials and secret-classified documents are refused, including provider credential item/source kinds. Storage-provider configuration must never enter the authoritative item adapter. No extraction of private provider credentials and no embedding of credential material is authorized. Human private content may be searched only under its live owner ACL; external embedding additionally requires explicit item authorization and deployment privacy approval. `messageSearchDocument` defaults to owner-only readers and `embeddingAllowed:false`.

Blob storage quota and search budgets are separate resources. The product's free blob quota is 500 MB; this module neither charges nor increases blob bytes. Provisional free search defaults: 10,000 active indexed documents, 2,000,000 lexical tokens, 10,000 vectors at configured dimension (default 384), 1,000 queued items, 200 candidates and 32 KiB text per item. Provisional paid defaults: 100,000 documents, 20,000,000 tokens, 100,000 vectors and 5,000 queued items. `planBudget()` reports these configuration placeholders and remaining independent resources; it is not a paid entitlement check, provider pricing or a production capacity claim. Actual model dimension must match the vector index configuration. Vector storage, embedding tokens/cost, extraction CPU, outbox retry/dead-letter policy and per-account DB size need explicit operational budgets before deployment.

Validation: `tests/search.test.mjs` covers lexical/tag/date/type filters, live source revisions, warm/cold ACL/deletion/quarantine protection, owner filtering before lookup, cursor binding/expiry/page revocation, transactional queue overflow/coalescing/acknowledgement, replay/tombstones, supplied semantic embeddings, actual local Node FTS5 queries/rollback, credential refusal, inert bounded extraction and injected Cloudflare adapter contracts. It does not perform network embeddings, live Vectorize mutations, performance benchmarks, human privacy approval or production conformance testing.

## 2026-10-12 hosted semantic rollout

Hosted configuration selects `@cf/baai/bge-small-en-v1.5`, mean pooling, 384 dimensions and cosine indexes. Four fixed shards route by owner SHA-256; search queries one shard. An account persists layout version/shard and rejects silent routing changes. The compact English model receives only the first 480 UTF-8 bytes of supported message text, including title; this is an intentionally incomplete excerpt index, not multilingual or full-document coverage. Enabling remote processing requires owner opt-in in the localized portal. Disabling stops use immediately and schedules bounded remote cleanup; API mutation acceptance alone is not deletion-completion proof.

Each owner has 1,000 document embedding attempts/month, 500 semantic queries/month, 768,000 conservative input-token units/month, 576,000 vector-operation dimensions/month and 25,000 micro-USD estimated operation allowance/month. The initial 1,000-vector reservation ceiling is **lifetime upsert attempts**, including replacements and ambiguous failures; it does not automatically recycle on deletion. This conservative prototype ceiling can pause semantic enrichment permanently until verified reclamation is implemented. Text indexing continues when semantic work is paused. Query modes are lexical, semantic and hybrid; hybrid can fall back to text with an explicit notice.

Service admission is split across sixteen fixed budget authorities whose limits sum to $5/month and $0.50/day of estimated AI/vector-operation allowance, 1,000 lifetime owner admissions and two million lifetime upsert reservations. Per-account durable leases reserve 128 micro-USD or 16 upsert slots independently, amortizing shared RPCs. Daily expiry, restart or ambiguous outcomes never refund credits. Fixed partitions may exhaust unevenly; migration must preserve all spent and outstanding obligations. The estimate excludes recurring vector storage, base-corpus queried dimensions, R2, SQL, Workers ingress and other services, and **is not a Cloudflare invoice cap**. See [economic model](../architecture/unit-economics.md) and [independent audit](../release/semantic-spend-audit.md).