Building Modern Data Repositories: Architecture for Search, Semantic Retrieval, and RAG

Every organization piles up more information than any one person can keep in their head. Policies, contracts, product docs, support records, operational knowledge, regulated filings, meeting notes, API schemas. The list keeps growing. The hard part is rarely having the data. It is finding the right fragment at the right moment, with enough context to act on it.

This article walks through the architecture choices behind modern data repositories: durable storage, metadata, access control, retrieval, and how AI systems sit on top of that foundation.

What a data repository does

A modern data repository is the system of record for discoverable organizational knowledge: content you can ingest, store, describe, index, search, and retrieve under clear access controls.

That is broader than “search”:

• Data ingestion: connectors from PDFs, databases, APIs, email archives, scanners; normalize formats; detect duplicates.
• Durable storage: immutable originals in object storage, plus structured records you can rebuild from.
• Metadata management: owner team or business unit, effective date, classification, language, ACL, lineage.
• Indexing: keyword and vector structures that keep lookup sub-linear as the corpus grows.
• Search and retrieval: exact, filtered, faceted, semantic, and hybrid paths that return provenance people and models can trust.

In production we split these into separate components, not one giant process:

At a glance:

Keyword search is precise when the vocabulary lines up. Vector search is forgiving when the meaning lines up. We usually run both, and fuse the rankings in the app before anything hits an LLM.

Start with what works without the model

Most “AI knowledge” projects start at the wrong end. A team wires a chat UI to a pile of PDFs, embeds everything, and hopes the model will figure it out. That demo holds up until someone asks for yesterday’s policy update, a team-scoped answer, or a citeable clause number.

We start from a different question: what must still be true if the model is offline?

• Users can still find documents by identifier and title.
• Access control still holds.
• Originals remain recoverable.
• Publish events still refresh indexes within a known lag.

If those hold, chat and assistants are add-ons. If they do not, every model upgrade means digging through broken foundations.

Principles we design around

When we design repositories for enterprise and public-sector systems, a few decisions show up again and again. These are not academic preferences. They are how the system stays operable after demo week.

1. Keep the durable store as the source of truth

Object storage holds the PDF. Postgres holds ACL, lineage, and workflow state. Search and vector indexes are rebuildable projections. If a search node dies, you replay from durable state. The hot index is never the source of truth.

type IndexDocument = {
  id: string
  title: string
  body: string
  department: string
  document_type: string
  language: string
  effective_date: number // unix seconds; sortable
}

type DurableRecord = {
  id: string
  objectKey: string // s3://…/policy-1042.pdf
  searchable: IndexDocument // fields projected into search
  acl: { departments: string[] }
  contentHash: string // change detection for reindex
}

async function publishPolicy(record: DurableRecord) {
  await saveMetadata(record) // Postgres commit first
  await upsertSearch(record.searchable) // keyword projection
  await enqueueEmbed(record.id) // vectors can lag if SLO is monitored
}

Ordering matters. The metadata commit is the durability checkpoint. Keyword upsert should follow closely. Embedding jobs can run asynchronously if you measure and alert on publish-to-searchable lag.

2. Store the metadata early

Without owner team, effective date, document type, and language, you cannot constrain hybrid retrieval. Facets, filters, and RAG safety all depend on metadata quality at ingest. We budget for extraction in the pipeline, not as a cleanup ticket later.

3. Enforce access control in the app

Search engines rank well. They are a poor place to invent your security model. The app API applies session-scoped filters (team, tenancy, classification) before results leave the trust boundary. Clients never invent their own filter payload.

4. Keep service contracts simple

Ingest workers speak JSON and job queues. Search speaks HTTP. The app orchestrates. That split lets you swap embedding models or search engines without rewriting the portal.

5. Make human search work first

If a user cannot find Section 4(2)(b), or the equivalent clause in a contract or product policy, without an LLM, the repository is incomplete. AI should speed up a path that already works.

6. Measure retrieval quality

We keep a golden query set next to the service: real user and agent queries, not marketing prompts. CI catches catastrophic regressions. A monthly review catches slow drift after analyzer or embedding changes. Without that loop, hybrid fusion is guesswork.

How the layers work in practice

Principles only help when they show up in the pipeline. The layers below are where those decisions get concrete.

Ingest from messy sources

Real corpora are messy: scanned policies and circulars, product PDFs, bilingual titles, duplicate uploads, partial OCR, attachments nested in case or ticketing systems. Most repository quality is won or lost here.

We usually stage ingest as an explicit state machine:

type IngestStatus = 'received' | 'extracted' | 'chunked' | 'metadata_ready' | 'indexed' | 'failed'

type IngestJob = {
  sourceId: string
  status: IngestStatus
  attempts: number
  lastError?: string
}

Choices that matter more than which model you pick:

• Idempotent publishes via content hash, so retries do not duplicate vectors.
• Structure-aware chunking (heading / clause / page) instead of only fixed windows.
• OCR as a first-class path, with confidence flags so weak scans do not silently poison semantic search.
• Dead-letter queues with an operator UI, because failed PDFs will happen.

For multi-team or multi-tenant organizations, model tenancy early: source system IDs, owning team or business unit, and classification travel with every durable record. Retrofitting tenancy after vectors exist costs far more than a few extra metadata fields on day one.

Publish from durable storage

Durable records and derived indexes stay separate. The hot path needs fast lookup. Recovery and large payloads need disk.

Put keyword and vector indexes on a dedicated tier because lookup latency dominates search and RAG context assembly, prefix and typo-tolerant queries are access-heavy, and capacity tracks tokens and vectors rather than PDF gigabytes alone.

Keep documents and metadata durable because rebuilds need a source of truth, large bodies do not belong in every index, and audit and retention attach to the durable record, not to a search hit.

Shape the search document

The application owns a stable schema for what enters the keyword index. Keep it boring and filter-friendly. The IndexDocument shape above is enough for most operational corpora: identity, text fields, and the dimensions you filter or sort on.

async function upsertPolicy(doc: IndexDocument) {
  await fetch(`${process.env.SEARCH_URL}/indexes/policies/documents`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${process.env.SEARCH_API_KEY}`,
    },
    body: JSON.stringify(doc),
  })
}

Example payload:

{
  "id": "policy-1042",
  "title": "Automobile insurance coverage",
  "body": "This circular clarifies premium slabs for four-wheeler policies.",
  "department": "transport",
  "document_type": "circular",
  "language": "en",
  "effective_date": 1719792000
}

Filterable fields shrink candidate sets before ranking. Sortable dates keep “latest amendment” workflows sane. Merge authz filters from the session in the app API before calling search.

When the same logical document has multiple languages or versions, prefer explicit version rows (or language + supersedes_id) over overwriting the only record. Users need last year’s policy or circular as often as this year’s amendment.

How search works under the hood

You do not need to reimplement a search engine to design a repository. You do need a clear mental model of what that POST triggers, so each query class lands on the right path.

Use keyword search for exact matches

Classical full-text search builds an inverted index: tokens map to documents, often with positions. Query flow is tokenize → look up postings → intersect candidates → rank.

// Conceptual shape of a keyword index, not a storage engine
type SearchIndex = {
  [token: string]: {
    documentIds: number[]
    positions: number[]
  }
}

Positions unlock phrase matching, proximity ranking, and snippet highlights. Dropping them saves RAM and costs phrase quality. That trade should be deliberate.

Engines keep token dictionaries in memory-efficient trees so prefix search and bounded typo tolerance stay fast. For application design, the implication is simple: exact identifiers, clause numbers, and title lookups belong on the keyword path. Do not outsource those to embeddings.

Tokenization choices (locale, stemming, field weights) define the atomic units of the index. Two teams “indexing the same PDF” with different analyzers are not running the same system.

When keyword search is not enough

Query: How can I protect my vehicle?
Document: Automobile insurance coverage

After stop-word removal there may be zero shared tokens. Stemming and synonym lists help only where you maintain them. Conceptual questions, cross-lingual phrasing, and multi-hop “what changed in 2024?” tasks need more than token overlap.

Combine semantic and keyword search

Semantic search embeds text into vectors so paraphrases land nearby. At ingest: chunk → embed → store. At query time: embed the question, retrieve neighbors, then apply the same metadata filters used for keyword search.

type ChunkRecord = {
  id: string
  documentId: string // always point back to the durable parent
  text: string
  embedding: number[]
  department: string
}

Our default is hybrid:

1. Keyword path for identifiers, titles, and exact phrases.
2. Semantic path for natural-language and paraphrase recall.
3. Fusion in the app API (reciprocal rank fusion or a weighted merge).
4. Optional rerank on a shortlist.
5. Hard ACL filters on every path.

async function searchPolicies(q: string, department: string) {
  const filter = { department }

  const [keywordHits, semanticHits] = await Promise.all([
    keywordSearch(q, filter),
    semanticSearch(q, filter),
  ])

  return reciprocalRankFusion([keywordHits, semanticHits]).slice(0, 20)
}

Keyword keeps precision. Vectors extend recall. The app owns fusion and policy. That split is an architectural decision, not a model fad.

Separate search UI and assistants

Once you have that hybrid retriever, both read paths should reuse it and the same ACL rules.

Write paths get most of the architecture attention. Read paths need the same clarity.

• Search / browse UI: ranked documents with snippets, facets, and links to the durable object. Optimize for skimming and verification.
• Assistant / RAG: a short list of cited chunks, then generation. Optimize for grounded answers and refusal when evidence is thin.

Packaging changes (document cards versus cited context windows). The security model should not. If those diverge, drift is inevitable.

Add RAG on top of the repository

Retrieval-augmented generation should not own storage. It should borrow the repository’s retrieval paths, then ask a model to answer using evidence.

async function answerQuestion(input: {
  question: string
  user: { id: string; departments: string[] }
}) {
  const hits = await searchPoliciesForUser(input.question, input.user)
  const reranked = await rerank(input.question, hits.slice(0, 8))

  return generateAnswer({
    question: input.question,
    context: reranked, // documentId + quote spans required
  })
}

This shape buys four properties:

• Grounding: answers cite repository passages instead of model priors.
• Freshness: update indexes; do not fine-tune the foundation model for every policy or document change.
• Access control: retrieval stays inside the user’s permission boundary.
• Auditability: log which document IDs supported which answer.

Design explicitly for retrieval miss, context stuffing, citation hallucination, prompt injection via malicious documents, and stale indexes after publishes. Mitigations are operational: refusal thresholds, required quote spans, sanitized retrieved text, lag monitors. Prompt wording alone will not save you.

Across enterprise and public-sector UX, show sources by default, distinguish “answer” from “excerpt,” and keep a human escalation path. The assistant is a faster path into the repository.

Put the full architecture together

Defaults we recommend

Run the repository day to day

Architecture without an operating model is incomplete:

• Lag SLOs for publish → searchable (for example, p95 under five minutes for routine document publishes).
• Rebuild playbooks tested quarterly, not only after an outage.
• Index capacity planning tied to token and vector growth, not PDF gigabytes alone.
• Access reviews that sample RAG answers against ACL fixtures.

Those practices sound mundane. They are what separate a repository you can defend to leadership from a chatbot that worked on staging.

Know when you are done

• A user can find a clause by number and by plain-language intent.
• Results respect organizational ACLs under automated tests, not only demos.
• Every AI answer can show a source span back to the durable object.
• Indexes rebuild from durable storage within an agreed RTO.
• A living query set has owners who review failures monthly.
• Upstream publish events reindex within a known lag budget.

Start with the repository, then add assistants. Invert that order and every model upgrade means digging through broken foundations.

Further reading

• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP (RAG)
• Malkov & Yashunin: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW)
• Manning, Raghavan, Schütze: Introduction to Information Retrieval