Context sources
Documents
Wikis · PDFs · object storage
Operational facts
MySQL · PostgreSQL via CDC
Event streams
Kafka · APIs · webhooks
All use cases
Structured facts, text and vectors in one retrieval path
Engineer reliable context for AI agents
Combine documents, operational facts and live events with metadata filters, full-text relevance and vector retrieval before serving bounded context to AI applications.
All sources / last 1 hour
Unified event stream
ALL SOURCESAPPLICATIONKUBERNETESSECURITY
TIMESOURCELEVELMESSAGE
10:42:18.904checkout-apiERRORpayment upstream timeout · trace=8fa2
10:42:18.887k8s / podWARNreadiness probe failed · checkout-7c4
10:42:17.512doris-feINFOquery finished · latency=184ms
Error spike began 3 min after the checkout deploymentInvestigate →
The agent can retrieve semantically similar passages, but it also needs current business facts, access boundaries and exact identifiers that a vector-only store does not understand.
Good context is not a prompt trick; it is a data quality, retrieval and governance system.
The pain
Relevant is not the same as correct, current or allowed
01
Documents and operational facts drift in separate stores.
02
Vector retrieval ignores tenant, time and permission boundaries unless they are modeled deliberately.
03
Keyword, semantic and structured ranking are combined in application glue.
04
Agent queries are hard to inspect after an answer is produced.
REFERENCE PATH / AI CONTEXT ENGINEERING
Build context as a governed data pipeline.
Keep documents, operational facts and live events together; combine structured filters, full-text relevance and vector similarity before serving bounded context to agents.
Ingest & model
Embedding pipeline
Chunk and embed externally
Streaming loads
Continuous facts and events
Metadata model
Tenant · time · access labels
Store & index
HNSW / IVF
Approximate vector retrieval
Inverted index
Full-text and BM25 relevance
VARIANT
Flexible source metadata
Retrieve & rank
Pre-filter
Apply SQL predicates first
Hybrid retrieval
Text and vector signals
Single query plan
Facts, labels and relevance
Serve to AI
MCP Server
Bounded tools for agents
REST or SQL
Application retrieval paths
Audit trail
Inspect queries and evidence
Embeddings are produced by the pipeline you choose; Doris stores and retrieves vectors alongside structured and text context.
BUILD PLAN / CONTEXT ENGINEERING
Treat retrieval quality as a governed data contract
Start with one agent question and a small, permissioned corpus. Define identity, freshness and access filters before tuning semantic recall or adding another retrieval signal.
01Documents · facts · events
02Chunk · embed · stream
03Text · vector · metadata
04SQL · REST · MCP
- Store fixed-dimension embeddings with the source row and stable identity.
- Use inverted indexes and BM25 for lexical relevance; ANN indexes for vector candidates.
- Pre-filter tenant, policy, time and status fields before approximate ranking.
- Keep source metadata flexible in VARIANT while promoting policy fields to typed columns.
- Onboard the event and metadata sources with reviewable table and collector definitions.
- Use Data Catalog and SQL Studio to inspect schema, filters and retrieval results.
- Expose read-bounded access through the integration surface chosen for the agent.
- Track retrieval queries and evidence so a produced answer can be investigated.
BEFORE CUTOVER
Acceptance gates
- 1Access filters cannot be bypassed
- 2Freshness policy is explicit
- 3Retrieval set is inspectable
- 4Offline relevance test is recorded
What changes
The operating pattern changes—not only the tool.
- Fresh operational facts beside document context
- Policy-aware filtering before semantic ranking
- Fewer synchronization points between analytical and vector stores
- Inspectable retrieval queries for agent workflows
CONTACT / HUBSPOT
Bring one workload. Leave with a measured decision.
We’ll map the current cost and investigation path, then define success criteria before you replace anything.