Self-hosted AI: the data layer you run next to a private LLM
A private LLM needs more than a model runtime. This guide lists the data services around it, which open source engines fill each job, and which of them OSStend runs, including Langfuse, Qdrant, ClickHouse and Temporal.
- ✓Data layer only
- ✓Checked 17 Nov 2026
Self-hosted AI needs a model runtime and a data layer: a vector store for retrieval, a cache, memory for agents, an event stream that keeps embeddings current, and a store for traces. OSStend runs the data layer: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, and Langfuse with the ClickHouse it needs. It does not host models or GPUs.
What does a self-hosted AI stack need besides the model?
Six jobs sit around the model, and five of them are data services. The model runtime serves tokens; vector retrieval, a cache, agent memory, an event stream and a trace store keep the answers grounded, fast and inspectable.
| Layer | Job | Open source options | OSStend runs it |
|---|---|---|---|
| Model runtime | Serves the model: generates text and embeddings | Ollama is one that runs open models locally | No. OSStend does not host models or GPUs |
| Vector retrieval | Finds the passages nearest to a question | pgvector on PostgreSQL, OpenSearch, or a dedicated engine such as Qdrant or Milvus | PostgreSQL with pgvector, OpenSearch and a dedicated Qdrant: yes. Milvus and other dedicated engines: no |
| Cache | Reuses answers to repeated or similar questions, and holds sessions | Valkey or Redis | Yes: Redis and Valkey |
| Agent memory | Keeps what an agent learned between sessions | PostgreSQL, with Valkey for short-term state | Yes: PostgreSQL and Valkey |
| Event stream | Carries source changes so embeddings are re-computed when documents change | Apache Kafka with change data capture | Yes: Kafka |
| Trace store | Records prompts, responses and evaluations for debugging | Langfuse, whose self-hosted stack uses PostgreSQL, Redis or Valkey, ClickHouse and object storage | Yes: managed Langfuse with its PostgreSQL, ClickHouse and Valkey |
Sources: Langfuse self-hosting docs, Ollama repository, pgvector README, checked 17 Nov 2026. The right column is what OSStend runs, from its engine list.
Which vector store should a private LLM use?
Use the one that sits where your data is. If the source data is in PostgreSQL, pgvector keeps vectors beside it; if you need keyword and vector search together, OpenSearch does both; a dedicated engine such as Qdrant suits vector-only workloads at large scale.
Embedding models usually return vectors of a few hundred to about a thousand dimensions: Ollama's docs say the length depends on the model, typically 384 to 1,024 (Ollama embeddings docs, checked 17 Nov 2026). pgvector indexes vectors up to 2,000 dimensions (README, checked 17 Nov 2026), so a typical private-LLM embedding model fits an index. The six options are compared in best open source vector databases, the retrieval pattern is in vector database for RAG, and the head-to-head with a dedicated engine is pgvector vs Qdrant.
From our on-call engineers: what breaks first in a self-hosted AI data layer
In a self-hosted AI stack the data layer breaks at the seams: embeddings that no longer match the model, filters that hide results, and a trace store with an unsafe queue. These are the checks we run on the data services, not the model.
- ✓A pgvector column has a fixed dimension, so changing the embedding model means re-embedding every row into a new column or table, and we store the model name with each vector.
- ✓Filtered vector search can return fewer rows than requested on approximate indexes, so we test the real filter with the real data, and use pgvector 0.8.0 iterative scans where they apply.
- ✓Langfuse needs a web service and worker, Postgres, ClickHouse, Redis or Valkey and S3-compatible storage, so a trace store is five services, not one.
- ✓Langfuse persists incoming events to blob storage before processing them, and requires Redis maxmemory-policy set to noeviction so queue jobs are never evicted.
- ✓Semantic caches and agent memory belong in Redis with explicit TTLs and maxmemory, because unbounded caches fill memory and then evict or fail.
How do you keep a private LLM's knowledge current?
Stream changes from the source database into Kafka and re-embed only what changed. Without it, retrieval answers from the old version of a document until someone re-runs a full load.
The pattern is documented on keeping embeddings fresh with Kafka and CDC. The streaming side is managed Kafka; the store that receives the vectors is PostgreSQL or OpenSearch.
Where do an agent's memory and a semantic cache live?
Long-term memory fits PostgreSQL, because it needs durability, queries and access control. Short-term state and cached answers fit Valkey or Redis, because they are small, fast and expire.
See agent memory and semantic cache for the patterns, context engineering for how the stores combine, and managed Redis and Valkey.
What does the trace and evaluation store need?
Langfuse is one open source example, and its own docs list what it needs: a web service and a worker, PostgreSQL for transactional data, Redis or Valkey for cache and queue, ClickHouse for observability data, and S3-compatible or blob storage for events and attachments.
| Langfuse component | What it holds | Who runs it in a private stack |
|---|---|---|
| Langfuse Web and Worker | The UI and API, and asynchronous event processing | Your team, or OSStend |
| PostgreSQL | Transactional data | Your team, or managed PostgreSQL from OSStend |
| Redis or Valkey | Cache and job queue | Your team, or managed Redis or Valkey |
| ClickHouse | Observability data | Your team, another provider, or a dedicated ClickHouse from OSStend |
| S3 or blob storage | Events and attachments | Your team or your cloud provider |
Source: Langfuse self-hosting overview, checked 17 Nov 2026. The same page says Docker Compose is for local use and testing on a single VM without high availability, scaling or backups, and lists Kubernetes and the major clouds for production. OSStend runs Langfuse and the stores in this table; see managed Langfuse.
What does OSStend run for a self-hosted AI stack, and what does it not?
OSStend runs the data engines: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, ClickHouse and Langfuse, designed per workload, in your environment. It also runs Temporal and n8n for workflows and automation. It does not host models, GPUs, agent frameworks or gateways.
Contact us if
- ✓The model runs inside your network and the retrieval data must too.
- ✓You want the vector, cache, memory and event stores run 24/7 by named engineers.
- ✓You want one provider for PostgreSQL, Valkey, OpenSearch, Kafka, Qdrant and Langfuse.
Not a fit if
- ✕You want someone to host the model or provide GPUs: OSStend does not.
- ✕You want a managed agent framework or a model gateway.
- ✕You want Milvus or Weaviate run for you: OSStend runs pgvector and Qdrant, not those two.
OSStend is operated by AceMQ. The wider picture is on the AI data layer page; start from managed PostgreSQL if retrieval and memory come first.
FAQ
What is a self-hosted LLM?
A model you run on your own infrastructure rather than calling a hosted API. Ollama, for example, is an MIT-licensed tool that runs open models locally (repository, checked 17 Nov 2026). The model still needs a data layer for retrieval, caching and memory.
Do I need a vector database for a private LLM?
If the model must answer from your documents, yes: retrieval needs somewhere to store embeddings. PostgreSQL with pgvector is often enough when the data already lives there. See best open source vector databases.
Can OSStend host my model or provide GPUs?
No. OSStend runs the data layer only: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, ClickHouse and Langfuse.
What databases does a self-hosted AI stack need?
A vector store, a cache, a durable store for agent memory, an event stream to keep embeddings fresh, and a trace store. The table above names an open source option for each.
Does OSStend run Langfuse?
Yes. Langfuse's own docs list the databases it needs, and OSStend runs Langfuse with its PostgreSQL, ClickHouse and Valkey. See managed Langfuse.
Related pages
- LangfuseManaged Langfuse hosting on your infrastructureLangfuse pricing on Langfuse Cloud, what self-hosted Langfuse needs, and managed Langfuse run with PostgreSQL, ClickHouse and Valkey in your environment.
- EnginesManaged database service for enterprise workloadsA managed database service for enterprise workloads: PostgreSQL, MySQL, Redis or Valkey, OpenSearch and Kafka, designed, run and tuned by engineers.
- AlternativeQdrant Cloud pricing and alternatives (2026)Qdrant Cloud pricing tiers and SLAs from its published page, and dedicated Qdrant run by OSStend engineers in your own environment, next to PostgreSQL.
- GuideBest open source vector databases (2026)The best open source vector databases in 2026 compared: pgvector, Qdrant, Milvus, Weaviate and more, with the method we used and where each one fits.
- AI data · RetrievalVector database for RAG on your own infrastructureA vector database for RAG on your own infrastructure: when PostgreSQL with pgvector is enough, when OpenSearch hybrid search fits, and how to size retrieval.
- Comparisonpgvector vs Qdrant for RAG on PostgreSQLpgvector vs Qdrant for RAG: when vectors should stay in PostgreSQL, when a dedicated vector engine wins, and what each costs to run. Sourced and dated.
Design the data layer with an engineer
45 minutes: the stores your private LLM needs are reviewed, and an engineer says what to run where.
- ✓A design sketch for your workload, not a plan menu
- ✓A zero-downtime cutover path for what you run today
- ✓A price for that design, after the review