Guide

Self-hosted AI: the data layer you run next to a private LLM

A private LLM needs more than a model runtime. This guide lists the data services around it, which open source engines fill each job, and which of them OSStend runs, including Langfuse, Qdrant, ClickHouse and Temporal.

  • Data layer only
  • Checked 17 Nov 2026

Self-hosted AI needs a model runtime and a data layer: a vector store for retrieval, a cache, memory for agents, an event stream that keeps embeddings current, and a store for traces. OSStend runs the data layer: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, and Langfuse with the ClickHouse it needs. It does not host models or GPUs.

What does a self-hosted AI stack need besides the model?

Six jobs sit around the model, and five of them are data services. The model runtime serves tokens; vector retrieval, a cache, agent memory, an event stream and a trace store keep the answers grounded, fast and inspectable.

LayerJobOpen source optionsOSStend runs it
Model runtimeServes the model: generates text and embeddingsOllama is one that runs open models locallyNo. OSStend does not host models or GPUs
Vector retrievalFinds the passages nearest to a questionpgvector on PostgreSQL, OpenSearch, or a dedicated engine such as Qdrant or MilvusPostgreSQL with pgvector, OpenSearch and a dedicated Qdrant: yes. Milvus and other dedicated engines: no
CacheReuses answers to repeated or similar questions, and holds sessions
Agent memoryKeeps what an agent learned between sessionsPostgreSQL, with Valkey for short-term stateYes: PostgreSQL and Valkey
Event streamCarries source changes so embeddings are re-computed when documents changeApache Kafka with change data capture
Trace storeRecords prompts, responses and evaluations for debuggingLangfuse, whose self-hosted stack uses PostgreSQL, Redis or Valkey, ClickHouse and object storageYes: managed Langfuse with its PostgreSQL, ClickHouse and Valkey

Sources: Langfuse self-hosting docs, Ollama repository, pgvector README, checked 17 Nov 2026. The right column is what OSStend runs, from its engine list.

Which vector store should a private LLM use?

Use the one that sits where your data is. If the source data is in PostgreSQL, pgvector keeps vectors beside it; if you need keyword and vector search together, OpenSearch does both; a dedicated engine such as Qdrant suits vector-only workloads at large scale.

Illustrative: similar documents sit close together, so retrieval is a nearest-neighbour search.

Embedding models usually return vectors of a few hundred to about a thousand dimensions: Ollama's docs say the length depends on the model, typically 384 to 1,024 (Ollama embeddings docs, checked 17 Nov 2026). pgvector indexes vectors up to 2,000 dimensions (README, checked 17 Nov 2026), so a typical private-LLM embedding model fits an index. The six options are compared in best open source vector databases, the retrieval pattern is in vector database for RAG, and the head-to-head with a dedicated engine is pgvector vs Qdrant.

From our on-call engineers: what breaks first in a self-hosted AI data layer

In a self-hosted AI stack the data layer breaks at the seams: embeddings that no longer match the model, filters that hide results, and a trace store with an unsafe queue. These are the checks we run on the data services, not the model.

  • A pgvector column has a fixed dimension, so changing the embedding model means re-embedding every row into a new column or table, and we store the model name with each vector.
  • Filtered vector search can return fewer rows than requested on approximate indexes, so we test the real filter with the real data, and use pgvector 0.8.0 iterative scans where they apply.
  • Langfuse needs a web service and worker, Postgres, ClickHouse, Redis or Valkey and S3-compatible storage, so a trace store is five services, not one.
  • Langfuse persists incoming events to blob storage before processing them, and requires Redis maxmemory-policy set to noeviction so queue jobs are never evicted.
  • Semantic caches and agent memory belong in Redis with explicit TTLs and maxmemory, because unbounded caches fill memory and then evict or fail.

How do you keep a private LLM's knowledge current?

Stream changes from the source database into Kafka and re-embed only what changed. Without it, retrieval answers from the old version of a document until someone re-runs a full load.

The pattern is documented on keeping embeddings fresh with Kafka and CDC. The streaming side is managed Kafka; the store that receives the vectors is PostgreSQL or OpenSearch.

Where do an agent's memory and a semantic cache live?

Long-term memory fits PostgreSQL, because it needs durability, queries and access control. Short-term state and cached answers fit Valkey or Redis, because they are small, fast and expire.

See agent memory and semantic cache for the patterns, context engineering for how the stores combine, and managed Redis and Valkey.

What does the trace and evaluation store need?

Langfuse is one open source example, and its own docs list what it needs: a web service and a worker, PostgreSQL for transactional data, Redis or Valkey for cache and queue, ClickHouse for observability data, and S3-compatible or blob storage for events and attachments.

Langfuse componentWhat it holdsWho runs it in a private stack
Langfuse Web and WorkerThe UI and API, and asynchronous event processingYour team, or OSStend
PostgreSQLTransactional dataYour team, or managed PostgreSQL from OSStend
Redis or ValkeyCache and job queueYour team, or managed Redis or Valkey
ClickHouseObservability dataYour team, another provider, or a dedicated ClickHouse from OSStend
S3 or blob storageEvents and attachmentsYour team or your cloud provider

Source: Langfuse self-hosting overview, checked 17 Nov 2026. The same page says Docker Compose is for local use and testing on a single VM without high availability, scaling or backups, and lists Kubernetes and the major clouds for production. OSStend runs Langfuse and the stores in this table; see managed Langfuse.

What does OSStend run for a self-hosted AI stack, and what does it not?

OSStend runs the data engines: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, ClickHouse and Langfuse, designed per workload, in your environment. It also runs Temporal and n8n for workflows and automation. It does not host models, GPUs, agent frameworks or gateways.

Contact us if

  • The model runs inside your network and the retrieval data must too.
  • You want the vector, cache, memory and event stores run 24/7 by named engineers.
  • You want one provider for PostgreSQL, Valkey, OpenSearch, Kafka, Qdrant and Langfuse.
Talk to an engineer

Not a fit if

  • You want someone to host the model or provide GPUs: OSStend does not.
  • You want a managed agent framework or a model gateway.
  • You want Milvus or Weaviate run for you: OSStend runs pgvector and Qdrant, not those two.

OSStend is operated by AceMQ. The wider picture is on the AI data layer page; start from managed PostgreSQL if retrieval and memory come first.

FAQ

What is a self-hosted LLM?

A model you run on your own infrastructure rather than calling a hosted API. Ollama, for example, is an MIT-licensed tool that runs open models locally (repository, checked 17 Nov 2026). The model still needs a data layer for retrieval, caching and memory.

Do I need a vector database for a private LLM?

If the model must answer from your documents, yes: retrieval needs somewhere to store embeddings. PostgreSQL with pgvector is often enough when the data already lives there. See best open source vector databases.

Can OSStend host my model or provide GPUs?

No. OSStend runs the data layer only: PostgreSQL with pgvector, Qdrant, OpenSearch, Redis or Valkey, Kafka, ClickHouse and Langfuse.

What databases does a self-hosted AI stack need?

A vector store, a cache, a durable store for agent memory, an event stream to keep embeddings fresh, and a trace store. The table above names an open source option for each.

Does OSStend run Langfuse?

Yes. Langfuse's own docs list the databases it needs, and OSStend runs Langfuse with its PostgreSQL, ClickHouse and Valkey. See managed Langfuse.

Talk to an engineer

Design the data layer with an engineer

45 minutes: the stores your private LLM needs are reviewed, and an engineer says what to run where.

  • A design sketch for your workload, not a plan menu
  • A zero-downtime cutover path for what you run today
  • A price for that design, after the review
See pricing →

Your request goes to the OSStend team. No newsletter sign-up.

Get a quoteTalk to an engineer
2Engines
3Where should it run?
4What do you need?

Your request goes to the OSStend team. No newsletter sign-up.