Vector Database Infrastructure

The vector database is the retrieval backbone of every RAG system. It must handle low-latency nearest-neighbor search at scale while supporting filtered queries, incremental updates, and high availability. The right choice depends on your scale, latency requirements, and operational maturity.

1. What RAG Demands from a Vector DB

RAG workloads have specific requirements that differ from generic vector search:

Requirement	Why it matters	Target
Low query latency	Retrieval is on the critical path (before LLM generation)	<50ms P99
Filtered search	“Search only in user’s documents” or “only recent docs”	Pre-filter + vector search
Incremental updates	Documents change — must update without full reindex	Seconds, not hours
Horizontal scaling	Corpus grows from thousands to millions of vectors	Sharding + replication
High availability	If retrieval fails, RAG falls back to no-context (bad quality)	99.9%+
Hybrid search	Combine vector similarity + keyword matching (BM25)	Built-in or plugin

2. Vector DB Options Compared

Feature	Milvus	Qdrant	Weaviate	Pinecone	pgvector
Architecture	Distributed (etcd + Pulsar)	Single-node or distributed	Distributed	Managed SaaS	Postgres extension
Language	Go + C++	Rust	Go	Proprietary	C
Index types	IVF, HNSW, DiskANN, GPU-IVF	HNSW	HNSW, flat	Proprietary	IVF, HNSW
Max vectors	Billions	100M+ (per node)	100M+	Billions (managed)	10M (practical)
Hybrid search	Built-in (sparse + dense)	Built-in	Built-in (BM25)	Built-in (sparse)	Via `ts_rank` + pgvector
Filtered search	Pre-filter + post-filter	Payload filtering	Where clauses	Metadata filtering	SQL WHERE
GPU acceleration	Yes (GPU-IVF)	No	No	Unknown	No
Self-hosted	Yes (open source)	Yes (open source)	Yes (open source)	No (SaaS only)	Yes (Postgres)
Managed offering	Zilliz Cloud	Qdrant Cloud	Weaviate Cloud	Pinecone (only)	Supabase, Neon
Operational complexity	High (3+ components)	Low (single binary)	Medium	None (managed)	Low (just Postgres)

When to Use Each

Decision tree:
├── Already have Postgres, <5M vectors?  → pgvector
├── Want zero ops, willing to pay?       → Pinecone
├── Need billion-scale, GPU search?      → Milvus
├── Want simple self-hosted?             → Qdrant
└── Need multi-modal + built-in BM25?    → Weaviate

3. Index Selection for RAG

The index type determines the speed-recall tradeoff:

Index	Build time	Query latency	Memory	Recall@10	Best for
Flat (brute force)	None	O(n) — slow at scale	Low	100% (exact)	<100K vectors
IVF	Minutes	1-5ms	Low	95-99%	1M-100M, memory-constrained
HNSW	Hours	0.5-2ms	High (graph in RAM)	97-99.5%	<50M, latency-critical
DiskANN	Hours	2-10ms	Low (SSD-resident)	95-99%	Billion-scale, cost-sensitive

For RAG: HNSW is the default — sub-millisecond latency, high recall, all major DBs support it.

HNSW Tuning for RAG

# Milvus HNSW configuration
index_params = {
    "index_type": "HNSW",
    "metric_type": "COSINE",    # or IP (inner product) for normalized vectors
    "params": {
        "M": 32,                 # connections per node (higher = better recall, more memory)
        "efConstruction": 256,   # build quality (higher = better graph, slower build)
    }
}

# Search parameters
search_params = {
    "ef": 128,                   # search beam width (higher = better recall, slower query)
}

# Tuning guide:
# M=16:  good balance for <10M vectors
# M=32:  better recall for 10M-100M vectors
# M=64:  diminishing returns, high memory
# ef=64:  fast queries, recall ~95%
# ef=128: balanced, recall ~98%
# ef=256: high recall ~99.5%, 2× latency

4. Filtered Search Architecture

RAG almost always needs filtered search: “find similar documents from this user’s workspace.”

Pre-Filtering vs Post-Filtering

Pre-filter:  Filter first → search among filtered subset
  Pros: Exact results, no waste
  Cons: Smaller search space → may degrade recall if filter is very selective

Post-filter: Search all → filter results after
  Pros: Full vector search quality
  Cons: May return fewer than top-K results after filtering

Production approach: Most databases use a hybrid — pre-filter when the filter is selective (removes >90% of vectors), post-filter when it’s broad.

# Qdrant: filtered search
from qdrant_client import QdrantClient, models

client = QdrantClient("localhost", port=6333)

results = client.search(
    collection_name="documents",
    query_vector=query_embedding,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="workspace_id",
                match=models.MatchValue(value="ws_123"),
            ),
            models.FieldCondition(
                key="created_at",
                range=models.Range(gte="2024-01-01"),
            ),
        ]
    ),
    limit=20,
)

Partition by Tenant

For multi-tenant RAG, partition data to avoid cross-tenant leakage and improve filter performance:

# Option 1: Separate collections per tenant (strongest isolation)
# Good for <100 tenants with large datasets each
collection_name = f"docs_{tenant_id}"

# Option 2: Partition key within one collection
# Good for 100-10K tenants, simpler management
client.create_collection(
    collection_name="documents",
    vectors_config=...,
    # Qdrant supports payload-based partitioning for efficient filtered search
)

5. Operational Concerns

Sizing

# Memory estimation for HNSW
def estimate_memory(num_vectors, dimensions, M=32):
    """Estimate HNSW memory requirement."""
    vector_bytes = num_vectors * dimensions * 4  # float32
    graph_bytes = num_vectors * M * 2 * 8         # bidirectional edges, 8 bytes each
    metadata_overhead = num_vectors * 200          # ~200 bytes metadata per vector
    total_bytes = vector_bytes + graph_bytes + metadata_overhead
    return total_bytes / (1024**3)  # GB

# Examples:
# 1M vectors, 1024 dims:  ~4.5 GB
# 10M vectors, 1024 dims: ~45 GB
# 100M vectors, 1024 dims: ~450 GB (need distributed setup or DiskANN)

Replication and Availability

# Milvus: configure replicas for high availability
# 2 replicas = survives 1 node failure
replicas: 2
consistency_level: "bounded"  # eventual, bounded, strong, session
# "bounded" is best for RAG: reads may lag by a few seconds, but queries are fast

Backup and Recovery

# Qdrant: snapshot-based backup
curl -X POST "http://localhost:6333/collections/documents/snapshots"
# → creates point-in-time snapshot, downloadable as tar

# Milvus: backup to S3
# Uses milvus-backup tool for collection-level backup/restore

Compaction and Index Maintenance

After many inserts and deletes, index quality degrades:

# Schedule periodic compaction (e.g., weekly)
# Milvus
client.compact(collection_name="documents")

# Rebuild index if recall drops
# (measure recall by comparing against flat/exact search on a test set)
if measured_recall < 0.95:
    client.drop_index(collection_name="documents")
    client.create_index(collection_name="documents", index_params=optimized_params)

6. Performance Benchmarks

Real-world numbers (single node, 1M vectors, 1024 dims, HNSW):

Database	Query latency P50	Query latency P99	QPS (1 thread)	QPS (32 threads)	Recall@10
Qdrant	0.8ms	2.1ms	1,200	8,500	98.5%
Milvus	1.2ms	3.5ms	900	7,200	98.2%
Weaviate	1.5ms	4.2ms	700	5,800	97.8%
pgvector	3.5ms	12ms	300	2,100	97.0%

These are approximate — performance varies significantly with hardware, tuning, and workload. Always benchmark on your data.

For large-scale vector DB internals, see Index Structures and Performance Tuning.

Self-Check

Choose a vector database given specific scale, latency, and operational constraints
Configure HNSW parameters (M, efConstruction, ef) for a RAG workload
Design a multi-tenant filtered search architecture
Estimate memory requirements for 10M vectors at 1024 dimensions
Explain the pre-filter vs post-filter tradeoff for filtered vector search