Operational Concerns

Running a vector database in production requires managing index builds, live updates, compaction, backups, and quality monitoring.

---

1. Index Build Time

Index	Build time (10M vectors, 768d)	Incremental update?
Flat	Minutes	✅ (just append)
IVF	10-30 minutes	⚠️ (need to retrain centroids periodically)
HNSW	30-60 minutes	✅ (insert into graph)
DiskANN	1-2 hours	⚠️ (rebuild for best quality)

HNSW is preferred for production — supports incremental inserts without full rebuild.

---

2. Live Updates

New document → embed → insert into vector index
Delete document → remove from index (mark as deleted)
Update document → delete old + insert new

Compaction: Deleted vectors leave “holes” in the index. Periodic compaction reclaims space:

# Milvus compaction
collection.compact()  # merge segments, reclaim deleted space

---

3. Monitoring

Metric	Why	Alert threshold
Query latency P99	User experience	> 2× baseline
Recall (sampled)	Search quality	< target recall
Index size	Capacity planning	> 80% node memory
Insert latency	Ingestion pipeline health	> 100ms
Segment count	Compaction needed	> 2× expected

---

4. Backup and Restore

# Milvus backup
./milvus-backup create --name daily_backup

# Restore to different cluster
./milvus-backup restore --name daily_backup --target new_cluster

Strategy: Daily incremental backups + weekly full backups. Test restore regularly.

---

Self-Check

• Estimate index build time for a given vector count and index type
• Explain compaction and when it’s needed
• Design a monitoring dashboard for a production vector database
• Set up backup and restore for a vector database

---

References

• Milvus Operations Guide