The Vector Database Gap: A Production Blueprint for Embedding-Based Enterprise Search

Abstract

Transformer-based embedding models have solved the representation problem in enterprise search, converting documents, queries, and catalog items into dense vectors of high semantic fidelity. However, serving these vectors at production scale—on corpora of tens of millions of documents—exposes a systems gap that algorithm libraries and general-purpose search engines do not adequately fill. In this paper, we benchmark three vector computation libraries (FAISS, Annoy, and ScaNN) and Elasticsearch 7.14 against six operational requirements critical to production deployments: incremental index mutation, low-latency search under concurrent load, metadata-filtered vector search, embedding-model version migration, cross-region consistency, and recall observability. Experiments were conducted on a production-mirror index of 85 million documents using a 768-dimensional transformer model. We find that no existing system scores above two out of three on more than two requirements, and that recall observability is entirely absent across all evaluated platforms. We further note that Apache Solr, as of August 2021, does not yet expose dense vector field types, and that Elasticsearch 7.x relies on brute-force script_score evaluation, making approximate nearest-neighbor search unavailable in the dominant open-source search-engine stack. To close this gap, we propose a three-layer reference architecture that decouples mutable storage, incremental indexing, and filter-aware query planning, and we contextualize this design against the emerging generation of dedicated vector databases.