Opens in a new tab

Vector Search on Apache Iceberg – Without Moving Your Data

StarTree has extended Apache Pinot’s vector indexing to work with embeddings stored in the data lake. This makes it possible to combine ANN search with SQL analytics.

Written By
Published
Reading Time

Suppose you’re storing hundreds of millions of log records in Apache Iceberg. Each record contains an error message, a timestamp, a service name, a customer ID, and perhaps an embedding that represents the meaning of the message.

Now you want to find incidents that resemble a particular failure, even when the error messages use completely different words. And you want to restrict the results to a particular customer, service, or time window.

Vector similarity search can solve the first part of this problem. But the typical architecture requires additional systems to generate embeddings, load them into a vector database, and then keep that index synchronized with the records in your lakehouse.

That might be a reasonable tradeoff when vector retrieval is your primary workload. But if Iceberg is already your source of truth, it introduces additional infrastructure, data movement, and operational complexity.

What if you could build a vector index over your existing Iceberg data and query it directly, without copying the source dataset into a separate serving database?

That’s what we’ve been working on at StarTree.

By combining Apache Pinot’s vector indexing capabilities with StarTree External Tables, we’ve made it possible to create HNSW indexes over embeddings stored in Parquet files and execute approximate nearest-neighbor (ANN) searches using SQL.

The source records remain in your lakehouse. Embeddings live in a float[] column in your Iceberg/Parquet files, written by your own pipeline. Pinot builds the vector index on that column in place, with no data copy. 

StarTree builds and maintains the search indexes alongside the source records, allowing applications to combine vector similarity with the structured filters and analytical queries that Pinot already supports.

In an initial proof of concept using 100,000 documents and an S3-backed storage tier, we measured approximately 50 ms median latency for warm vector similarity queries, including queries with structured SQL filters.

While that’s not a production-scale performance guarantee, it demonstrates the feasibility of indexed vector search over data stored in object storage.

In this article, we’ll explain how the architecture works, why vector indexes are challenging to serve from object storage, how we adapted Pinot to make it possible, and what we’ve learned from our initial testing.

Why vector search on lakehouse data is challenging

Apache Iceberg has become a useful foundation for storing and managing large analytical datasets. Organizations can retain enormous volumes of structured and unstructured information in open formats while using different engines to process and analyze it.

But not every query pattern is equally efficient over files in object storage.

Consider a vector similarity query. An embedding model converts text, images, or other content into an array of numbers representing characteristics of the original content. Embeddings that are semantically similar tend to occupy nearby positions in a high-dimensional space.

A search for “database ran out of connections” might therefore identify a log message containing:

HikariPool-1 - Connection is not available, request timed out

Even though the two strings share few meaningful tokens, their embeddings may be close together.

If you already have embeddings stored in your lakehouse, it might seem straightforward to calculate the distance between the query vector and every stored embedding.

Unfortunately, that approach becomes expensive as the dataset grows.

A collection of 100 million records, each with a 768-dimensional embedding, contains 76.8 billion individual vector values. Finding the closest records by exhaustively comparing every embedding requires substantial computation and data access for each query.

This is why vector databases and search engines use specialized indexes.

Approximate nearest-neighbor algorithms avoid comparing the query against every vector. Instead, they navigate an index that helps identify likely nearest neighbors, trading some recall for a significant reduction in work.

Apache Pinot already supports this type of search through vector indexes based on Hierarchical Navigable Small World (HNSW).

The challenge was to extend that capability to data managed in the lakehouse.

An index designed around fast, random reads from local storage cannot simply be pointed at Parquet files in S3 and expected to perform well. Object storage has different access patterns and latency characteristics.

We needed to preserve the efficiency of ANN search while allowing the source data and its derived search indexes to live in object storage.

How StarTree runs vector search directly on Iceberg

StarTree External Tables provide a way to query Parquet data in an existing lakehouse without first ingesting the source records into native Pinot segments.

Rather than copying the underlying dataset, StarTree connects to the source table, builds the required index structures, and uses Pinot’s query engine to execute SQL against the source data.

For vector search, the architecture has three main components.

1. The source data remains in Iceberg

Your existing Iceberg table continues to contain the original records.

For example, a product catalog might have a schema containing:

ColumnDescription
doc_idRecord identifier
titleProduct title
categoryProduct category
tenant_idCustomer or tenant identifier
embeddingFloating-point array representing the product description

The embedding column can be produced by an embedding model as part of an upstream data-processing pipeline. Because the embeddings are stored alongside the other columns, the same records can support vector similarity searches and conventional analytical queries.

2. StarTree builds an HNSW index alongside the data

StarTree uses Apache Pinot’s vector indexing capabilities to create an HNSW index over the embedding column.

HNSW organizes vectors into a navigable graph. During a similarity search, the engine traverses this graph to find nearby vectors instead of calculating distances against every record.

The resulting index is stored separately from the source Parquet data in the object-storage environment.

This is an important distinction: the source dataset doesn’t have to be duplicated into a native Pinot table, but vector search still requires an additional index.

That index consumes storage and has to be built and maintained as the source data changes. The vector index is roughly the size of the raw embeddings themselves; the HNSW graph adds only a few percent on top. The index lives on object storage alongside the segment, not on local disk. 

Each segment has its own index, and queries search all segments in parallel across servers. Add servers to scale. Data and indexes stay on object storage, so compute and storage grow independently.

3. Pinot executes the similarity search using SQL

Applications query the External Table through Pinot’s SQL interface.

For example, assuming a configured vector index, a query can identify candidate products with embeddings similar to a supplied query vector:

SELECT doc_id, title,
  cosineDistance(
    embedding,
    ARRAY[0.021, -0.118, ...]
  ) AS dist
FROM catalog
WHERE VECTOR_SIMILARITY(
  embedding,
  ARRAY[0.021, -0.118, ...],
  10
)
ORDER BY dist ASC
LIMIT 10;Code language: SQL (Structured Query Language) (sql)

The vector shown here is abbreviated for readability. An actual query must supply an embedding with the dimensions expected by the index.

VECTOR_SIMILARITY performs the ANN search and identifies candidate records. It acts as a filter predicate rather than returning a similarity score directly.

The cosineDistance function then calculates distances for the returned candidates, allowing them to be ordered by similarity.

Behind the scenes, Pinot distributes the query across servers responsible for different segments and combines the results.

The outcome is that an application can execute vector search without first copying the source records into a separate vector-serving database.

Hybrid queries: vector similarity with SQL

Finding similar embeddings is useful, but most production applications need more than a nearest-neighbor search.

Consider a customer-facing support platform. An engineer wants to find incidents similar to a particular connection failure, but only for one customer and within a recent time window.

Or perhaps a product recommendation system. The application needs products that resemble a customer’s request, but they must also belong to an allowed category, be available for sale, and meet a price constraint.

These are hybrid search workloads: queries that combine semantic similarity with structured predicates.

Because StarTree executes vector search inside Pinot’s SQL engine, applications can express these requirements together.

For example:

SELECT doc_id, title, category
FROM catalog
WHERE tenant_id = 42
  AND category = 'accessories'
  AND VECTOR_SIMILARITY(
    embedding,
    ARRAY[...],
    10
  )
ORDER BY cosineDistance(
  embedding,
  ARRAY[...]
) ASC
LIMIT 10;
Code language: SQL (Structured Query Language) (sql)

This query combines vector similarity with tenant and category restrictions in a single SQL statement.

Why is that better?

When you use a vector database for similarity search and another system for structured data, applications must first retrieve candidate identifiers from the vector store, fetch matching records from the analytical database, apply additional filtering, and then potentially repeat the process if too many candidates are rejected.

Retrieving the nearest ten records and then discarding those that fail a restrictive filter can produce fewer results than expected. Searching only within a filtered subset can require a different execution strategy.

Two naive ways to combine a filter with ANN versus the engine side approach

And while many vector databases support metadata filtering natively, maintaining a separate vector-serving system can still require another index, another data pipeline, and another set of consistency and operational considerations.

But if your source data already lives in Iceberg, StarTree lets you use a distributed analytical query engine to access indexed vectors and associated structured data together.

Pinot chooses from a selection of vector execution strategies based on the available index and the predicates in the query.

How Pinot picks the vector execution mode.

The query engine balances filtering, recall, and execution cost.

How we made HNSW work with object storage

Apache Pinot has supported vector search for native tables for some time.

Under the hood, Pinot builds on Apache Lucene’s HNSW implementation. Lucene provides the graph-based indexing and approximate nearest-neighbor search functionality.

That works well when Pinot’s indexes are stored on local disks.

But bringing those indexes to object storage required us to solve a different problem.

HNSW expects efficient random access

Unlike a conventional column scan, an HNSW query follows connections through a graph.

The search starts at an entry point, explores neighboring vectors, and progressively navigates toward likely nearest neighbors.

That traversal involves numerous small, irregular reads.

When indexes are stored locally, those reads are relatively inexpensive. When index data must be accessed remotely, each additional network interaction can become significant.

Lucene’s vector index also uses a collection of files with local file-access semantics.

In the implementation described here, those include the vector data (.vec), graph connections (.vex), and metadata (.vem).

Simply moving those files to S3 doesn’t provide the same execution environment.

We needed a way to make the index accessible through Pinot’s object-storage-aware segment infrastructure.

Step 1: Consolidate the index files

The first change was to consolidate the Lucene vector-index files into Pinot’s combined segment file, columns.psf.

Instead of expecting a separate directory of Lucene index files, Pinot can represent the vector index as a typed entry within the combined segment structure.

This makes it possible for the index to participate in the same storage and access mechanisms used for tiered Pinot segments.

Step 2: Read the index through a storage-aware buffer

The second change was to adapt the HNSW index reader to use a PinotDataBuffer backed by the consolidated segment file.

The reader can access the vector index through that abstraction rather than depending on a local Lucene directory.

When the required index data isn’t already cached, the storage layer can retrieve the relevant byte ranges from object storage.

The engineering challenge wasn’t inventing a new ANN algorithm. It was adapting an established, efficient indexing implementation to a storage architecture with very different access characteristics.

Step 3: Apply the same mechanism to External Tables

Once vector indexes could be served through Pinot’s tiered-storage infrastructure, we could apply the same mechanism to External Tables.

The difference is significant.

With a native Pinot table, Pinot ingests the records, builds segments, and owns the serving representation of that data.

With an External Table, the records remain in their original Parquet files. Pinot builds the derived index structures needed for efficient queries, while the source column values remain in the lakehouse.

That is how we extended vector search from data ingested into Pinot to data already stored in Iceberg.

Configuring a vector index in StarTree

StarTree’s documented vector index configuration attaches an index to a multi-valued FLOAT column with RAW encoding.

For example, a 768-dimensional embedding column can use the following configuration:

{
  "fieldConfigList": [
    {
      "name": "embedding",
      "encodingType": "RAW",
      "indexes": {
        "vector": {
          "vectorIndexType": "HNSW",
          "vectorDimension": 768,
          "vectorDistanceFunction": "COSINE",
          "version": 1
        }
      }
    }
  ]
}Code language: JSON / JSON with Comments (json)

The key settings are:

  • vectorDimension: The number of dimensions in each embedding. This must match the embedding model’s output.
  • vectorDistanceFunction: The similarity metric, such as cosine distance.
  • vectorIndexType: The indexing algorithm, HNSW in this example.
  • encodingType: RAW encoding, required for the vector column.

For External Tables, the index must also be configured to be stored inside the segment file. This is enabled through the storeInSegmentFile property.

That storage setting is particularly important because it allows the index to be accessed through the same object-storage infrastructure used for External Tables.

For general configuration details, see the StarTree Vector Index documentation.

What happens when your Iceberg data changes?

One of the practical advantages of a lakehouse is that multiple applications and pipelines can continuously add or update data.

But vector search introduces an additional concern: when the source data changes, the search index must remain consistent with it.

StarTree External Tables use a synchronization process that detects changes to the source dataset and builds the index structures used for querying.

For Iceberg tables, this process follows the table’s metadata and snapshots to discover changes to the underlying files. Snapshot onboarding is configurable, so freshness depends on how often you sync.

As new data becomes available and is processed by the synchronization mechanism, the corresponding indexes can be generated or updated.

This means applications don’t need a separate pipeline that copies their source records into a native Pinot dataset solely to make the lakehouse data searchable.

However, it’s important to distinguish this process from Pinot’s native real-time ingestion.

A Kafka-backed real-time Pinot table can build supported vector indexes as records arrive. An External Table has a different lifecycle, governed by the synchronization of its source files and metadata.

How well does vector search from object storage perform?

To test the approach, we built a proof of concept using a product catalog stored as Parquet in S3.

We configured an Iceberg External Table with 768-dimensional embeddings and an HNSW index stored through StarTree’s object-storage tier.

The objective was to verify that Pinot could execute ANN search using the index rather than falling back to an exhaustive scan.

First, verify that Pinot is using the index

The initial test was to inspect the query execution plan.

We used EXPLAIN PLAN to confirm that vector similarity queries selected the HNSW ANN execution path.

The plan reported:

executionMode: ANN_TOP_K
backend: HNSW

This confirmed that Pinot was using the configured vector index rather than silently evaluating the similarity function against every record.

The test ran on a single node with segments on an S3 tier. Read these numbers as a trend, not benchmark throughput.

Warm query latency (100k docs on the S3 tier, p50)

We then measured several query types over a dataset containing 100,000 documents on the S3-backed tier.

The results show that indexed vector search can provide interactive query latency over this dataset, even when the source records and index structures are stored on an object-storage-backed tier.

In this example, adding the structured filter did not measurably increase the median query latency.

The full-scan example was more than twice as slow as the indexed ANN query, even at this relatively small data size.

That illustrates why an index matters. But it doesn’t establish how the performance gap will behave across larger datasets or different query patterns.

Vector search performance depends on several factors, including dataset size, index configuration, cache behavior, graph traversal, filtering selectivity, and the resources available to execute the query.

What about index size?

Vector indexes introduce additional storage overhead.

In our test, a 768-dimensional float32 HNSW index required approximately 148.6 MB for a segment containing 50,000 documents.

Most of that space was used to represent the vector values themselves.

Each 768-dimensional float32 embedding takes 768 × 4 = 3,072 bytes

The HNSW graph adds additional data for its connections and metadata.

This explains why index storage is an important architectural consideration, particularly when scaling to hundreds of millions or billions of embeddings.

The experiment also evaluated more compact IVF-based index formats using quantization, producing significantly smaller index footprints.

However, those storage savings come with tradeoffs that need to be evaluated against recall, query latency, and supported execution modes.

Product quantization brings the vector index down to about the size of a text index

How can you use this?

Vector search over existing lakehouse data opens up several interesting application patterns.

Search historical observability data by meaning

Organizations often retain large volumes of logs, traces, and events in their lakehouse.

Embeddings can make this historical data searchable by semantic similarity, allowing engineers to find incidents with related symptoms even when messages contain different terminology.

Combining similarity with SQL filters makes it possible to scope searches to particular services, environments, customers, or time periods.

Build recommendations from existing analytical data

Product catalogs and customer interactions are commonly stored in analytical tables.

Vector similarity can identify products or content related to an item, query, or user interest. SQL filters can apply additional business constraints such as inventory availability, price, geographic region, or category.

This allows applications to combine semantic relevance with the structured information already maintained in the lakehouse.

Retrieve relevant data for AI applications

Retrieval-augmented generation (RAG) systems often need to find relevant documents or records before passing them to a language model.

When those records are already stored in Iceberg, a vector index over the lakehouse offers an alternative to maintaining a separate full copy in a vector-serving system.

Structured filters can further restrict retrieval to the appropriate dataset or tenant, alongside whatever access-control mechanisms the application requires.

From vector similarity to semantic search in plain SQL

There’s still one awkward part of the SQL examples above: the application has to supply an embedding vector.

That means the developer needs to know which embedding model was used, generate a compatible embedding for the query, and pass the resulting array into VECTOR_SIMILARITY.

For many application developers, that’s an implementation detail they’d rather not manage.

StarTree’s semantic search capabilities take this a step further.

Using SEMANTIC_MATCH, a developer can express a semantic search using natural-language text:

SELECT doc_id, title
FROM docs
WHERE SEMANTIC_MATCH(
  body,
  'wireless keyboard for iPad',
  5
)
LIMIT 5;
Code language: JavaScript (javascript)

Learn more about semantic search in the follow-on blog post. 

Try vector search on your own lakehouse

Your lakehouse doesn’t have to be a passive repository. With Pinot’s vector indexing and StarTree External Tables, you can build a search index alongside your source data and execute similarity queries using a distributed analytical query engine.

If you don’t already have a StarTree account, Request a Trial. We’ll set you up to test these features on your own data and workloads.

Contents
Share
Have Questions?

Talk to a Pinot Expert

We're here to provide advice and guidance on how best to leverage Apache Pinot for a wide range of use-cases. Get in touch and we'll help you get the ball rolling.
Subscribe to get notifications of the latest news, events, and releases at StarTree