Full-Text Search on Iceberg: One Engine for Analytics and Search

Apache Iceberg has become the default table format for the lakehouse. Analytics, reporting, ML feature pipelines, streaming, and batch all read and write the same Iceberg tables — one copy of the data, in open Parquet, in your bucket, queried by many engines.

Then someone asks to search.

“Find every session where error_message contains the phrase connection reset by peer, in the last 90 days, grouped by region.”

The clean picture cracks. Iceberg has no inverted index. Parquet’s built-in statistics are excellent for WHERE price > 100 and useless for WHERE error_message LIKE '%reset by peer%'. So you stand up a second system — Elasticsearch, OpenSearch, Solr — and build a pipeline to copy data out of the lakehouse and into it.

You now pay twice: twice the storage, twice the ingestion, twice the operational surface, and a reverse-ETL pipeline whose only job is to keep two copies of the truth from drifting apart.

This post is about deleting the second system. We’ll walk through how StarTree runs Lucene-grade full-text search directly on Iceberg Parquet — no data movement, no second cluster — and, in detail, the I/O paths it takes to make that fast when the index lives on S3 instead of a local NVMe drive. And because the same engine also ingests streaming data, fresh rows are searchable within seconds of arriving: Iceberg holds the history, Pinot’s real-time ingestion serves the newest rows, and one TEXT_MATCH query spans both.

The whole lakehouse reads one copy — except search, which has historically needed its own.

What if one system did both?

The thesis is easy to state and hard to build:

Point the OLAP engine at the Parquet files you already have, build a real inverted index next to them, and serve both analytical aggregations and text predicates from a single query.

That means a query like this runs end to end with no second system:

SELECT region, COUNT(*)
FROM sessions
WHERE TEXT_MATCH(error_message, '"connection reset" AND peer')
  AND event_time > now() - 90 * 86400000
GROUP BY regionCode language: SQL (Structured Query Language) (sql)

TEXT_MATCH resolves through a Lucene inverted index. The event_time range and the GROUP BY run through Apache Pinot’s columnar engine. Same table, same scan, same answer — sub-second.

To explain how, we build from the bottom: what Lucene actually is, how Pinot already embeds it, why moving that index to object storage breaks the assumptions Lucene was built on, and the techniques that fix it.

Lucene primer

You almost certainly already run Lucene — it is the indexing engine inside Elasticsearch, OpenSearch, and Solr. Three ideas carry the whole design.

1. The inverted index. Instead of mapping documents to the words they contain, Lucene maps each term to the list of document IDs that contain it (the postings list). Searching becomes a dictionary lookup plus a list read — not a scan.

2. The FST term dictionary. Lucene stores the set of all terms as a Finite State Transducer — a compact, in-memory automaton mapping a term to the byte offset of its postings. The FST is what makes prefix (conn*), range, and regex queries cheap: you walk the automaton instead of scanning a flat term list.

3. Immutable segments. Lucene never updates a file in place. Indexing produces write-once segments; new data means new segments, and a background merge compacts them. Immutability is exactly what lets the same bytes be cached aggressively and shared across readers — a property that becomes decisive once those bytes live on S3.

A term lookup is: walk the FST → get an offset → read one postings range. On local NVMe this happens in microseconds. Hold that number — it is the one that breaks on object storage.

Lucene inside Pinot

Apache Pinot is a real-time OLAP database, and it already embeds Lucene (currently Lucene 9.x) as one of several per-column index types. Understanding how the index is built and queried locally is the foundation for everything that follows.

How a text index is built

During segment build, Pinot picks up every column marked for text indexing and feeds its values to Lucene, which produces the inverted index — term dictionary, postings, FST, and positions. Two details matter later:

• Every row becomes a Lucene document holding the analyzed text (indexed, not stored) plus a small stored field carrying the Pinot row ID. That stored field is the bridge back to Pinot’s row space.
• The default analyzer is case-aware: a standard tokenizer, optional lowercasing, and a curated English stop-word list. Text is parsed with Lucene’s classic boolean query syntax, so phrase, wildcard, fuzzy, and AND / OR / NOT queries all work out of the box.

When the segment is sealed, Pinot force-merges Lucene’s internal segments down to one — minimizing open file descriptors and small-file fan-out (the RAM buffer used during the build defaults to 500 MB). The index is written as a per-column Lucene directory inside the Pinot segment, alongside two small artifacts that come up again later: a docID-mapping file (Lucene’s internal row positions → Pinot row IDs, memory-mapped at query time) and a properties file recording the analyzer and parser settings.

The result is one self-contained Pinot segment holding all index types side by side — data and indexes in the same immutable unit:

There is also a mode that packs the whole per-column Lucene directory into a single combined file. Remember that shape — collapsing many tiny Lucene files into one byte-addressable blob is precisely what you need to serve an index efficiently from object storage.

How a text query executes

This flow is the heart of why search and analytics compose cheaply in one engine. Follow a TEXT_MATCH from the broker down:

A few things make this fast, and they’re worth naming precisely:

• Pinot runs the Lucene query with a custom collector that simply gathers matching doc IDs. Because Pinot doesn’t rank results, it tells Lucene to skip scoring entirely — no relevance computation, no top-K priority queue, and none of the term-frequency or norm reads that scoring would otherwise require.
• Each matching Lucene doc ID is translated to a Pinot row ID and added to a Roaring bitmap. When the build-time order already lines up, Pinot detects the identity mapping and skips the translation step (and the mapping file) altogether.

Then comes the part that makes mixing search with analytics nearly free: late binding. The text bitmap is just one leaf in the filter tree. All index-produced bitmaps are sorted by cardinality, smallest first, and AND-merged with native Roaring bitmap intersections before any column is touched. The forward index is read only for the rows that survive every predicate.

So a selective TEXT_MATCH over a wide table reads a handful of rows, not the whole segment — and adding SUM(impressions) costs almost nothing, because the aggregation runs over that same surviving row set.

Why Lucene on object storage is hard

Everything above assumes the index files sit on a local disk — ideally memory-mapped. Lucene was designed for exactly that. Move the files to S3 or GCS and every assumption inverts:

Lucene assumes…	On object storage you get…
Memory-mapped files (mmap) — the FST walk is a pointer chase	No mmap. Every access is an HTTP GET.
Cheap random reads	Time to first byte (TTFB) dominates — tens to hundreds of ms per request, independent of size
The OS page cache absorbs repeat reads	No page cache. The same byte costs the same every time.
Many tiny reads are free	Each GET has fixed cost and quota; high request rates get throttled
Files stay open and warm	Cold opens re-fetch the FST and headers from scratch
Many small files are fine	Per-object overhead and LIST costs punish small-file fan-out

The headline: Lucene runs a query path in microseconds on NVMe; the same path against raw S3 runs in seconds. The algorithm didn’t change — the storage went from nanosecond-latency mapped memory to a high-latency, per-request-billed, cache-less object store. Put bluntly: on object storage, latency — not bandwidth — is the bottleneck, and a term lookup is a latency-bound workload by construction.

The engineering problem, then, is not “can Lucene read from S3” (it can, slowly). It is: how do you preserve microsecond-class term lookups when the bytes live behind 100 ms of network latency? The rest of this post is the answer.

The durable principle behind that answer: keep the object store as the source of truth, and put a local hot tier of RAM + NVMe in front of it to absorb the random reads. Everything that follows is the engineering of that hot tier.

Enabling search on Iceberg

StarTree already serves high-QPS, low-latency analytical queries directly on Iceberg — no ingestion, no materialization (500+ QPS on a 1 TB Iceberg table, benchmarked against Trino and ClickHouse). Full-text search extends that same direct-on-Iceberg model to text predicates.

First the data model: how does a Pinot table come to point at Iceberg Parquet without copying it?

Pseudo-segments: metadata, not data

Two components do the integration, and neither moves your data:

1. A table creator introspects the Iceberg catalog for the schema and creates a matching Pinot table.
2. A watcher — a controller periodic task (default period 5m, controller.periodic.task.externalTableWatcher.frequencyPeriod) — discovers new Parquet files and, for each, has the controller register a metadata-only “pseudo-segment.”

Critically, the watcher does not scan a directory. It asks the Iceberg REST catalog for the current snapshot ID, walks the Iceberg manifests, and diffs against a checkpoint (the last processed snapshot ID, persisted in ZooKeeper). It enqueues only the data files added since that snapshot — true snapshot-diff discovery that respects Iceberg’s metadata layer.

A pseudo-segment is a real Pinot v3 segment that contains only metadata — populated from the Parquet footer (row count, per-column min/max, time range), with zero-length entries everywhere the column data and indexes would normally live. Building it opens the Parquet file exactly once, to read the footer; no row data is ever downloaded. The segment is registered by metadata push, so servers receive only the ZooKeeper metadata — including the pointer to the remote file:

segment.tier            = myS3Tier
segment.index.version   = v3
segment.total.docs      = 2000
segment.size.in.bytes   = 2202          # metadata footprint — NOT the data
custom.parquet_uri      = s3a://bucket/demo/parquet/synthetic_0_0d555657.parquetCode language: PHP (php)

A 2,000-row segment that weighs 2,202 bytes. The data stays in Parquet at custom.parquet_uri. Zero ETL, zero data movement — Pinot queries the remote Parquet directly.

Parquet’s indexes vs. a Lucene text index

Why add Lucene at all? Because Parquet’s built-in structures and a Lucene inverted index solve different problems — they are complementary, not redundant.

	Parquet index	Lucene text index
Structures:	Row-group/page min-max, dictionary encoding, optional bloom filters	Inverted index (term → postings) + FST term dictionary
Granularity:	File → row group → page (coarse)	Doc-level (fine)
Good for:	Range, equality, IN	Phrase, fuzzy, wildcard, boolean text
Storage:	Built into the Parquet format — free	Sidecar artifact, served through the Index Buffer
Built by:	The writer	StarTree — one per Parquet file

Parquet can tell you “this row group might contain error_code = 500.” Only an inverted index answers “which exact rows contain the phrase connection reset by peer.” So StarTree builds one Lucene index per Parquet file and keeps it as a sidecar.

Building the index, cloud-native

The index build is itself a snapshot-aware pipeline, and — this is the contract that matters to data owners — it never writes to your bucket:

1. The watcher diffs the Iceberg catalog between snapshots and enqueues newly added Parquet files.
2. An async indexer reads each Parquet file, runs Lucene to build the index, and consolidates the output into one merged artifact — small-file fan-out is poison on object storage, so a single byte-addressable blob per file is the goal.
3. Your Iceberg bucket stays read-only. Your data, your bucket, untouched.
4. The merged artifact lands in a separate, StarTree-managed index store.
5. Pinot publishes the pseudo-segment via content-addressable storage (CAS): a pair of pointers, one to the Parquet in Iceberg (parquet_uri) and one to the index artifact (index_uri).

As Iceberg advances snapshots, the watcher keeps the index following along automatically.

Making it fast

Back to the hard problem: the index lives on S3, and naïve Lucene-on-S3 is seconds per query. The fix is not one trick but a stack of them, and they all serve five principles that fall straight out of how object storage behaves:

1. Pin the hot, small bits local — the FST, the term dictionary, hot postings, the Parquet footer.
2. Merge tiny files into few big objects — one index artifact per file, so reads are byte ranges into one object, not a fan-out of GETs and LISTs.
3. Hide the ~100 ms S3 latency with asynchronous, parallel reads and concurrent decode.
4. Tier the storage — hot data on local RAM + SSD, cold bulk on S3.
5. Match the access pattern — random, surgical reads on the hot index; bulk reads on the cold data.

Here is how those principles turn into machinery.

1. A remote I/O engine, not a filesystem shim

The foundation is a reader purpose-built for object stores — not a shim that makes S3 look like a local file. The rest of Pinot reads through its ordinary data-buffer abstraction; underneath, the remote reader decides where each byte actually comes from, and adds three things a plain filesystem never needed:

• A hierarchical cache. Every byte range can live in one of several tiers — an in-memory tier, a memory-mapped on-disk (SSD) tier, and the cold object store behind them. A read is served from the closest tier that already has the bytes.
• Smart, asynchronous prefetching. When a segment loads or a query plans, the reader fetches the byte ranges it knows it will need next — in the background, before they’re asked for — so the network round-trip overlaps with useful work instead of stalling the query thread.
• Concurrent decompression and deserialization. Parquet pages and index blocks arrive compressed; the reader decodes them in parallel across threads, so CPU work on already-fetched bytes overlaps with network waits on the next ones. On a 100 ms-latency store, this overlap is the difference between adding up your latencies and hiding them behind each other.

Three caches sit on top of this engine, each holding a different kind of byte:

Cache	What it holds	Tiers
Index cache	Byte ranges of the segment’s index files — Lucene text, inverted, range, JSON	in-memory → mmap on-disk
Parquet data cache	Compressed Parquet column and dictionary pages from the remote files	in-memory → mmap on-disk, plus an in-JVM buffer of the hottest decoded pages
Footer cache	Each Parquet file’s footer (row-group metadata + schema)	local disk; survives restarts

Each cache fills two ways: eagerly at segment load — preload.enable pulls index files into the on-disk tier, and pinot.parquet.prewarm.enabled fetches each column’s first page in the background — or lazily on a miss, where the needed byte range is fetched from S3 once and kept for next time. On-disk tiers evict by LRU on a background sweep.

2. The Index Buffer — random-access reads on remote Lucene

This is the linchpin. The Lucene artifact for each pseudo-segment is exposed through a buffer whose entire contract is a single operation:

read(offset, length) → bytes

No file API, no LIST, no full-file reads. The buffer resolves each read(offset, length) from the closest tier that has the bytes — heap or local SSD if hot, S3 if cold. That one primitive is what lets Lucene’s “walk the FST → read one postings range” pattern survive on object storage, and it is why consolidating the index into a single blob (step 2 of the build) matters: one object, many precise byte-range reads.

The payoff: single-digit-millisecond latency at thousands of queries per second per node for random-access reads on remote Lucene data — orders of magnitude better than a GET per access.

3. Precise reads on Lucene — fetch only what the query needs

Byte-range reads are only as good as the query path using them surgically. For a text predicate:

• The FST resolves each query term to a byte offset in the index — a walk over an in-memory automaton, not a scan.
• We range-fetch only the postings for the matched terms — never the full posting list for the column.
• Skip lists short-circuit boolean operators (AND / OR / NOT), so a postings scan stops as soon as the outcome is decided.
• The index header is cached locally, so there is no per-query metadata round-trip to S3.
• Because scoring is off (recall the no-scoring collector), Lucene never reads the term frequencies or norms it would need to rank — fewer bytes pulled off S3.

A multi-term boolean query thus becomes a handful of small, precise byte-range reads, most of them served from the hot tier, instead of dragging the index across the network.

4. Precise fetching on Parquet — the same discipline, extended to open formats

The same discipline extends from the index to the data. The remote reader parses each Parquet footer once, caches it on local disk, and uses it as a map — per-column min/max plus the byte offset of every page — to skip whole row groups and seek straight to the pages a query actually touches:

A Parquet file is a tree of row groups → column chunks → pages:

PARQUET FILE
├── Row Group 1
│   └── col "impressions" → [ page0 | page1 | page2 ]   ← fetch page1 only
├── Row Group 2   ✗ pruned by footer min/max
└── Footer  →  min/max stats · page offsets · row counts   (cached locally)Code language: JavaScript (javascript)

Two complementary signals decide which pages to prefetch:

• The sparse index (footer min/max). For a filter like WHERE price > 100, the footer’s per-row-group and per-page min/max prune everything that can’t match before a single data byte is fetched.
• The matching-docs bitmap. For WHERE TEXT_MATCH(…), the Lucene index runs first and yields the bitmap of surviving rows; that bitmap then selects exactly which pages of the projected columns are worth fetching.

The two compose with late binding. Consider:

SELECT SUM(impressions)
FROM ads
WHERE TEXT_MATCH(browser, 'Chrome')Code language: SQL (Structured Query Language) (sql)

The text predicate runs first against the Lucene index (precise byte-range reads) and produces the Roaring bitmap of matching rows. That bitmap then drives which impressions pages are fetched — the filter narrows the read, and the projection pulls only the pages covering surviving rows. (Page-level predicate skipping via Parquet’s column-index min/max is an active area of work, not a shipped claim; today’s win is row-group pruning plus bitmap-driven page seeking.)

5. Keep hot bits local — pin + cache + prefetch

Tie it together and the same hot tier serves both halves of every query:

• Lucene hot tier. The FST is pinned in memory after first touch, so term lookups run in microseconds; the index header and hot postings live on local SSD, served by the Index Buffer; and because the index is one artifact per pseudo-segment, there is no small-file fan-out and no LIST.
• Parquet hot tier. The footer and page index are cached locally, so pruning runs without touching S3; hot column pages sit on SSD via the hierarchical cache; cold pages stay on S3 and are fetched as byte ranges on demand.

The principle across both: hot small bits stay local, cold bulk stays on S3, and one piece of I/O machinery serves both the index and the data.

The behavior is configurable per tier. The keys that matter most in practice:

Config key	Purpose
preload.enable	download index files into the mmap tier at segment load
preload.enable.index.consolidation	consolidate index files into one artifact (default true for external tables)
enable.prefetch.page.cache	use the index page cache for the query
pinot.parquet.prewarm.enabled	async pre-warm of each column’s first Parquet data page on load
pinot.parquet.page.cache.disk.snapshot.enabled	resume on-disk caches after a restart (otherwise cleared)
skip.remote.table.cache	bypass the Parquet data cache for a query (read straight from remote)

Benchmarks

Two comparisons tell the story. Charts below are illustrative of the shape, not measured results — numbers depend on cluster, dataset, and selectivity, and a full methodology writeup will follow.

Local vs. remote

Moving the index to S3 costs a one-time cold fetch per pseudo-segment; once the FST and header are pinned, a warm remote query lands in the same single-digit-millisecond band as local.

Pruning vs. no pruning

A selective TEXT_MATCH should read a small fraction of a full scan: row-group pruning drops most of the file, then the matching-docs bitmap selects only the pages worth fetching.

What’s next

Search on Iceberg is the foundation, not the destination. The same Index Buffer and remote I/O machinery generalize to the retrieval workloads that matter for AI:

• Vector search on Iceberg. The same Index Buffer, a new sidecar shape for the vector index, served by the same byte-range read path from the hot tier.
• Hybrid retrieval. Text and vector predicates in one query, intersected exactly the way TEXT_MATCH already intersects with range and equality filters today.
• Pinot as the lakehouse retrieval engine for RAG. Sub-second retrieval over years of data sitting in open Parquet, with no separate vector database to operate.
• Agentic AI storage on Iceberg. A durable memory and retrieval substrate for agents, served from the same lakehouse the rest of the business already runs on.

The through-line is the one we started with: one copy of the data, in open formats, in your bucket — and one engine that serves analytics, search, and retrieval over it. No second system, no reverse ETL, no drift.

Learn more

StarTree + Iceberg — StarTree turns Iceberg into a real-time serving layer with sub-second queries, seconds-level freshness, and predictable high-concurrency performance, without duplicate data, reverse ETL, or extra systems to operate. See the resource hub at startree.ai/resources (StarTree’s docs, webinars, and benchmarks for Pinot + Iceberg).

Apache Pinot text search — the OSS TEXT_MATCH and Lucene text-index documentation at docs.pinot.apache.org (official Pinot docs: text-index configuration and query syntax).

Apache Iceberg — the open table-format specification at iceberg.apache.org (snapshots, REST catalog, manifests, and the Parquet data layout this work builds on).