Log analytics has become one of the most expensive and operationally painful parts of the modern data stack.
The ELK stack and Splunk were built from a search-first design center. That design made sense for the original problem: index machine data aggressively so teams could search it quickly. But log analytics evolved beyond search. Teams now need to retain more data, run aggregations, analyze trends, investigate incidents, support observability workflows, and query semi-structured events at massive scale.
Those requirements arrived after the core architecture was already established.
As log volumes grew from gigabytes to terabytes per day, the search-first model started to show its limits. To keep queries fast, teams pay to index large amounts of data up front, store it on expensive hot infrastructure, and continuously tune clusters as volumes and query patterns change. Most of that data is rarely searched, but it is still treated like high-frequency search data.
The result is the pattern many platform and infrastructure teams recognize today:
- Log retention requirements keep increasing
- Hot storage and indexing costs keep growing
- Large clusters require constant tuning and rebalancing
- SREs spend time operating log infrastructure instead of improving reliability
- Security, observability, product, and support teams still expect interactive query performance
- Data teams end up maintaining separate systems for retention, search, and analytics
This is the core tension: logs are too valuable to discard, but too expensive to keep forever.
Apache Iceberg changes the storage economics. StarTree changes the query economics.
Together, Iceberg and StarTree create a new architecture for log analytics: keep high-volume log data in open, low-cost object storage while still delivering the low-latency, high-concurrency query performance required for production investigations, observability workflows, and operational analytics.
The Shift: Logs Are Moving to the Lakehouse
Many organizations are moving logs into Apache Iceberg because it gives them an open, durable, and primarily cost-efficient foundation for long-term retention.
Iceberg is attractive because it provides:
- Open table format
- Object-storage economics
- Schema evolution
- Partition evolution
- Compatibility with existing lakehouse tools
- Separation of storage and compute
- Reduced dependency on proprietary log storage systems
For platform leaders, this is appealing because it creates a path away from escalating Elasticsearch, Splunk, or proprietary observability storage costs.
But Iceberg alone does not solve the query-performance problem.
A log table in object storage may be inexpensive to retain, but difficult to query interactively. When an engineer is investigating an incident, waiting minutes for a forensic query is not acceptable. When a customer-facing support workflow depends on log exploration, slow queries create operational drag. When security teams are searching for suspicious activity, broad scans across object storage can be too slow and too expensive.
The lakehouse solves the storage problem.
The remaining question is: how do you make it fast enough for production log analytics?
The Core Problem: Most Query Engines Fetch Too Much Data
Log analytics is often a “needle in a haystack” workload.
An engineer may search for:
- A specific
trace_id - A failing
service_name - A customer ID inside nested JSON
- An error pattern over the last 15 minutes
- A spike in failed requests by region, endpoint, or deployment version
- A small set of events across billions of rows
The challenge is not simply running SQL over Iceberg. Many engines can do that.
The challenge is avoiding unnecessary data access.
Traditional lakehouse query engines often rely on file-level pruning, partition pruning, and column statistics. These techniques help, but they are not enough for high-volume log analytics. Even after pruning by time, service, or partition, the engine may still need to scan large Parquet files or column chunks to find a small number of relevant records.
That creates the operational problem platform teams already know:
- Latency increases
- Object storage reads increase
- Compute cost increases
- Query concurrency becomes harder to support
- Engineers lose the interactive investigation loop
For log analytics, performance depends on precision. The winning architecture is not just “query Iceberg.” It is “query Iceberg while fetching dramatically less irrelevant data.”
Why StarTree for Log Analytics on Iceberg
StarTree, powered by Apache Pinot, acts as a high-performance analytical execution layer for logs in Iceberg and other open table formats like Delta Lake.
The design center is different from general-purpose query engines. StarTree is built for SLA-driven analytics: workloads where query performance has to remain fast and predictable under real production pressure.
That matters for log analytics because the workload is:
- High volume
- High cardinality
- Semi-structured
- Time-sensitive
- Concurrency-sensitive
- Operationally critical
StarTree improves log analytics on Iceberg by reducing how much data each query needs to touch. Instead of relying only on broad file-level pruning, StarTree uses indexing, pruning, routing, and precision fetching techniques to narrow the query path before expensive data access happens.
Reference Architecture: Log Lakehouse with StarTree and Iceberg
A modern log lakehouse with StarTree typically follows this pattern:
- Logs are ingested from application, infrastructure, and event streams
- Kafka
- Kinesis
- OpenTelemetry pipelines
- Custom event streams
- Batch or streaming log collectors
- Logs land in Apache Iceberg on object storage
- S3
- GCS
- ADLS
- Private object storage where required
- Iceberg (or Delta Lake) provides the open storage layer
- Long-term retention
- Schema evolution
- Partition evolution
- Interoperability with other lakehouse engines
- StarTree provides the low-latency query layer
- Sub-second analytical queries
- JSON and Lucene indexing
- Precision fetching from object storage
- Integration with operational dashboards and workflows
- Teams consume the data through existing interfaces
- SQL
- Grafana
- APIs
- Internal tools
- Observability workflows
- AI or agentic interfaces
This architecture allows teams to avoid building two separate worlds: one expensive hot log-search system for recent data, and another low-cost lake for everything else.
Instead, Iceberg becomes the system of record, and StarTree becomes the performance layer for the queries that need to be fast.
The Anatomy of an Iceberg Log Table
A performant log lakehouse starts with the table design.
Logs are usually high-volume, semi-structured, and unevenly queried. Most queries include time filters, service filters, severity filters, trace IDs, customer IDs, endpoint names, or nested attributes.
A simplified Iceberg log table might include:
CREATE TABLE logs_db.application_logs (
event_time TIMESTAMP,
service_name VARCHAR,
log_level VARCHAR,
message VARCHAR,
attributes VARIANT,
trace_id VARCHAR
)
USING iceberg
PARTITIONED BY (days(event_time), service_name);Code language: CSS (css)Partitioning by time and service helps engines skip irrelevant data. But this only solves part of the problem.
For example, if a team is investigating one trace_id, one customer ID, or one nested JSON attribute, partition pruning may still leave too much data to scan. A Parquet file can still contain many services, users, sessions, traces, or error types inside the selected partition.
That is where StarTree adds value.
StarTree helps narrow the query path below the partition and file level, reducing unnecessary reads from object storage and making interactive log analytics practical at scale.
Query Engine Tradeoffs for Log Analytics on Iceberg
Several modern query engines can read Apache Iceberg tables. The important distinction is not whether an engine can participate in an Iceberg architecture. The important distinction is whether it is well suited for SLA-driven log analytics: high-volume, high-cardinality, semi-structured investigations where engineers need interactive response times over data stored in object storage.
For this workload, the evaluation question becomes:
How much irrelevant data does the engine need to touch before it can answer the query?
That matters because log analytics is often a “needle in a haystack” workload. Teams are not only running broad aggregations. They are searching for specific trace IDs, customer IDs, error codes, service names, endpoints, IP addresses, deployment versions, and nested JSON attributes across billions or trillions of events.
Trino: Strong for Broad Lakehouse SQL and Federation
Trino is a strong general-purpose SQL engine for lakehouse analytics. It works well when teams need broad access to Iceberg data, federation across systems, and ad hoc analytical exploration.
For log analytics, the limitation is workload fit. Trino can use partition pruning, file metadata, and column statistics to reduce scan volume, but investigative log queries often require a narrower access pattern than file-level pruning can provide. After filtering by time or service, the engine may still need to scan large amounts of Parquet data to find a small number of relevant events.
Trino remains a valuable part of the lakehouse stack. It is often the right tool for broad exploration, transformation, and cross-source analysis. But for production log analytics where engineers expect interactive drilldowns during incidents, it is not usually the best serving layer.
StarRocks: Strong Lakehouse Analytics, But Log Search Has Different Pressure Points
StarRocks has real Apache Iceberg support. It can create external Iceberg catalogs and query Iceberg data directly without manually creating tables or loading the data into StarRocks first. StarRocks also supports using materialized views and other mechanisms to process Iceberg data into StarRocks-managed structures when teams want to accelerate repeated workloads.
That makes StarRocks a capable lakehouse analytics engine, especially for high-performance SQL, dashboards, and aggregation-heavy workloads.
The distinction is that log analytics stresses a different set of requirements. Incident investigations are often dominated by high-cardinality filters, semi-structured attributes, and selective lookups over very large datasets. StarRocks has invested in inverted indexes, including full-text inverted indexes, but those capabilities are primarily positioned around StarRocks-managed data structures and shared-data clusters, not as a universal answer for every external Iceberg log-search pattern.
So the argument is not that StarRocks cannot work with Iceberg. It can.
The argument is that SLA-driven log analytics on Iceberg requires more than general-purpose lakehouse acceleration. Teams need to evaluate whether the engine can preserve interactive performance for high-cardinality, semi-structured investigations without forcing another operationally complex acceleration layer.
ClickHouse: Excellent OLAP Engine with New Iceberg Support, But Performance Often Comes From Movement out of Iceberg
ClickHouse also supports Apache Iceberg. ClickHouse provides an Iceberg table engine and Iceberg table function that allow users to query existing Iceberg tables in object storage and other supported storage systems. Its documentation describes this as a read-only integration for existing Iceberg tables.
ClickHouse is a very strong OLAP engine, particularly when data is stored in ClickHouse-native formats and layouts. That is where ClickHouse typically delivers its best low-latency performance characteristics.
For Iceberg-first log analytics, the architectural tradeoff is important. Querying Iceberg directly gives teams open-table access, but it may not provide the same performance profile as loading, organizing, indexing, or materializing data into ClickHouse-managed storage. ClickHouse itself has noted that querying data in open table formats does not leverage ClickHouse’s optimized internal format, which can result in slower performance than querying data stored natively in ClickHouse.
That does not make ClickHouse a poor choice. It means teams need to be clear about the architecture they want.
If the goal is maximum performance from ClickHouse, teams may choose to maintain another optimized copy or layout. If the goal is to keep Iceberg as the primary operational log store while avoiding duplicated serving infrastructure, that tradeoff needs to be evaluated carefully.
StarTree: Precision Querying for SLA-Driven Log Analytics on Iceberg
StarTree is designed for the part of the log analytics workload where latency, concurrency, and cost per query matter most.
The core advantage is precision. StarTree is built to reduce the amount of data fetched and processed for each query, using indexing, pruning, routing, and precision-fetch techniques to avoid broad scans wherever possible. In the log analytics PDF, this is framed as the difference between file-level access and more precise page-level fetching, especially for high-volume Iceberg-backed log data.
This matters because object storage economics only solve half the problem. Iceberg can make log retention dramatically more cost-effective, but teams still need fast access when logs are part of an incident workflow, support workflow, security investigation, or customer-facing operational experience.
StarTree is strongest when teams need to query Iceberg-backed logs with:
- High-cardinality filters such as trace_id, customer ID, endpoint, region, or service name
- Semi-structured attributes stored in JSON
- Interactive drilldowns during live investigations
- High concurrency across engineering, SRE, support, or customer-facing users
- Sub-second or near-sub-second analytical response times
- Lower object-storage read amplification
- Fewer duplicated serving layers
The practical positioning is simple:
Trino, StarRocks, and ClickHouse can all participate in an Iceberg architecture. StarTree is purpose-built for the subset of Iceberg workloads where log analytics needs to behave like an interactive operational system, not a best-effort lakehouse query.
For platform leaders, that distinction matters. The goal is not just to store logs cheaply in Iceberg. The goal is to make Iceberg usable for production log analytics without recreating the cost, complexity, and duplication of the systems teams were trying to move away from.
Deep Dive: Semi-Structured Logs and JSON Indexing
The hardest part of log analytics is often not the timestamp, service name, or log level.
It is the attributes field.
Modern services emit rich semi-structured context:
- User IDs
- Session IDs
- Request IDs
- Deployment versions
- Region
- Device type
- API endpoint
- Tenant ID
- Error codes
- Feature flags
- Business-specific metadata
This data is essential during investigation, but it can be difficult to query efficiently.
In traditional search systems, teams often run into mapping complexity, schema drift, and index bloat. In lakehouse query engines, teams may be able to store nested attributes, but not always query them with the interactivity required during live investigations.
StarTree supports native JSON indexing so teams can query nested log attributes without turning every investigation into a broad scan.
Example:
SELECT count(*)
FROM application_logs
WHERE attributes.user.id = 'user_99'
AND event_time > now() - interval '1 hour';Code language: JavaScript (javascript)For the senior data engineer or architect, the important point is not just that the query works.
The important point is that the platform provides a practical way to make these queries fast and debuggable without requiring a fragile collection of pre-aggregations, hand-built indexes, or separate serving databases.
PromQL and Grafana: Meeting SREs Where They Already Work
Log analytics does not live in isolation.
SREs and engineering teams already work in Grafana, Prometheus, and other observability workflows. They do not want to pivot between separate tools every time they move from metrics to logs to operational context.
StarTree’s PromQL and Grafana support is important because it allows teams to bring log analytics closer to existing observability workflows.
For example, an engineer could investigate error rates with a PromQL-style query in Grafana:
rate(application_logs{service_name="auth-svc", log_level="ERROR"}[5m])Code language: JavaScript (javascript)The value is not just UI convenience.
The value is reducing workflow friction during incidents. If a service is failing, engineers need to move from signal to detail quickly. The more tools, copies, and query paths involved, the slower the investigation becomes.
With StarTree on Iceberg, teams can use familiar operational interfaces while querying raw or detailed log data stored in the lakehouse.
That matters for platform leaders because it increases adoption without forcing every team to learn a new operational workflow.
What This Means for Data Platform Leaders
For the data platform engineering leader, the value of StarTree on Iceberg is about reducing cost and complexity without compromising production SLAs.
The architecture supports several strategic goals:
- Move high-volume logs to lower-cost object storage
- Avoid runaway SSD-based log retention costs
- Reduce duplicated serving infrastructure
- Support sub-second interactive analytics
- Serve multiple teams from a common Iceberg foundation
- Reduce operational burden on SRE and platform teams
- Keep data in open formats
- Integrate with Kafka, S3/GCS, Iceberg, Grafana, and existing cloud infrastructure
StarTree lets you standardize log storage on Iceberg without accepting slow lakehouse query performance.
That is the difference between Iceberg as a cold archive and Iceberg as the foundation for operational analytics.
What This Means for Senior Data Engineers and Architects
For the senior engineer or architect, the question is more technical:
Can this architecture actually meet production query requirements without becoming another fragile system to operate?
StarTree addresses the issues engineers typically care about:
- Query performance predictability
- Indexing strategy
- Nested JSON access
- Object storage efficiency
- Integration with Iceberg
- Compatibility with existing observability workflows
- Reduced need for custom serving layers
- Lower operational tuning burden
- Clear separation between storage and performance layer
StarTree gives you a purpose-built execution layer for the log queries that general-purpose engines struggle to serve interactively.
It does not require abandoning Iceberg. It makes Iceberg more useful for operational workloads.
What This Means for Engineering Executives
For the VP of Engineering, CTO, or infrastructure executive, the business case is straightforward.
Log volumes are growing. Retention expectations are growing. Infrastructure budgets are under pressure. Engineering teams are spending too much time operating and tuning systems that do not directly differentiate the business.
The risk is not just cost.
The risk is that slow or fragmented log analytics affects:
- Incident response
- Customer support
- Security investigations
- Developer productivity
- Platform reliability
- Product experience
- AI and automation initiatives that depend on fresh operational context
StarTree helps de-risk log analytics at scale by combining open storage economics with production-grade query performance.
This gives engineering leaders a path to control cost growth while improving the speed and reliability of operational analytics.
Where StarTree Fits Best
StarTree is especially strong when log analytics has real production requirements, such as:
High-Volume Application Logs
Teams generating large volumes of application logs need long retention without paying hot-search storage prices for every byte. StarTree enables fast investigation over Iceberg-backed logs while keeping the storage layer cost-efficient.
Observability and Incident Investigation
SREs need to filter, aggregate, and drill into logs quickly during incidents. Slow queries extend time to resolution. StarTree supports interactive investigation across high-cardinality operational data.
Customer-Facing Operational Analytics
Some companies expose log-derived analytics to customers, support teams, or internal operations teams. These experiences need consistent latency and high concurrency, not best-effort query performance.
Security and Forensics
Security teams often search for rare events across massive datasets. Iceberg provides retention economics. StarTree provides the precision query layer needed to make investigations practical.
AI and Agentic Operations
AI agents and conversational interfaces are only useful if the underlying data access is fast enough to preserve the interaction loop. StarTree can serve as the low-latency analytical backend for agents querying operational and log data.
The Architecture Tradeoff
The market perception is that Iceberg delivers low-cost storage but slower analytics.
That perception exists for a reason: object storage is not designed to behave like a low-latency search index by default.
StarTree changes that tradeoff.
The point is not that every Iceberg query should run through StarTree. The point is that the high-value, SLA-sensitive log analytics workloads need a purpose-built execution layer.
Use general-purpose engines for broad exploration, transformation, and federation.
Use StarTree when the workload requires:
- Sub-second interactivity
- High concurrency
- High-cardinality filtering
- Nested JSON queries
- Cost-efficient object storage access
- Production reliability
- Real-time or near-real-time analytics
That is the practical architecture: Iceberg for open storage, StarTree for fast operational analytics.
Conclusion: From Log Archive to Log Analytics Platform
Moving logs to Iceberg is an important first step, but it is not the full solution.
Without a high-performance query layer, the lakehouse risks becoming another cold archive: cheap to store, but too slow for the teams that need answers during live operational workflows.
StarTree turns Iceberg into a practical foundation for log analytics by making queries faster, more precise, and more cost-efficient.
Apache Iceberg changes the storage economics. StarTree changes the query economics.
