7Signal’s wireless intelligence platform helps customers optimize wireless network availability and performance. Its’ prior data platform based on Apache Flink and relational databases that the company had used for years eventually met its match with a simple customer request: “Can I see what’s happening right now?”
In this talk from Real-time Analytics Summit, Joe Tennant (VP of Software Engineering) shares their story of how they migrated their analytics to Apache Pinot to make this possible:
The Challenge: The Latency Trade-Off
7SIGNAL ingests approximately 35 million metrics per hour to monitor enterprise Wi-Fi performance. To ensure fast dashboard loading (SLA <500ms), the engineering team originally relied on a stream-processing architecture using Apache Flink to pre-aggregate data into Postgres buckets (15-minute and 120-minute tumbling windows).
While this architecture guaranteed stable read performance, it created an unavoidable trade-off — data freshness.
- The bottleneck: Because data was only written after the tumbling window closed (plus buffer time for late arrivals), customers experienced a 20-23 minute lag between an event occurring and it appearing in the dashboard,.
- The friction: As customers demanded real-time visibility for immediate troubleshooting, the “pre-baked” aggregation model became a blocker.
The Solution: Shifting Aggregation to Query-Time
To eliminate latency without sacrificing query speed, 7SIGNAL re-architected their pipeline using Apache Pinot on StarTree Cloud, shifting from “write-time aggregation” to “query-time aggregation.”
Architecture Changes:
- Simplified Pipeline: The team removed the complex Flink pre-processing logic for this workload. The new flow is streamlined:
Agents > Kafka > Pinot. - Ingestion: Data is written directly from Kafka into Pinot segments, making it immediately available for querying.
Technical Implementation & Optimization:
7SIGNAL utilized specific Pinot indexing strategies to maintain sub-second query speeds despite handling raw data:
- Inverted Indexes: Applied to low-cardinality fields (e.g., Device ID, Platform, Location) to accelerate filtering.
- Range Indexes: Applied to timestamps to optimize time-slice queries.
- Pre-Truncated Bucketing: To balance raw data access with aggregation speed, they store truncated timestamp versions (1-min, 15-min buckets) alongside raw timestamps. This allows the engine to group by time intervals efficiently without full pre-aggregation.
Why StarTree Cloud?
While Pinot solved the architectural requirement, hosting it introduced operational complexity (managing brokers, minions, and deep storage). As a lean engineering team, 7SIGNAL chose StarTree Cloud to offload the operational burden of the cluster, allowing them to focus purely on application logic and data modeling.
Key Results
- Real-Time Visibility: Data freshness delays dropped from ~23 minutes to <5 minutes,.
- Development Velocity: The cycle for deploying new aggregations/widgets dropped from 4 months to 1 month.
- Performance: Queries run 2-10x faster compared to the legacy implementation,.
- Cost Efficiency: Achieved >50% cost savings for time-series storage and aggregation compared to the previous AWS-based solution.
The move to Pinot accomplished its primary goal: real-time visibility. It also simplified the pipeline by removing Flink from the ingestion flow and consolidating processing and storage in a single system. The team retained the fast queries and beloved SQL interface that had worked so well in the past.
Get Started with Apache Pinot on StarTree Cloud
The best way to get experience real-time analytics and Apache Pinot is with StarTree Cloud. Request a trial to get set up with an instance of Apache Pinot for testing and development. Or Book a demo if you have questions and would like a quick tour of how StarTree Cloud can work for you.
