How 7Signal Reengineered Its Data Pipeline for Real-Time Analytics with Apache Pinot
At 7Signal, a company specializing in digital experience monitoring over Wi-Fi networks, an evolution in user expectations forced a fundamental rethink of how it ingested, processed, and served telemetry data to customers. Fortunately, recent enhancements to Apache Pinot made it the right choice at the right time, according to Joe Tennant, 7 Signal’s Vice President of Software Development. Tennant told his story at the Real-Time Analytics Summit in May 2025, hosted by StarTree.
7Signal’s wireless intelligence platform helps customers optimize wireless network availability and performance. The solid, performant data platform based on Apache Flink and relational databases that the company had used for years eventually met its match with a simple customer request: “Can I see what’s happening right now?”
7Signal navigated a series of trade-offs to ultimately replace its Flink-based stream processing pipeline with Apache Pinot. The revamped platform balances performance, complexity, and operational overhead in a way that makes sense for a small, focused team.
Fast, but delayed
The company’s endpoint monitoring platform previously used agents running on client devices to deliver a continuous stream of Wi-Fi and network performance metrics through Kafka to Apache Flink. The data was pre-aggregated into 15- and 120-minute time buckets and written to a PostgreSQL database.
Preprocessing ensured snappy query performance of less than 500 milliseconds. The database schema was tailored to the user interface, with tables partitioned each day and queries written in SQL, a language favored by the development teams.
But preprocessing takes time. Summarized records weren’t written until a tumbling window closed, along with a buffer for late-arriving data. That resulted in 20-minute delays before new metrics appeared in the UI. While great for historical and trend analysis, this lag proved unacceptable for users who wanted real-time visibility into device behavior.
The team reassessed its priorities with several tradeoffs in mind. They wanted to retain fast query speeds and show data immediately, a tricky combination. They hoped to preserve SQL, but the language wasn’t originally designed to handle time-ordered event streams, correlation across multiple streams, and event pattern detection. There was also a risk that new architecture would be more complex to manage.
One alternative was to modify Flink to update partial records during the window. However, that introduced fragmentation that degraded performance. Another option was to query raw data directly from Kafka or S3, a solution that was technically feasible but complex and inconsistent with the existing app flow.
Ideal fit
Each option solved some problems and introduced new ones. That’s when Apache Pinot entered the picture. The real-time distributed data store hadn’t been mature enough for 7Signal’s needs when the original reporting platform was built, but recent architectural enhancements made it an ideal fit. Pinot supports fast ingestion from Kafka, real-time query capabilities, rich indexing options, and SQL. It was also designed to serve user-facing applications, not just internal analytics dashboards.
The revamped architecture 7Signal built has data flowing directly from Kafka into Pinot, eliminating the preprocessing step. Aggregation happens at query time. This change immediately solved the “right now” visibility problem because metrics can be queried the moment they arrive.
Query performance remained fast, aided by inverted indexes on low-cardinality fields such as device type and location along with range indexes on timestamps. The team further boosted performance by pre-bucketing timestamps into multiple granularities to support more efficient group-bys and filters.
There were tradeoffs. Hosting Pinot can be complex. It’s a clustered system with brokers, servers, and minions. It also introduced a learning curve and operational overhead that challenged the small team. Pinot’s relative newness also means that bugs were a fact of life.
Managed option
To manage these trade-offs, 7Signal chose StarTree’s managed Pinot service. This let the team offload hosting complexity in exchange for a somewhat higher cost. It was an acceptable trade-off given their small size and limited bandwidth, Tennant said.
The move to Pinot accomplished its primary goal: real-time visibility. It also simplified the pipeline by removing Flink from the ingestion flow and consolidating processing and storage in a single system. The team retained the fast queries and beloved SQL interface that had worked so well in the past.
Pinot’s youth means the team still deals with some bugs and rough edges, but the trade-offs have been well worth it, Tennant said. The platform met evolving customer demands while maintaining performance and a manageable developer experience.
This story isn’t about the virtues of Pinot versus Flink, he said; it’s about understanding the trade-offs that shape system architecture over time. At 7Signal, what worked during one phase of the company’s growth eventually created friction during another. The developers didn’t just react; they re-evaluated, re-prioritized, and documented everything. Because other developers may face similar choices in the future.