A customer was streaming massive volumes of OpenTelemetry data through Kafka into Apache Pinot. Their queries needed to meet strict sub-second latency requirements, and the architecture relied heavily on partition pruning to keep query fan-out under control.
But pruning wasn’t working.
The customer believed Kafka was partitioning records on correlationId using an FNV-1a hash. Pinot was configured to reproduce that behavior, but the data was still landing in partitions Pinot didn’t expect. We eventually found that the upstream system was doing something slightly different: it was applying an MD5 hash to the ID before applying FNV-1a.
The actual logic was:
fnv1(md5(correlationId))
That undocumented detail was enough to prevent Pinot from pruning partitions correctly. Once we gave Pinot a way to reproduce the upstream partitioning logic, p99 query latency improved from ~10s to 100s of ms. A clean 100x improvement.
Why Kafka and Pinot need to agree on partitioning
In a pipeline such as:
OpenTelemetry → Kafka → Apache Pinot
Kafka decides which partition receives each event. Pinot ingests that data into segments and keeps partition metadata that its query broker can use when routing queries.
If a query filters on a partition key such as correlationId, Pinot can use the same partitioning logic as Kafka to work out where matching records should be located. Instead of sending the query across the cluster, the broker can target only the servers that could contain the relevant data.
For observability workloads, where large volumes of events may be queried under heavy concurrency, avoiding unnecessary fan-out can make a substantial difference to both latency and resource use.
The catch is that Pinot has to reproduce the upstream partitioning logic exactly.
Finding the mismatch
The customer believed their Kafka producer was partitioning data using:
fnv1(correlationId)
At the time, Pinot didn’t natively support the required FNV-1a function, so we added support for it.
That didn’t fix the problem. Partition pruning was still failing, and the data was appearing in different partitions from the ones Pinot calculated.
We inspected the partition metadata stored in Pinot’s segments and compared the expected routing with where the data had actually landed. That investigation exposed the missing step: the producer wasn’t applying FNV-1a directly to correlationId. It was first generating an MD5 hash and then hashing that result.
In other words:
fnv1(md5(correlationId))
The pipeline had a partitioning rule that mattered to query performance, but that rule wasn’t documented anywhere Pinot could see it.
Making the partition function configurable
We could have added a specific fnv1_md5 function to handle this pipeline, but that would only have solved one implementation.
Production pipelines often contain behavior like this. Producers use different client libraries, stream processors transform keys, and older systems may have partitioning logic that is difficult or undesirable to change. Treating each variation as a separate special case doesn’t scale particularly well.
Instead, we built a generic Custom Hash Function framework that lets the partitioning logic be defined directly in the Pinot table configuration.
For this pipeline, the configuration could reproduce the upstream behavior explicitly:
{
"tableIndexConfig": {
"segmentPartitionConfig": {
"columnPartitionMap": {
"correlationId": {
"functionName": "Custom",
"numPartitions": 64,
"partitionFunctionProperties": {
"customFunction": "fnv1(md5(correlationId))"
}
}
}
}
}
}Code language: JSON / JSON with Comments (json)This gives Pinot a way to match the partitioning scheme that already exists upstream rather than requiring the producer to conform to a fixed set of supported functions.
It also makes the mechanism useful beyond this particular customer. Other pipelines can define their own transformations while still allowing Pinot to understand where data should be located.
The effect on query performance
Once Pinot reproduced the upstream partitioning logic correctly, partition pruning started working as intended. Queries could be routed to the relevant portion of the cluster rather than involving servers that could not contain matching records.
In this case, the result was a 100x improvement in p99 query latency, enough for the workload to meet its sub-second latency requirements while reducing unnecessary cluster work.
Partitioning is part of the interface between systems
This kind of issue is easier to hide in a vertically integrated system, where one vendor controls data production, storage, and query execution. In a disaggregated stack, those boundaries are exposed, which gives engineering teams more flexibility but also makes the interfaces between systems more important.
Partitioning is one of those interfaces. A Kafka producer and a query engine may be independently implemented, but if Pinot is going to use partition information to optimize query routing, both systems need a shared understanding of how records are assigned.
In practice, that understanding may involve more than choosing the same named hash function. Transformations, custom producer logic, and inherited implementation details can all become part of the contract.
With a configurable partition function, Pinot can adapt to those details without requiring engineers to rebuild the upstream pipeline around the database. For this customer, that was the difference between ineffective pruning and sub-second queries.
