The first inversion
The Big Data era began when machines started producing more data than humans.
Before then, almost every row in an enterprise database was the fossil of a human decision. Someone placed an order, clicked a link, or called for assistance. Data volume had a natural ceiling because there are only so many people, we can make decisions only so quickly, and we sleep.
Then machines started writing. A turbine reports its bearing temperature every hundred milliseconds. A wearable records heart rate throughout the day. A distributed system emits a span for every hop of every request. These are not decisions. They are measurements. And measurements have no natural ceiling: machines outnumber people, operate continuously, and record at frequencies no human could match.
That inversion created an industry known as Big Data. The term was never a product or a technology. It was a name for an inflection point, the moment when the old assumptions about data stopped holding and every layer of the stack had to be rebuilt around a new reality. The name caught on because it gave people a way to talk about a shift they could already feel.
A second inversion is now underway, and it deserves a name of its own. This time, it is happening on the other side of the database. Machines are beginning to ask the questions.
Queries used to be scarce
For most of the history of analytics, queries were expensive to produce.
A company might employ a few hundred analysts. Each analyst opens a handful of dashboards and writes a limited number of queries, mostly during business hours. Every query consumes human attention to formulate and human patience to await. A person eventually stops drilling down, testing alternatives, or adding another dimension.
This scarcity shaped the modern data stack. Capacity plans assumed a bounded population of analysts, modest concurrency, and demand concentrated around predictable hours. Because a query represented meaningful human effort, it could be relatively slow or expensive and still be worthwhile.
Agents change that assumption at the root.
An agent does not necessarily ask one question on the way to an answer. It may ask dozens. It can generate a three page SQL statement in under a second. It examines an initial result, develops a hypothesis, tests it, changes the filters, follows an anomaly, and asks again. Some of those paths will be useful. Others will be speculative and lead nowhere. But producing the next query costs the agent almost nothing.
And agents rarely work alone. The emerging pattern is the swarm: an orchestrating agent breaks a goal into parts and dispatches sub-agents to pursue each one at the same time. One researches, one verifies, one compares alternatives, one checks policy. Each of them runs its own loop of questions, and each may spawn helpers of its own. A single request from a single person can become hundreds of agents asking thousands of questions in parallel, and then disband seconds later.
Big Data was the explosion of recorded measurements. What is coming is the explosion of machine-generated questions.
Call it Query Abundance: billions of small asks, issued by swarms of agents, arriving as one enormous demand on the data platform. Just as Big Data marked the point where data stopped being scarce, Query Abundance marks the point where questions stop being scarce.
A different kind of query workload
Query Abundance is not simply today’s analytics workload with a larger query count. It changes the shape of demand.
Queries become iterative. One answer determines the next question, creating loops that may be dozens of steps deep.
They become speculative. A swarm can test five explanations in parallel, one agent per hypothesis, because discarded queries consume compute, not human effort.
They fan out. One request from one person can produce queries across transactions, customer behavior, inventory, logs, and external signals, each pursued by a different member of the swarm.
They arrive in bursts. A swarm can go from idle to thousands of concurrent queries the moment a goal is assigned, then fall silent just as quickly.
They become less predictable. Agent plans branch dynamically, and swarms grow or shrink as the work demands, so the system cannot know in advance which paths will be taken or how many questions they will require.
And tail latency matters more. A single slow query can stall an entire reasoning loop, and when an orchestrator waits on every member of the swarm, the slowest answer sets the pace for all of them, just as one slow service can delay an application composed of many service calls.
Agents go mainstream
Until recently, this looked like a pattern for engineering teams and early adopters. OpenClaw showed what happens when a personal agent is given a person’s accounts and tools and told to go get things done, and it found an eager audience among people willing to wire it up themselves.
That behavior is now going mainstream. In September, Meta introduced Muse, its personal agent, and OpenAI launched dots, personal agents built to act on a user’s behalf, with the ability to direct several of them at once on the way. Others are following. The capabilities that OpenClaw users assembled by hand are becoming a default feature for billions of people.
None of these products require companies to make their analytics available to agents. No one is mandating an API or a data contract to integrate these agents with a company’s systems. But a personal agent is judged by one thing: whether it solves its user’s problem. When it needs a delivery estimate, current inventory, a price, an appointment slot, or the status of an order, it will go to the sources that answer quickly and reliably, and it will skip the ones that do not. The companies that help agents help their users will be part of the answer. The rest will simply not be consulted.
What breaks first
Two failures become visible under Query Abundance.
The first is latency. An agent operates within a time budget. When a data call exceeds it, the agent may retry, route around the source, narrow its investigation, or proceed without the result. Unlike a frustrated analyst, it may never file a ticket explaining that slowness changed the answer.
The data platform is not criticized. It is quietly omitted from the reasoning loop. For example, if an agent is waiting for current delivery times at Grubhub, it might just move on to DoorDash or Uber Eats where the answers come back immediately. Multiply that decision across a swarm, and across millions of users’ personal agents, and the omission becomes a lost channel.
The second failure is economic. Two hundred analysts running 50 queries a day produce 10,000 queries. If swarms of agents increase that workload by even two orders of magnitude, the same organization is suddenly issuing one million queries a day, and that is before customers’ personal agents start asking questions from the outside. A platform whose cost closely tracks query volume can turn into a budget event when agents arrive. The program gets constrained or cancelled not because it failed, but because it worked.
What Query Abundance looks like in practice
Consider a finance agent monitoring cash position and settlement data. It projects a liquidity gap, then dispatches sub-agents to test how payment method, refund rates, disputes, and the timing of large receivables change that projection, all at once. As new payments clear, the swarm recalculates and recommends actions within policies established by the finance team.
One forecast may require dozens of queries. Continuous monitoring may require hundreds each day. The value comes not from one perfect question but from the swarm’s ability to keep asking as conditions change.
A rideshare driver’s agent shows the same pattern. Planning the driver’s schedule for the coming week does not start with her own few hundred trips. It starts with every trip across the platform: millions of rides broken down by hour, zone, fare, wait time between requests, and surge, along with local events and traffic. Against that full marketplace history, the agent tests hundreds of candidate weekly schedules at once, each limited by the driver’s hours, preferred times, and distance from home, to find the one that earns the most. Scaled across the whole driver base, with every driver relying on a personal agent, one week of planning becomes a continuous, high-concurrency analytical workload against the platform’s complete trip data.
An incident-triage swarm makes the pattern clearer still. The moment an alert fires, one agent queries logs, another metrics, another traces, another recent deployments, another prior incidents. Together they test a dozen hypotheses in the time a human might need to locate the right dashboard. Each answer determines what the swarm investigates next.
These examples share a common shape: high-concurrency, low-latency, filtered analytical queries against fresh data. That workload is newly important, but it is not entirely new. Customer-facing and real-time analytics systems have been solving parts of it for years.
The precedent hiding in plain sight
Apache Pinot emerged at LinkedIn because analytics had to be delivered directly to members rather than to a bounded group of analysts. A feature such as Who’s Viewed Your Profile puts an analytical query behind a product interaction. That requires fresh data, predictable response times, and enough concurrency to serve a massive user population.
Pinot was designed around this workload. At LinkedIn, it supports more than 50 applications and hundreds of thousands of queries per second across hundreds of billions of records. At Uber, Pinot supports hundreds of real-time analytics use cases, including Healthline, the company’s system for detecting and investigating crashes and exceptions across its mobile applications and more than 5,000 microservices.
For one of over a hundred data products, Uber processes an average of 36 TB of data per day and retains 45 days of history. After migrating the workload from Elasticsearch to Pinot, Uber reported a 70% reduction in infrastructure costs, saving more than $2 million annually. It also reduced CPU cores by 80%, its data footprint by 66%, and page-load times by 64%. The Uber engineering case study describes the workload and architecture; the published cost analysis provides the migration results.
Delivering a fast query once is relatively easy. Delivering millions of fast queries economically requires a different design discipline. The same architecture that made customer-facing analytics affordable now provides a blueprint for machine-facing analytics: keep data fresh, keep tail latency predictable, and minimize the work required to answer each additional question. A swarm of agents is, in effect, a very large and very impatient user population.
The workload works only if the economics work
At large query volumes, the amount of data touched per query becomes decisive. Reading 4 MB instead of 40 KB may not matter for one request. Multiplied across millions of requests, it can determine whether an application is viable.
That is why real-time analytics systems focus so heavily on avoiding unnecessary work. Columnar storage limits reads to the columns a query needs. Partition and segment pruning prevent irrelevant servers and files from receiving the query. Inverted, sorted, range, JSON, and text indexes narrow the search further. Pre-aggregating common multidimensional queries can turn an aggregation over billions of rows into a lookup over a much smaller structure.
Streaming ingestion addresses the other side of the requirement. In Pinot, events arriving from Kafka can become queryable while consuming segments are still being written. A single query can combine the most recent events with years of historical context. Upserts can preserve a current view as facts such as order status change.
These mechanisms are not merely performance optimizations. They change the economics of asking another question.
That matters because the marginal cost of producing a query is collapsing. The marginal cost of executing one must follow.
From real-time analytics to Query Abundance
The agentic era adds two requirements to this established foundation.
The first is access to the data where enterprises increasingly keep it. Large analytical datasets now often live in open table formats such as Apache Iceberg and Delta Lake. Organizations may accept creating specialized copies for critical workloads, but they increasingly resist moving or duplicating data for every new application.
StarTree has extended Pinot’s index-driven and pruning-oriented approach to data in open table formats. Rather than treating every query as a broad scan, it can use indexes to identify relevant records and fetch the required Parquet pages at finer granularity. The principle is the same one that made customer-facing analytics economical: touch less data so the system can answer more questions.
The second requirement is an interface designed for machine consumers. Agents need a governed way to discover available tools, understand what they can ask, and invoke the appropriate query path. StarTree exposes these capabilities through the Model Context Protocol, or MCP.
Together, these capabilities connect a new interface to an established execution problem. Agents may be new query producers, and they may arrive in swarms, but they still need what demanding applications have always needed: fresh data, predictable latency, high concurrency, and economics that survive scale.
The second inversion
Big Data forced infrastructure to absorb machine-generated writes. Query Abundance will force it to absorb machine-generated questions.
The winning systems will not merely answer individual queries quickly. They will remain fast as swarms fan out, remain trustworthy as agents explore, and make the next question inexpensive enough that an agent never has to decide whether it can afford to ask.
