The Magic of Tiered Storage in StarTree Cloud | Neha Pawar & Tim Berglund
In this lightboard, Neha Pawar and Tim Berglund concoct a compelling metaphor comparing the concept of Tiered Storage in StarTree Cloud to the wizarding world of Harry Potter. The eccentric new Minister of Magic (representing a cost-sensitive cloud solution) has issued a mandate that all wizards must lug around every single item they might need at all times. The paraphernalia is kept in weighty trunks, requiring Hippogriffs and elf buddies for transportation, mirroring the burdensome cost and performance issues related to scaling storage.
Neha and Tim propose a more efficient, loosely coupled system akin to a magical charm, which allows for local, network attached, and cloud-based storage, reducing costs. Apache Pinot fetches specific data “pages” based on a query to minimize latency, and employs advanced strategies like selective columnar fetch and automatic local pinning based on usage patterns. Their approach has led to significant improvements in query response times, effectively demonstrating the efficiency of the system.