StarTree Cloud cluster health dashboard
The StarTree Cloud cluster health dashboard provides an overview of Pinot checks, which lets you observe pass/fail statuses and filter checks based on instances or tables. This dashboard offers a holistic view of the overall health of the cluster.
The ClusterHealthCheckTask task runs every 20 minutes by default. Dashboard checks are cached and kept in memory, and then overwritten with every run.
To use checks ad-hoc, use these controller API calls:
GET - /periodictask/run?taskName=ClusterHealthCheckTask(to run the checks now)GET - /clusterHealth(to fetch cluster health)GET - /clusterHealth/list(to list all available cluster health checks)
To view the cluster health dashboard
Log into StarTree Cloud and do the following:
- Click the organization, then select the workspace you want to view monitoring metrics for.
- Click the Services tab.
- Click the link next to My Apps.
- Click the Pinot Control Panel tile.
A dashboard containing a list of checks appears, and indicates whether the check passes or fails, and additional details about the check.
List of health checks
Current health checks are listed here.
Checks if a table has any segments whose ExternalView state does not match with IdealState
IDEAL_STATE_EV_MISMATCH_CHECK
Checks if a table has any segments whose ExternalView state does not match with IdealState
TABLE_SEGMENTS_RELOAD_CHECK
Checks if a table has any segments to be reloaded
SEGMENT_COUNT_CHECK
Checks if a table has more than 50000 segments
SEGMENT_SIZE_CHECK
Checks if a table has more than quarter of it’s segments of size less than 5MB
SEGMENT_RETENTION_CHECK
Checks if a table does not have segment retention configured
REPLICATION_CHECK
Checks if a table has replication factor less than the configured threshold in controller config (controller.replication.threshold) which is by default 3 if unspecified
TABLE_COLUMN_COUNT_CHECK
Check if the number of columns in a schema exceeds 500
TIME_COLUMN_GRANULARITY_CHECK
Checks if a table has any time columns with granularity set to MILLISECONDS / MICROSECONDS / NANOSECONDS
UPSERT_TABLE_SEGMENT_ASSIGNMENT_CHECK
For an upsert table, checks if segments of a partition for a single replica group are assigned to more than one server
INSTANCE_HEALTH_API_CHECK
Checks if any of the instance /health API is not live
SEGMENT_SKEW_HEALTH_CHECK
Checks whether any server associated with a specific table that has more than 50 segments has a segment count exceeding 50% of the average segment count across all servers for that table
CONSUMING_PARTITION_SKEW_HEALTH_CHECK
Checks whether any server associated with a specific table having more than 10 consuming segments has a consuming segment count that exceeds 50% of the average consuming segment count across all servers for that table
TABLE_SKEW_CHECK
Checks whether the number of tables hosted by a server exceeds 50% of the average number of tables hosted across all servers
CLUSTER_LEVEL_SEGMENT_SKEW_CHECK
Checks whether the number of segments on any server exceeds 50% of the average segment count across all servers
CLUSTER_LEVEL_CONSUMING_PARTITION_SKEW_CHECK
Checks whether the number of consuming segments on any server exceeds 50% of the average consuming segment count across all servers
HIGH_NUMBER_OF_SEGMENTS_CHECK
Checks whether the number of segments hosted on the server exceeds 10000
HIGH_NUMBER_OF_CONSUMING_PARTITIONS_CHECK
Checks whether the number of consuming segments hosted on the server exceeds 50
HIGH_NUMBER_OF_DOCUMENTS_CHECK
Checks whether the number of documents hosted by the server is greater than 1 Billion
HIGH_NUMBER_OF_TABLES_CHECK
Checks whether the number of tables hosted by the server is greater than 10
DATA_SIZE_SKEW_CHECK
Checks whether the skew in the size of data hosted by instances is asymmetric, specifically if the skew is not in the range [-2, 2]
CONTROLLER_TABLES_SKEW_CHECK
Checks whether the skew of table count hosted by lead controllers is asymmetric, specifically if the skew is not in the range [-2, 2]
HIGH_UPSERT_PK_PER_PARTITION_CHECK
Checks if a partition hosted by an instance has a primary key count exceeding 500 million
UPSERT_PRIMARY_KEY_SKEW_CHECK
Checks whether the skew in the distribution of primary keys among servers is asymmetric, specifically if skew is not in the range [-2, 2]
UPSERT_PREBUILT_SNAPSHOT_CHECK
Checks for any upsert table in a cluster if the prebuilt snapshot feature is disabled for preloading
UPSERT_PK_SKEW_PER_PARTITION_CHECK
Checks whether the skew for primary keys in table partitions is asymmetric, specifically if the skew is not in the range [-2, 2]
HIGH_UPSERT_PRIMARY_KEYS_CHECK
Checks if the primary key count for all tables hosted by the instance exceeds 800 Million
INSTANCE_POOLS_CHECK
Checks if any server is not part of instance pools or any pool is not disjoint if the cluster has more than 8 servers
INSTANCE_POOLS_N_REPLICA_GROUPS_CHECK
Checks if any table is not using instance pools and replica groups configurations
HELIX_HOST_NAME_INSTANCE_NAME_MISMATCH_CHECK
Checks if the Instance ID/Name does not match with the expected value derived from Instance Config
TABLE_SEGMENT_ASSIGNMENT_CHECK
Checks if re balance of segments is needed on the table
BROKER_RESOURCE_CHECK
Checks if broker resource for a table does not contain any of the brokers in the tenant
QUERY_HEALTH_CHECK
Checks if “select $docId from TABLE_NAME limit 1” executes unsuccessfully for any table in a cluster
SEGMENTS_TIME_PARTITION_CHECK
Checks if any segments of a table are not partitioned by time i.e if the current segment end time is less than prev segment end time given that segments are sorted by their start times and end times