DataFetcher

Fetches data from a datasource.

Inputs

None.

Outputs

One output. No naming constraint for outputKey and outputName.

Parameters

name	description	default value
`component.dataSource`	ThirdEye datasource to use.
`component.query`	Query to execute. The query can use macros.

Example

{
  "name": "currentDataFetcher",
  "type": "DataFetcher",
  "params": {
    "component.dataSource": "pinot",     # ThirdEye datasource
    "component.query": "SELECT __timeGroup(timeColumn, 'EPOCH', 'P1D') as ts, sum(views) as met FROM pageviews WHERE __timeFilter(timeColumn, 'EPOCH') GROUP BY ts ORDER BY ts LIMIT 10000"
  },
  "inputs": [],
  "outputs": [
    {
      "outputKey": "currentData",        # no constraint
      "outputName": "currentOutput"      # no constraint
    }
  ]
}

Macros

Macros are special functions that are translated by ThirdEye to SQL before the query is sent to the database. Macros have access to detection runtime information, like the start and the end time of the detection, the timezone.

Macro example	description
`__timeFilter(` `timeColumn,` `'timeFormat'` `)`	Replaced by a time range filter on the timeframe `[detectionStart, detectionEnd[` using the specified column name and the specified timeFormat. For example, `__timeFilter(timeColumn, 'EPOCH_MILLIS')` is replaced by `timeColumn >= 1494410242000 AND timeColumn < 1494410242000)`
`__timeFilter(` `timeColumn,` `'timeFormat,'` `'lookbackFromStartPeriod'` `)`	Replaced by a time range filter on the timeframe `[detectionStart-lookbackFromStartPeriod, detectionEnd[`. For example, for a detection between January 14 and January 15, `__timeFilter(timeColumn, 'yyyyMMdd', 'P7D')` is replaced by `timeColumn >= '20220107' AND timeColumn < '20220115')`
`__timeFilter(` `timeColumn,` `'timeFormat,'` `'lookbackFromStartPeriod,'` `'lookbackFromEndPeriod'` `)`	Replaced by a time range filter on the timeframe `[detectionStart-lookbackFromStartPeriod, detectionEnd-lookbackFromEndPeriod[`. For example, for a detection between January 14 and January 15, `__timeFilter(timeColumn, 'yyyyMMdd', 'P7D', 'P7D')` is replaced by `timeColumn >= '20220107' AND timeColumn < '20220108')`
`__timeGroup(` `timeColumn,` `'timeFormat',` `'granularity'` `)`	Replaced by a transformation of the timeColumn in bins of the given granularity, in milliseconds. Respects the timezone of the alert. For example, `__timeGroup(timeColumn, 'yyyyMMddhh', 'P1D')` is replaced by `DATETIMECONVERT(timeColumn, '1:DAYS:SIMPLE_DATE_FORMAT:yyyyMMddhh', '1:MILLISECONDS:EPOCH', '1:DAYS')`

AUTO keyword

If the timeColumn value is AUTO, the macro resolves to the primary time column of the table.
timeFormat is obtained internally and can be left blank.

For example:

__timeFilter(AUTO,'')
__timeGroup(AUTO,'','P1D')

timeFormat strings

Available timeformat strings are:

String	description	aliases
`EPOCH_MILLIS`	time column is in milliseconds since unix epoch	`1:MILLISECONDS:EPOCH`
`EPOCH`	time column is in seconds since unix epoch	`1:SECONDS:EPOCH`
`EPOCH_MINUTES`	time column is in minutes since unix epoch	`1:MINUTES:EPOCH`
`EPOCH_HOURS`	time column is in hours since unix epoch	`1:HOURS:EPOCH`
`EPOCH_DAYS`	time column is in days since unix epoch	`1:DAYS:EPOCH`
any SimpleDateFormat string	See SimpleDateFormat documentation. Eg `yyyyMMdd`, `yyyy-MM-dd'T'HH:mm:ss.SSSZ`	`1:DAYS:SIMPLE_DATE_FORMAT:[SIMPLE_DATE_FORMAT]`

period strings

Period strings uses the ISO-8601 format. Examples:

'P7D': 7 days
'PT1H': 1 hour