Skip to content

Latest commit

 

History

History
400 lines (290 loc) · 13.1 KB

README.md

File metadata and controls

400 lines (290 loc) · 13.1 KB
Raw

Redis Timeseries Lex

Redis Timeseries is a Lua library implementing queryable time series on top of Redis using lexicographically sorted sets.

Encoding

Each data point is stored as a ZSET entry with score 0 and value in the form timestamp|value, where value is a set of arbitrary key value pairs encoded using message pack. The encoding is to preserve space, as well as to preserve numeric values.

Because of Lua => Redis conversion issues ( float truncation specifically ) we try to preserve float values by converting them to string as necessary on return from command calls.

Testing

A set of tests (using node/jest) are available in __tests__. To run these, you will need node. If you have node installed:

npm install
npm run test

If you have Redis running somewhere other than localhost:6379, you can supply the REDIS_URL environment variable:

REDIS_URL='redis://host:port' npm run test

Structure

We recommend using git submodules to keep redis-timeseries-lex in your bindings.

Usage

Load the script into redis, using SCRIPT LOAD, or better yet, use any of the redis client libraries available for the programming language of your choice. The docs assumes that you have successfully done so and have acquired the associated sha. For all implemented commands, the script takes a single key representing the timeseries. The general format of calls is as follows

evalsha sha 1 key command [args1 ...]

For example

evalsha b91594bd37521... 1 wait-time:5s range 1548149180000 1548149280000 AGGREGATION 5000 max(duration)

Commands

add

Add one or more data points to the timeseries at key. An abitrary number of key-value pairs can be specified.

evalsha sha 1 key timestamp [key value ...]

Values which can be interpreted as numbers will be stored and treated as such.

Example

evalsha b91594bd37521... 1 readings:temp add 1564632000000 temperature 18 altitude 500 geohash gbsvh tag important

get

Get the value of a key associated with timestamp

evalsha sha 1 key get timestamp hash_label [LABELS field ...] [REDACT field ...]
Return Value

Array reply, JSON or msgpack depending on FORMAT option

The command returns the key-value data associated with a timestamp. The returned order of fields and values is unspecified.

Examples


redis-cli> evalsha b91594bd37521 1 purchases add 1564632000000 item_id cat-987H1 cust_id 9A12YK2 amount 2500
1564632000000
redis-cli> evalsha b91594bd37521 1 purchases get 1564632000000
1564632002100
1) "item_id"
2) "cat-987H1"
3) "cust_id"
4) "9A12YK2"
5) "amount"
6) "2500"

The LABELS and REDACT options can be used to specify which fields are returned


redis-cli> evalsha b91594bd37521 1 purchases add 1564632000000 item_id cat-987H1 cust_id 9A12YK2 amount 2500
1564632000000
redis-cli> evalsha b91594bd37521 1 purchases get 1564632000000 LABELS item_id amount
1564632002100
1) "item_id"
2) "cat-987H1"
5) "amount"
6) "2500"
redis-cli> evalsha b91594bd37521 1 purchases get 1564632000000 REDACT cust_id
1564632002100
1) "item_id"
2) "cat-987H1"
5) "amount"
6) "2500"

del

Removes the specified members from the series stored at the given timestamp. Non existing members are ignored. Returns the number of deleted entries.

evalsha sha 1 key del [timestamp ...]

Example

evalsha b91594bd37521... 1 temperature del 1564632000000 1564632010000
Return Value

Integer reply: the number of entries actually deleted.

pop

Remove and return a value at a specified timestamp

evalsha sha 1 key pop timestamp [LABELS label ...] [REDACT label ...]

Example

evalsha b91594bd37521... 1 temperature pop 1564632000000

size

Returns the number of items in the timeseries

evalsha sha 1 key size
Return Value

Integer reply: the number of items in the series.

count

Returns the number of items in the timeseries between 2 timestamps. The range is inclusive. Note that this and all other commands expecting a range also accept the special characters - and + to specify the lowest and highest timestamps respectively.

evalsha sha 1 key count min max [FILTER condition ...]

Example

evalsha b91594bd37521... 1 temperature count 1564632000000 1564635600000
Return Value

Integer reply: the count of items in the subset defined by the given range and filter.

exists

Checks for the existence of a timestamp in the timeseries.

evalsha sha 1 key exists timestamp

Example

evalsha b91594bd37521... 1 temperature exists 1564632000000
Return Value

Integer reply: 1 if the timestamp exists, 0 otherwise.

span

Returns a 2 element array containing the highest and lowest timestamp in the series.

evalsha sha 1 key span

times

Returns a list of timestamps between min and max

evalsha sha 1 key min max

set

Set the value associated with timestamp.

evalsha sha 1 key set timestamp hash_label1 value [hash_label2 value ...]
evalsha b91594bd37521... 1 rainfall set 1564632000000 geohash gbsut inches 2.0

incrBy

Increment the values of fields associated with timestamp.

evalsha sha 1 key incrBy timestamp hash_label1 value [hash_label2 value ...]

example

evalsha b91594bd37521... 1 wait_time incrBy 1564632000000 count 1 value 2000

Querying

range/revrange/poprange

Query a timeseries by range and optionally aggregate the result. revrange returns the range of members in reverse order of timestamp, but is otherwise to its non-reversed counterpart. poprange functions similarly to range, but removes the data in the range before returning it.

evalsha sha 1 key [range|revrange|poprange] min max [FILTER condition ....] [AGGREGATION timeBucket rollup(field) ...] [LABELS label ....] [REDACT field ...] [FORMAT [json|msgpack]]
  • key the timeseries redis key
  • min the minimum timestamp value. The special character - can be used to specify the smallest timestamp
  • max the maximum timestamp value. The special character + can be used to specify the largest timestamp

min and max specify an inclusive range.

Options

FILTER

The range commands support a list of query filters.

  • field=value field equals value
  • field!=value field is not equal to value
  • field>value field greater than value
  • field>=value field greater than or equal value
  • field<value field less than value
  • field<=value field less than or equal value
  • field=null value is null or the field field does not exist in the data
  • field!=null the field has a value
  • field=(v1,v2, ...) field equals one of the values in the list
  • field!=(v1,v2, ...) field value does not exist in the list.

Any number of filter conditions can be provided, and the results are ANDed by default, however you may use an OR if required.

FILTER purchase_price>5000 OR customer_status=preferred

For contains comparisons, avoid leading and trailing spaces with numbers. In other words use

FILTER size!=(10,20,30)

instead of

FILTER size!=(10 , 20, 30)

The parser supports quoted as well as non quoted strings e.g.

FILTER state!=(ready,"almost done",burnt to a crisp)

String as well as numeric comparisons are supported.

evalsha b91594bd37521... 1 game_scores range - + FILTER tag=playoffs value>10
AGGREGATION

A timeseries range can be rolled up into buckets and aggregated by means of the AGGREGATION option :

evalsha sha 1 key range_command min max AGGREGATION timebucket aggegation(field) [ aggregation(field) ... ]
  • timeBucket - time bucket for aggregation. The units here should be the same as used when adding data.
  • aggregation - avg, sum, min, max, median, stdev, range, count, first, last, stats, distinct, count_distinct
  • field - the field to aggregate
Aggregation Description
first the first valid data value
last the last valid data value
min minimum data value
max maximum data value
avg mean of values in time range
median median of values in time range
stdev standard deviation of values in time range
count the number of data points
range the difference between the max and min values
sum the sum of values
distinct the list of unique values in the range
distinct_count the count of unique values in the range

Example

evalsha b91594bd37521...  1 temperature:3:32 range 1548149180000 1548149210000 AGGREGATION 5000 avg(value)

For range and revrange, each key will be aggregated as appropriate, subject to any supplied LABELS. In this case, the query will return a pair of [timestamp, object] where the values of object are the aggregated values for the appropriate keys in the given min/max range.

LABELS

The LABELS option may be specified to limit the fields returned from a query or available for aggregation. If this option is specified, only the mentioned fields are returned from queries (opt-in).

evalsha b91594bd37521... 1 temperatures range - + FILTER temperature>=45 LABELS name lat long
REDACT

REDACT is used to specify fields which should NOT be returned from a query. All fields not explicitly specified are returned (opt-out)

evalsha b91594bd37521... 1 purchases range - + FILTER amount>=5000 REDACT credit_card_number
FORMAT

Selects the output format of query. Currently json and msgpack are supported. If not specified, range and aggregation responses are sent using a redis multi-bulk reply.

evalsha b91594bd37521... 1 purchases range - + FILTER amount>=5000 FORMAT json

copy

Executes a range and copies the result to another key.

evalsha sha 2 src dest COPY min max [FILTER condition ....] [AGGREGATION timeBucket aggregate(field) aggregate(field) ...] [LABELS label ....] [REDACT field ...] [STORAGE ["timeseries"|"hash"]]
  • key the timeseries redis key
  • min the minimum timestamp value. The special character - can be used to specify the smallest timestamp
  • max the maximum timestamp value. The special character + can be used to specify the largest timestamp

min and max specify an inclusive range.

Options

All options for range are accepted with the exception of FORMAT. In addition we may specify a STORAGE option

  • timeseries (default) store results in a timeseries sorted set
  • hash stores the result in a hash where the key is the timestamp

merge

Merges a range of values from 2 time series and copies the result to another key. Numeric fields are added together and non-numeric fields are ignored in the output

evalsha sha 3 first second dest MERGE min max [FILTER condition ....] [LABELS label ....] [REDACT field ...] [STORAGE ["timeseries"|"hash"]]
  • first the first timeseries redis key
  • second the second timeseries redis key
  • dest the destination timeseries redis key
  • min the minimum timestamp value. The special character - can be used to specify the smallest timestamp
  • max the maximum timestamp value. The special character + can be used to specify the largest timestamp

min and max specify an inclusive range.

Options

All options for range are accepted with the exception of FORMAT and AGGREGATE. In addition we may specify a STORAGE option

  • timeseries (default) store results in a timeseries sorted set
  • hash stores the result in a hash where the key is the timestamp