All data in Shooju is represented as series. You find series using queries.
Series are the only things that can be stored at Shooju.
- Series are composed of:
- Series ID
- Fields
- Field Values
- Fields can be:
- Internal fields
- Set field
- Meta Fields
- Dynamic Fields
- Field Value Templates
- Points
Series are composed of:
- sid: a unique identifier called a Series ID (sid). Example:
Weather\Japan\Tokyo
. - Fields: an association between names and values. Example:
{ city: "Tokyo", aspect: "Temperature" }
. - Points (optional): a list of dates and numbers. Example:
[[2020-06-01, 25.6], [2020-12-02, -3.0]]
.
All versions of a series throughout time are saved and can be retrieved in the Shooju website or with the @asof
Operators
Series ID
A unique identifier (sid) is used when information is called from the Shooju database and needs to be distinguished from other information in the database. The Series ID (sid) is a tree-like structure separated by \. The following structured query is an example:
sid=Weather\Japan\Tokyo
Fields
A field is an area in Series Explorer that accepts input from the user.
A Shooju series may have 0 or more fields. Each must have two key elements: the field name, and its value. The field name must be at least 4 characters long and may only contain numbers, underscores (except as the first character), and lower-case letters. The fields of a series may look like this:
field | value | notes |
city | Tokyo | |
aspect | Temperature | |
description | Tokyo, Japan Temperature | The description field is often used as a human-readable description, but it's not required. |
Field Values
- All field values are saved in the submitted json-style format (boolean, numeric, string, object, and arrays).
- Field value can be arrays.
- By default field values are processed as text for purposes of finding,
Queries , and
Facets series. To enable type-specific processing and type checking, use the field suffixes below:
Field Suffix | Meaning | Example Values | notes |
_date | Date | 2015-01-01
2015-01-01T12:42
1642204799000 | ISO-formatted (as YYYY-MM-DD or YYYY-MM-DDThh:mm) or milliseconds since epoch (same as native javascript format). Note: does not support localization; only stores and retrieves one date representation. |
_geo | GeoJSON | {“type”: “point”, “coordinates”: [10, 50]} | Fully supports GeoJSON. The only difference is that the type is lower case. |
_num | Number | 42.67 | Any numeric value (stored as a float). |
_obj | Object | {“hello”:”world”} | The keys of the object become normal Shooju fields and are accessible through the dot-notation (e.g. me_obj.hello). |
_tree | Tree | this\is\a\tree | \-delimited text with special tree-like query abilities. |
- normal: accepts input from the user
- internal: cannot be changed by the user
Fields can be:
- normal: accepts input from the user
- internal: cannot be changed by the user
Internal fields
- set: lists fields in a series
- meta: prefixed with meta
- dynamic: changes values for each query
Set
field
"set" is a field that holds a list of every field name present in the series.
Meta Fields
Meta fields cannot be edited directly by the user and they are prefixed with meta. to avoid conflicting with similarly named normal fields. Below is the list of meta fields. Note that they are all referenced by meta.<meta field name>, like meta.updated_by.
field | example value | notes |
meta.points_count | 42 | Number of Points |
meta.points_min | 6.7 | Minimum value across all points |
meta.points_max | 7.6 | Maximum value across all points |
meta.updated_by | jane.doe | Username who last updated the series |
meta.updated_at | 1595348051000 | Timestamp of when then the last update to the series ocurred, either to points or to fields, returned as milli since epoch. Note that for XPR series the timestamp only reflects changes to fields, as points are generated dynamically. |
meta.freq | H | One-letter description of the time interval: Yearly, Quarterly, Daily, Weekly, Hourly, and X for less than Hourly) |
meta.processor | weather | ID of the |
meta.upload_preset | mydata | ID of the Uploader that last updated this series. |
meta.dates_min/max | 2015-01-01... | Minimum and maximum point date. |
meta.repdates_min/max | 2015-01-01... | Minimum and maximum reported date. |
Dynamic Fields
These fields have values that change for each query, ie, they are not fixed for the series holding them.
They are the only fields whose name may begin with an underscore.
field | example value | notes |
_query | sid=Weather\Virginia\Arlington | This gives back the |
_score | 317.59604 | Relative importance of a given series as the result of the query in comparison with other series. There is no defined range, it may go from negative to very big numbers. The score may be affected by custom score functions. |
_repdate | 1624445100000 | This returns the |
Field Value Templates
- Field templates can be used to generate new field values on the fly when retrieving fields from existing series.
- Field values can be combined, transformed, etc. using a jinja2 template.
- For example, if we have a series with <field name>:<value> of first_name:John & last_name:Smith, we can combine these and request a field generated on the fly that will give us the format: <first_name> <last_name>. The below example will return the value: 'John Smith'
- We can also transform existing field values and add static text. Consider a field that contains a datetime value futures_date:2020-01-01T00:00:00. We can shorten this to only show the date portion and add some descriptive static text. The following example will return the value: 'Future Date: 2020-01-01'.
- Note that when passing a template field inner quotes will sometimes be required. The inner quotes will need to be escaped with a backslash. Example:
={{ first_name }} {{ last_name }}
=Future Date: {{ future_date[:10] }}
"\"={{ first_name|replace('_', '') }}\""
Points
A Shooju series may have 0 or more points. Each point has two key elements: a date/time, and a numeric value. Dates with a time element are stored in UTC. The points of a series may look like this:
date | value |
2012-01-01 | 6 |
2012-01-02 | 42 |
2012-01-07 | -7 |
2012-01-12 | -6 |
Note that there is no rule to how the dates are sequenced (hourly, daily, ad-hoc, etc). However, there may only be one value for one date.
Some series have different collections of points associated with different report dates. If available for a certain series, you can retrieve points as they looked in a different report date with the @repdate
Operators .