# JSON Job

# Overview

The JSON job provides a way to download JSON files from remote systems or read files from the local file system to convert the files into series, properties, and message commands sent into the the Axibase Time Series Database.

# Workflow

# File Downloaded via HTTP

1. Download the target JSON file from a remote server.
2. Parse the file into memory as a JSON document.
3. Select objects from the JSON document with the specified JSON Path expression.
4. Build a series, property, or message command from the object's fields.
5. Each matched object is translated into a separate set of commands.
6. Repeat Steps 3-5 for each configuration setting/JSON expression.
7. Send commands into the Axibase Time Series Database.

# File on Local File System

1. Locate one or multiple files at the specified path on the local file system.
2. Parse the file into memory as a JSON document.
3. Select objects from the JSON document with the specified JSON Path expression.
4. Build a series, property, or message command from the object's fields.
5. Each matched object is translated into a separate set of commands.
6. Repeat Steps 3-5 for each configuration setting/JSON expression.
7. Send commands into Axibase Time Series Database.
8. If the Delete on Upload setting is enabled and commands were accepted by ATSD, delete the source file.
9. Repeat Steps 2-8 for each matched file.

# JSON Path

The JSON path is an expression evaluated against the JSON document to select its objects or specific fields.

• The expression starts with $, which represents the root object followed by a dot-separated path to matched objects.
• .{cname} denotes a selector of the given object's child object with name cname.
• {arr-name}[*] stands for all elements of the specified array arr-name.

Example:

$.store.book[*]

The expression will select all elements of the book array in the root's child named store.

{
    "store": {
    "book": [{
    "category": "reference",
    "author": "Nigel Rees",
    "title": "Sayings of the Century",
    "price": 8.95
    }, {
    "category": "fiction",
    "author": "Evelyn Waugh",
    "title": "Sword of Honour",
    "price": 12.99
    }]
    }
}

# Download Settings

# Conversion Settings

# JSON Fields

# Entity Fields

# Series Fields

# Property Fields

# Time Fields

# Message Fields

# Placeholders

# Examples

• Australia Bureau of Meteorology Weather
• JSON Lines
• GitHub Daily Summary
• Nginx Status
• BLS
• Fields with non alphanumeric characters

# Additional Examples

• Json Fields
  • Custom Tags
• Entity Fields
  • Default Entity
  • Entity Field
• Series Fields
  • Metric Prefix
  • Included Fields
  • Excluded Fields
  • Metric Name and Value Fields
• Property Fields
  • Property Default Type
  • Property Type Field
  • Property Key and Value Fields
• Time Fields
  • Time Default
  • Time Field
• Message Fields
  • Message Defaults
  • Message Fields

# Json Fields Examples

# Custom tags

JSON:

{
  "upstreams": {
    "demo-backend": {
      "peers": [
        {
          "active": 0,
          "responses": {
            "1xx": 0,
            "total": 0
          },
          "id": 1,
          "server": "1.2.3.4:1234"
        }
      ]
    }
  }
}

Result:

series e:tst d:2016-07-07T15:22:59.593Z t:name=demo-backend t:server=1.2.3.4:1234 t:type=upstreams.peers m:id=1 m:active=0 m:responses.total=0 m:responses.1xx=0

# Entity Fields Examples

# Default Entity

JSON:

{
  "upstreams": {
    "demo-backend": {
      "peers": [
        {
          "active": 0,
          "responses": {
            "1xx": 0,
            "total": 0
          },
          "id": 1,
          "server": "1.2.3.4:1234"
        }
      ]
    }
  }
}

• Default Entity contains placeholder ${HOST}:

  Field Name	Field Value
  Path	http://example.com
  Default Entity	${HOST}
  JSON Path	$.upstreams.demo-backend.peers.*
  Depth	2

  Result:

  series e:example.com d:2016-07-07T15:19:01.365Z m:id=1 m:active=0 m:responses.total=0 m:responses.1xx=0
  

• Default Entity contains placeholder ${PARENT(n)}:

  Field Name	Field Value
  Path	http://example.com
  Default Entity	${PARENT(2)}
  JSON Path	$.upstreams.demo-backend.peers.*
  Depth	2

  Result:

  series e:demo-backend d:2016-07-07T15:19:34.924Z m:id=1 m:active=0 m:responses.total=0 m:responses.1xx=0
  

• Default Entity contains text:

  Field Name	Field Value
  Path	http://example.com
  Default Entity	tst
  JSON Path	$.upstreams.demo-backend.peers.*
  Depth	2

  Result:

  series e:tst d:2016-07-07T15:19:34.924Z m:id=1 m:active=0 m:responses.total=0 m:responses.1xx=0
  

# Entity Field

JSON:

{
  "upstreams": {
    "demo-backend": {
      "peers": [
        {
          "active": 0,
          "responses": {
            "1xx": 0,
            "total": 0
          },
          "id": 1,
          "server": "1.2.3.4:1234",
          "type": "peer"
        }
      ]
    }
  }
}

Result:

series e:tst.peer d:2016-07-06T08:14:42.540Z m:id=1 m:active=0

# Series Fields Examples

# Metric Prefix

JSON:

{
  "has_more": false,
  "items": [
    {
      "count": 878240,
      "name": "java"
    }
  ],
  "quota_max": 10000,
  "quota_remaining": 9923
}

Result:

series e:tst d:2016-07-06T07:27:48.184Z m:mp.quota_max=10000 m:mp.items.0.count=878240 m:mp.quota_remaining=9923

# Included Fields

JSON:

{
  "has_more": false,
  "items": [
    {
      "count": 878240,
      "name": "java"
    }
  ],
  "quota_max": 10000,
  "quota_remaining": 9923
}

Result:

series e:tst d:2016-07-06T07:14:42.540Z t:has_more=false m:quota_remaining=9923

# Excluded Fields

JSON:

{
  "has_more": false,
  "items": [
    {
      "count": 878240,
      "name": "java"
    }
  ],
  "quota_max": 10000,
  "quota_remaining": 9923
}

Result:

series e:tst d:2016-07-06T07:27:48.184Z m:quota_max=10000 m:items.0.count=878240

# Metric Name and Value Fields

JSON Lines:

[{"data":[{"pitagname":"metric1","pitagvalue":350.0,"timestamp":"2016-07-01T15:59:07.6382972+05:30"}]}]
[{"data":[{"pitagname":"metric2","pitagvalue":250.0,"timestamp":"2016-07-01T15:58:07.6382972+05:30"}]}]

Result:

series e:tst d:2016-07-01T10:29:07.638Z m:metric1=350
series e:tst d:2016-07-01T10:28:07.638Z m:metric2=250

# Property Fields Examples

# Property Default Type

JSON:

{
  "upstreams": {
    "demo-backend": {
      "peers": [
        {
          "active": 0,
          "responses": {
            "1xx": 0,
            "total": 0
          },
          "id": 1,
          "server": "1.2.3.4:1234"
        }
      ]
    }
  }
}

Result:

property t:upstreams.peers e:tst d:2016-07-06T07:42:46.824Z v:state=up v:server=10.0.0.2:15431 v:backup=false

# Property Type Field

JSON:

{
  "has_more": false,
  "items": [
    {
      "count": 878240,
      "name": "java"
    }
  ],
  "quota_max": 10000,
  "quota_remaining": 9923,
  "type": "type1"
}

Result:

property t:type1 e:tst d:2016-07-06T07:46:58.874Z v:type=type1 v:items.0.name=java v:has_more=false

# Property Key and Value Fields

JSON:

{
  "has_more": false,
  "items": [
    {
      "count": 878240,
      "name": "java"
    }
  ],
  "quota_max": 10000,
  "quota_remaining": 9923,
  "type": "type1"
}

Result:

property t:type1 e:tst d:2016-07-06T07:46:58.874Z k:name=java v:quota_max=100000 v:has_more=false

# Time Fields Examples

# Time Field

JSON:

{
  "data": [
    {
      "date": "2016-01-01",
      "ok": 10,
      "fail": 2
    },
    {
      "date": "2016-01-02",
      "ok": 15,
      "fail": 2
    }
  ]
}

Result:

series e:tst d:2016-01-01T00:00:00.000Z m:fail=2 m:ok=10
series e:tst d:2016-01-02T00:00:00.000Z m:fail=2 m:ok=15

# Time Default

JSON:

{
  "2016-01-01": {
    "ok": 10,
    "fail": 2
  },
  "2016-01-02": {
    "ok": 15,
    "fail": 2
  }
}

Result:

series e:tst d:2016-01-01T00:00:00.000Z m:fail=2 m:ok=10
series e:tst d:2016-01-02T00:00:00.000Z m:fail=2 m:ok=15

# Minimum time

JSON:

{

    "count": 1848,
    "uniques": 123,
    "views": [
        {
            "timestamp": "2018-05-22T00:00:00Z",
            "count": 177,
            "uniques": 20
        },
        {
            "timestamp": "2018-05-23T00:00:00Z",
            "count": 269,
            "uniques": 15
        },
        {
            "timestamp": "2018-05-24T00:00:00Z",
            "count": 128,
            "uniques": 18
        }
    ]
}

If current time is 2018-05-23T17:00:00Z, it adds following commands to ATSD.

Result:

series e:axibase/atsd d:2018-05-23T00:00:00.000Z m:repo.traffic.uniques=6 m:repo.traffic.count=15
series e:axibase/atsd d:2018-05-24T00:00:00.000Z m:repo.traffic.uniques=5 m:repo.traffic.count=43

# Message Fields Examples

# Message Defaults

• Defaults contain placeholders:

  JSON:

  {
    "upstreams": {
      "demo-backend": {
        "peers": [
          {
            "active": 0,
            "responses": {
              "1xx": 0,
              "total": 0
            },
            "id": 1,
            "server": "1.2.3.4:1234"
          }
        ]
      }
    }
  }
  

  Field Name	Field Value
  Default Entity	tst
  JSON Path	$.upstreams..peers.
  Depth	1
  Message Default Type	${PARENT(3)}.${PARENT}
  Message Default Source	${PARENT(2)}
  Message Default	${PARENT(1)}

  Result:

  message e:tst d:2016-07-06T08:19:30.563Z t:source=demo-backend t:type=upstreams.peers m:peers
  

• Defaults contain text:

  JSON:

  {
    "upstreams": {
      "demo-backend": {
        "peers": [
          {
            "active": 0,
            "responses": {
              "1xx": 0,
              "total": 0
            },
            "id": 1,
            "server": "1.2.3.4:1234"
          }
        ]
      }
    }
  }
  

  Field Name	Field Value
  Default Entity	tst
  JSON Path	$.upstreams..peers.
  Depth	1
  Message Default Type	upstream
  Message Default Source	demo
  Message Default	test

  Result:

  message e:tst d:2016-07-06T08:19:30.563Z t:source=demo t:type=upstream m:test
  

# Additional Message Fields

JSON:

{
  "upstreams": {
    "demo-backend": {
      "peers": [
        {
          "active": 0,
          "responses": {
            "1xx": 0,
            "total": 0
          },
          "id": 1,
          "server": "1.2.3.4:1234",
          "type": "peer"
        }
      ]
    }
  }
}

Result:

message e:tst d:2016-07-06T08:19:30.563Z t:id=1 t:source=1.2.3.4:1234 t:type=peer m:""

# Configuration Example

# Settings Example

# JSON Viewer Example