Structured data in Drivers Assets

Some filed protocol represent data using structured types (e.g. arrays, structs etc).

Mapping such types into the Driver/Asset and Wires models cannot be done in a natural way, because the TypedValue class hierarchy only supports singleton types (except for byte array).

Alternatives

1. Represent structured types as JSON string

A possible solution for representing structured data with Wires/Assets is to encode such data in a StringValue using some serialization format.

Some of the existing drivers already follow this approach, but using different and non-standard formats.

A first possibility for solving the problem would be recommending to driver/wire component developers to use a common serialization format for structured data. A good candidate for this format can be JSON.

• pros:

  • No API changes needed
  • No changes to existing wire components needed
  • No impact on Kapua index after publishing size if compared to flattening
  • JSON strings can be published using wires publisher and stored in a database.
  • JSON strings can be shown by Drivers and Assets Web Ui section and Kapua Asset data section without modifications.
  • JSON can be produced and consumed by the Wires Script Filter out of the box (JSON.parse() and JSON.stringify()).
  • Once JSON has been parsed, data grouping is explicit.

• cons:

  • JSON String needs to be parsed before consuming the content
  • A component must know in advance that a StringValue contains JSON data
  • Need to add parameters to enable producing JSON
    • A Driver must be told to produce JSON as channel output, this can be done with additional channel configuration parameters.
    • If a JSON String should be produced, then the value.type field must be necessarily set to STRING. This implies that the value.typefield cannot be used to provide hints about data format (e.g. if the Driver should read a number, a string, or raw data as byte array).
  • Need to adapt existing components/create new ones that do JSON tranformations
  • If a JSON string is stored in a database, it is not easy or not possible to perform queries that involve inner data. Moreover, inner data cannot be indexed.

2. JSON string/byte array with content type annotation

If the previous approach is chosen, one possible extension to the model can be adding to StringValue and ByteArrayValue a field that describes the content type. In this way consumers can know that the received TypedValues contain JSON data.

This additional field should be encoded in the KuraPayload as well, otherwise it will be lost during publishing.

• pros:

  • components know that the string/byte array contains JSON
  • Web UI can display JSON in a more use friendly way

• cons:

  • Requires more changes if compared with the previous solution.

3. Introduce convention for flattening structured data

Another possibility would be representing a single structured data type by flattening it in a set of (String, TypedValue) pairs.

A possible way to perform this flattening can be using the following JSONPath-like format. The examples below show a natural flattened representation for the data of an Asset channel.

channel name: trouble-codes structured data represented as JSON:

[{
  "s": 1,
  "v": ["P001", "P002"]
}]

trouble-codes[0].s = 1
trouble-codes[0].v[0] = "P001"
trouble-codes[0].v[1] = "P002"

channel name: trouble-codes structured data represented as JSON:

{
  "1": ["P001", "P002"]
}

trouble-codes.1[0] = "P001"
trouble-codes.1[1] = "P002"

channel name: my-channel structured data represented as JSON:

[{
    "deviceObjectIdentifier": "device 4194303",
    "deviceAddress": {
      "networkNumber": 1,
      "macAddress": ["c0", "a8", "2", "1", "ba", "c0"]
    }
  },
  {
    "deviceObjectIdentifier": "device 1",
    "deviceAddress": {
      "networkNumber": 20,
      "macAddress": ["1", "0", "0", "0", "0", "0"]
    }
  }
]

my-channel[0].deviceObjectIdentifier = "device 4194303"
my-channel[0].deviceAddress.networkNumber = 1
my-channel[0].deviceAddress.macAddress[0] = "c0"
my-channel[0].deviceAddress.macAddress[1] = "a8"
my-channel[0].deviceAddress.macAddress[2] = "2"
my-channel[0].deviceAddress.macAddress[3] = "1"
my-channel[0].deviceAddress.macAddress[4] = "ba"
my-channel[0].deviceAddress.macAddress[5] = "c0"
my-channel[1].deviceObjectIdentifier = "device 1"
my-channel[1].deviceAddress.networkNumber = 20
my-channel[1].deviceAddress.macAddress[0] = "1"
my-channel[1].deviceAddress.macAddress[1] = "0"
my-channel[1].deviceAddress.macAddress[2] = "0"
my-channel[1].deviceAddress.macAddress[3] = "0"
my-channel[1].deviceAddress.macAddress[4] = "0"
my-channel[1].deviceAddress.macAddress[5] = "0"

The (String, TypedValue) pairs above can be naturally represented using WireRecord properties. Wire Publisher and Db components can naturally handle such flattened representation. An Asset would emit structured data using the above flattened representation. The Asset configuration must contain a single channel for the structured data. There must be some convention for exchanging structured data between the Driver and the Asset.

• for reads:
  • the Driver and its client (e.g. Asset) need to agree that a given channel value type is structured
    • using some channel configuration parameter
  • the Driver must return the structured data somehow
    • add additional records to the List that is passed as argument (list might be immutable)
• for writes:
  • the Asset should supply all (String, TypedValue) pairs to the driver
    • use one ChannelRecord per singleton field, same channel config, driver needs to reconstruct the final value

Web UI components can display data obtained from a read more naturally (e.g. with a tree view) For writing data, web ui must provide dedicated widgets for entering the structured data A schema of the data could enable validations, and in case of writes, enable building forms specific for the data type

• pros:

  • publisher friendly
  • db store friendly
  • wire components do not need to support JSON
  • flattened representation of data can be a natural extension given the current model
  • backwards compatible in most cases (would break at least ASSET-V1)

• cons:

  • need to integrate with ASSET-V1
    • structured data channels unsupported?
    • convert to JSON?
  • some drivers in this release might produce JSON, switching to flattened only repr will introduce breaking change and require heavy refactor
  • manipulating flattened representation is not straightforward
  • adding support to existing web ui requires heavy changes
  • need to define a way for interaction between Driver and its clients (e.g. Asset).
  • publishing flattened data will increase size of indexes on Kapua

4. Allow referencing singleton fields only in Asset configuration

Another possibility could be designing Drivers so that a single channel only addresses singleton fields (leaves) in structured data.

• pros:

  • no need to deal with flattening
  • data can be naturally consumed by existing components
  • avoids emitting unnecessary information

• cons:

  • need "selector" string (e.g. JSONPath)
    • complex to implement
      • driver should optimize aggregating channels with same root
  • how to manage variable sized arrays?
  • need to know data structure beforehand

5. extend typed values including ArrayValue -> List<TypedValue>> and MapValue -> Map>

Another possibility can be extending typed values by adding ArrayValue -> List<TypedValue>> and MapValue -> Map>, introducing structured data as first-class citizen in wires/assets type system. This could provide a clean and uniform way for managing structured data, but would require some changes in all existing components for introducing support for new types, moreover, structured data needs to be flattened or stringified for publishing/storing into a database.

• pros:

  • data format is visible to all components
  • data grouping is explicit
  • data manipulation is easier

• cons:

  • need to integrate with ASSET-V1
  • some drivers in this release might produce JSON, switching will introduce breaking change and require heavy refactor
  • adding support to existing web ui requires heavy changes
  • what to put in value.type field? STRUCTURED?
  • needs to flatten/stringify for publisher and dbstore
  • if data is flattened, issue with index size on Kapua
  • no backwards compatibility, need to adapt existing components
  • difficult to support WireSubscriber

6. use JSON at Driver/Asset level and introduce flattening/unflattening wire component

Another possibility can be using JSON at the Driver/Asset level for representing structured data (like in solution 1 and 2) and introduce some wire components that explicitly perform JSON flattening and unflattening.

• pros:

  • no need to change existing components or APIs (unless we decide to introduce content type tagging as per alternative 2)
  • keeps compatibility with existing drivers that use JSON format
  • keeps compatibility with ASSET-V1 and existing web ui (can be extended later on with better representation for JSON data)

• cons:

  • introduces two representations for structured data
  • introduces some more work for the wire graph designer

Possible extension of alternative 3

a flattened repr of structured data might feel more natural for wires:

• publisher friendly
• db store friendly
• wire components do not need to support JSON
• flattened representation of data can be a natural extension given the current model
• backwards compatible in most cases

open points for solution 3:

• need to integrate with ASSET-V1
• some drivers in this release might produce JSON, switching to flattened only repr will introduce breaking change and require heavy refactor
• manipulating flattened representation is not straightforward and inefficient
• adding support to existing web ui requires heavy changes
• need to define a way for interaction between Driver and its clients (e.g. Asset).

Derived from alternative 3, focused on the flat representation. Keeps JSON around to solve some backwards compatibility issues, even if this implies there will be two formats for representing structured data. If JSON support is removed, can be seen as a possible implementation of 3 (in this case, what is the result of getValue() -> TypedValue)?. Provides some APIs for manipulating flattened representation.

StructuredChannelRecord extends ChannelRecord {

  @Override
  getValueType() -> STRING allows backwards compatibility

  @Override
  getValue() -> StringValue ( (Typed?)Json ) <- backwards compatibility to (current) web ui, ASSET-V1

  @Override
  setValue(StringValue ((Typed?)Json) jsonValue); <- backwards compatibility from (current) web ui, ASSET-V1, WireAsset for writes if received TypedValue is of String type (parse),

  getFlattenedValue() -> Map<String, TypedValue> <- used by WireAsset for emission

  setValue(Map<String, TypedValue>); <- used by WireAsset for writes (if it detects flattened structured data)

}

Builder API, for constructing flattened repr

StructuredValueBuilder {
  build() -> Map<String, TypedValue>
}

ObjectValueBuilder : RecordValueBuilder {
  ObjectValueBuilder(String channelName) // for the root builder only

  putObject(String key) -> ObjectValueBuilder
  putArray(String key) -> ArrayValueBuilder
  putSingleton(String key, TypedValue value)
}

ArrayValueBuilder : RecordValueBuilder {
  ArrayValueBuilder(String channelName) // for the root builder only

  putObject() -> ObjectValueBuilder
  putArray() -> ArrayValueBuilder
  putSingleton(TypedValue value)
}

StructuredValue API, for consuming flattened repr can be implemented by traversing the underlying map in place or by parsing the map to a tree-like structure

StructuredValue {
  StructuredValue(String channelName, Map<String, TypedValue> values)

  asObject() -> Optional<ObjectValue>
  asArray() -> Optional<ArrayValue>
  asSingleton() -> Optional<TypedValue>
}

ObjectValue {
  keySet() -> Set<String>
  get(String) -> Optional<StructuredValue>
}

ArrayValue {
  length() -> int
  get(int) -> Optional<StructuredValue>
}

"open" points from last solution:

• need to integrate with ASSET-V1

  • StructuredChannelRecord uses String as value.type and its value is a Json repr (no changes needed)
  • ASSET-V1 will report structured channels as STRING channels
  • Asset is able to automatically produce StructuredRecord from JSON -> writes supported without changes

• some drivers in this release might produce JSON, switching to flattened only repr will introduce breaking change and require heavy refactor

  • the driver produces json directly without passing for intermediate reprs -> use StructuredChannelRecord.fromJson
  • the driver produces json passing through intermediate structures -> switching to StructuredValueBuilder should be straightforward

• manipulating flattened representation is not straightforward and not efficient

  • using StructuredValueBuilder provides a more easy to use API for constructing flattened repr
  • if the data format is known in advance, the driver can use the flat map directly, otherwise it can traverse the map using the StructuredValue helper

• adding support to existing web ui requires heavy changes

  • current web ui will continue to work in the same way as the ASSET-V1 cloudlet until we take the time to update it

• needs clean way for interaction between driver and asset

  • introduces an explicit API

• publishing flattened data will increase size of indexes on EC?

  • no way for addressing this on the device side

flattened repr:

#"my-channel"[0]."deviceObjectIdentifier" = "device 4194303"
#"my-channel"[0]."deviceAddress"."networkNumber" = 1
#"my-channel"[0]."deviceAddress"."macAddress"[0] = "c0"
#"my-channel"[0]."deviceAddress"."macAddress"[1] = "a8"
#"my-channel"[0]."deviceAddress"."macAddress"[2] = "2"
#"my-channel"[0]."deviceAddress"."macAddress"[3] = "1"
#"my-channel"[0]."deviceAddress"."macAddress"[4] = "ba"
#"my-channel"[0]."deviceAddress"."macAddress"[5] = "c0"
#"my-channel"[1]."deviceObjectIdentifier" = "device 1"
#"my-channel"[1]."deviceAddress"."networkNumber" = 20
#"my-channel"[1]."deviceAddress"."macAddress"[0] = "1"
#"my-channel"[1]."deviceAddress"."macAddress"[1] = "0"
#"my-channel"[1]."deviceAddress"."macAddress"[2] = "0"
#"my-channel"[1]."deviceAddress"."macAddress"[3] = "0"
#"my-channel"[1]."deviceAddress"."macAddress"[4] = "0"
#"my-channel"[1]."deviceAddress"."macAddress"[5] = "0"

• the # prefix avoids conflicts between different channels in emitted wire records, since # is prohibited inside channel names
• representing every field between double quotes allows arbitrary characters in object keys (require " escaping)