Custom YAML data types

Custom YAML data types

Sometimes you need to serialize complex data types such as classes. To do this you could use plain nodes such as mappings with classes' fields. YAML also supports custom types with identifiers called tags. That is the topic of this tutorial.

Each YAML node has a tag specifying its type. For instance: strings use the tag tag:yaml.org,2002:str. Tags of most default types are implicitly resolved during parsing - you don't need to specify tag for each float, integer, etc. D:YAML can also implicitly resolve custom tags, as we will show later.

Constructor

D:YAML uses the Constructor class to process each node to hold data type corresponding to its tag. Constructor stores functions to process each supported tag. These are supplied by the user using the addConstructorXXX() methods, where XXX is Scalar, Sequence or Mapping. Constructor is then passed to Loader, which parses YAML input.

Structs and classes must implement the opCmp() operator for YAML support. This is used for duplicate detection in mappings, sorting and equality comparisons of nodes. The signature of the operator that must be implemented is const int opCmp(ref const MyStruct s) for structs where MyStruct is the struct type, and int opCmp(Object o) for classes. Note that the class opCmp() should not alter the compared values - it is not const for compatibility reasons.

We will implement support for an RGB color type. It is implemented as the following struct:

struct Color
{
    ubyte red;
    ubyte green;
    ubyte blue;

    const int opCmp(ref const Color c)
    {
        if(red   != c.red)  {return red   - c.red;}
        if(green != c.green){return green - c.green;}
        if(blue  != c.blue) {return blue  - c.blue;}
        return 0;
    }
}

First, we need a function to construct our data type. The function will take a reference to Node to construct from. The node is guaranteed to contain either a string, an array of Node or of Node.Pair, depending on whether we're constructing our value from a scalar, sequence, or mapping, respectively. If this function throws any exception, D:YAML handles it and adds its message to a YAMLException that will be thrown when loading the file.

In this tutorial, we have functions to construct a color from a scalar, using CSS-like format, RRGGBB, or from a mapping, where we use the following format: {r:RRR, g:GGG, b:BBB} . Code of these functions:

Color constructColorScalar(ref Node node)
{
    string value = node.as!string;

    if(value.length != 6)
    {
        throw new Exception("Invalid color: " ~ value);
    }
    //We don't need to check for uppercase chars this way.
    value = value.toLower();

    //Get value of a hex digit.
    uint hex(char c)
    {
        import std.ascii;
        if(!std.ascii.isHexDigit(c))
        {
            throw new Exception("Invalid color: " ~ value);
        }

        if(std.ascii.isDigit(c))
        {
            return c - '0';
        }
        return c - 'a' + 10;
    }

    Color result;
    result.red   = cast(ubyte)(16 * hex(value[0]) + hex(value[1]));
    result.green = cast(ubyte)(16 * hex(value[2]) + hex(value[3]));
    result.blue  = cast(ubyte)(16 * hex(value[4]) + hex(value[5]));

    return result;
}

Color constructColorMapping(ref Node node)
{
    ubyte r,g,b;

    //Might throw if a value is missing is not an integer, or is out of range.
    //If this happens, D:YAML will handle the exception and use its message
    //in a YAMLException thrown when loading.
    r = node["r"].as!ubyte;
    g = node["g"].as!ubyte;
    b = node["b"].as!ubyte;

    return Color(cast(ubyte)r, cast(ubyte)g, cast(ubyte)b);
}

Next, we need some YAML data using our new tag. Create a file called input.yaml with the following contents:

scalar-red: !color FF0000
scalar-orange: !color FFFF00
mapping-red: !color-mapping {r: 255, g: 0, b: 0}
mapping-orange:
    !color-mapping
    r: 255
    g: 255
    b: 0

You can see that we're using tag !color for scalar colors, and !color-mapping for colors expressed as mappings.

Finally, the code to put it all together:

void main()
{
    auto red    = Color(255, 0, 0);
    auto orange = Color(255, 255, 0);

    try
    {
        auto constructor = new Constructor;
        //both functions handle the same tag, but one handles scalar, one mapping.
        constructor.addConstructorScalar("!color", &constructColorScalar);
        constructor.addConstructorMapping("!color-mapping", &constructColorMapping);

        auto loader = Loader("input.yaml");
        loader.constructor = constructor;

        auto root = loader.load();

        if(root["scalar-red"].as!Color == red &&
           root["mapping-red"].as!Color == red &&
           root["scalar-orange"].as!Color == orange &&
           root["mapping-orange"].as!Color == orange)
        {
            writeln("SUCCESS");
            return;
        }
    }
    catch(YAMLException e)
    {
        writeln(e.msg);
    }

    writeln("FAILURE");
}

First, we create a Constructor and pass functions to handle the !color and !color-mapping tag. We construct a Loader and pass the Constructor to it. We then load the YAML document, and finally, read the colors to test if they were loaded as expected.

You can find the source code for what we've done so far in the examples/constructor directory in the D:YAML package.

Resolver

Specifying tag for every color can be tedious. D:YAML can implicitly resolve scalar tags using regular expressions. This is how default types are resolved. We will use the Resolver class to add implicit tag resolution for the Color data type (in its scalar form).

We use the addImplicitResolver() method of Resolver, passing the tag, regular expression the scalar must match to resolve to this tag, and a string of possible starting characters of the scalar. Then we pass the Resolver to Loader.

Note that resolvers added first override ones added later. If no resolver matches a scalar, YAML string tag is used. Therefore our custom values must not be resolvable as any non-string YAML data type.

Add this to your code to add implicit resolution of !color:

//code from the previous example...

auto resolver = new Resolver;
import std.regex;
resolver.addImplicitResolver("!color", std.regex.regex("[0-9a-fA-F]{6}"),
                             "0123456789abcdefABCDEF");

auto loader = Loader("input.yaml");

loader.constructor = constructor;
loader.resolver = resolver;

//code from the previous example...

Now, change contents of input.yaml to this:

scalar-red: FF0000
scalar-orange: FFFF00
mapping-red: !color-mapping {r: 255, g: 0, b: 0}
mapping-orange:
    !color-mapping
    r: 255
    g: 255
    b: 0

We no longer need to specify the tag for scalar color values. Compile and test the example. If everything went as expected, it should report success.

You can find the complete code in the examples/resolver directory in the D:YAML package.

Representer

Now that you can load custom data types, it might be good to know how to dump them. D:YAML uses the Representer class for this purpose.

Representer processes YAML nodes into plain mapping, sequence or scalar nodes ready for output. Just like with Constructor, this is done by user specified functions. These functions take references to a node to process and to the Representer, and return the processed node.

Representer functions can be added with the addRepresenter() method. The Representer is then passed to Dumper, which dumps YAML documents. Only one function per type can be specified. This is asserted in addRepresenter() preconditions. Default YAML types already have representer functions specified, but you can disable them by constructing Representer with the useDefaultRepresenters parameter set to false.

By default, tags are explicitly output for all non-default types. To make dumped tags implicit, you can pass a Resolver that will resolve them implicitly. Of course, you will need to use an identical Resolver when loading the output.

With the following code, we will add support for dumping the our Color type:

Node representColor(ref Node node, Representer representer)
{
    //The node is guaranteed to be Color as we add representer for Color.
    Color color = node.as!Color;

    static immutable hex = "0123456789ABCDEF";

    //Using the color format from the Constructor example.
    string scalar;
    foreach(channel; [color.red, color.green, color.blue])
    {
        scalar ~= hex[channel / 16];
        scalar ~= hex[channel % 16];
    }

    //Representing as a scalar, with custom tag to specify this data type.
    return representer.representScalar("!color", scalar);
}

First we get the Color from the node. Then we convert it to a string with the CSS-like format we've used before. Finally, we use the representScalar() method of Representer to get a scalar node ready for output. There are corresponding representMapping() and representSequence() methods as well, with examples in the Resolver API documentation.

Since a type can only have one representer function, we don't dump Color both in the scalar and mapping formats we've used before. However, you can decide to dump the node with different formats/tags in the representer function itself. E.g. you could dump the Color as a mapping based on some arbitrary condition, such as the color being white:

void main()
{
    try
    {
        auto representer = new Representer;
        representer.addRepresenter!Color(&representColor);

        auto resolver = new Resolver;
        import std.regex;
        resolver.addImplicitResolver("!color", std.regex.regex("[0-9a-fA-F]{6}"),
                                     "0123456789abcdefABCDEF");

        auto dumper = Dumper("output.yaml");
        dumper.representer = representer;
        dumper.resolver    = resolver;

        auto document = Node([Color(255, 0, 0),
                              Color(0, 255, 0),
                              Color(0, 0, 255)]);

        dumper.dump(document);
    }
    catch(YAMLException e)
    {
        writeln(e.msg);
    }
}

We construct a new Representer, and specify a representer function for the Color (the template argument) type. We also construct a Resolver, same as in the previous section, so the !color tag will be implicit. Of course, identical Resolver would then have to be used when loading the file. You don't need to do this if you want the tag to be explicit.

We construct a Dumper to file output.yaml and pass the Representer and Resolver to it. Then, we create a simple node containing a sequence of colors and finally, we dump it.

Source code for this section can be found in the examples/representer directory of the D:YAML package.