Data serialization

This page explains how to perform data serialization when using the OpenDHT API, to store and retrieve structured data in a robust way from the distributed data store.

Building a dht::Value object

dht::Value is a class representing a value stored on, or retrieved from the DHT. dht::Value is a publicly exposed C++ class including a data payload (the data field), as well as metadata.

From raw data

If you don't need, or already implement serialization, building a dht::Value object from existing binary data is straightforward using the constructor:

Value(const uint8_t* data_ptr, size_t data_size)

Example usage:

std::string the_data {"42 cats"};
dht::Value the_dht_value {(const uint8_t*)the_data.data(), the_data.size()};
// ...
std::string the_data_from_value {the_dht_value.data.begin(), the_dht_value.data.end()};
assert(the_data == the_data_from_value);

From structured data with MsgPack

MessagePack is an efficient binary serialization format, used by OpenDHT internally for message serialization as well as an API facility to serialize structured user data. OpenDHT uses msgpack-c, a simple, header-only C++11 implementation of MessagePack.

See this page for an introduction about how to serialize data with msgpack-c.

Any data structure implementing msgpack-c serialization methods can be used directly to build a dht::Value object. For instance:

struct MyClass {
    std::string m_str;
    std::vector<int> m_vec;

    // implements msgpack-c serialization methods
    MSGPACK_DEFINE(m_str, m_vec);
};

MyClass the_data {"42 cats", {43, 44, 45}};
dht::Value the_dht_value { the_data };

Serializable msgpack objects can be implicitly converted to dht::Values when using OpenDHT methods:

dht::DhtRunner node;

// Implicit conversion to `dht::Value`
node.put("data_key", MyClass{"precious data", {42, 43, 44}} );

// Conversion to MyClass (with get or listen)
dht->get<MyClass>(key, [](MyClass&& myObject) {
    // use or move myObject
    // this callback is called when deserialization succeeds for a value
    return true;
}

Customizing serialization

Here is an abstract of the dht::Value class:

struct Value
{
    Id id;

    // Public key of the signer (if signed).
    crypto::PublicKey owner;

    // Public key ID of the recipient (if decrypted).
    InfoHash recipient;

    // OpenDHT data type.
    ValueType::Id type;

    // Serialized user data payload
    std::vector<uint8_t> data;

    // Custom user-defined data type information
    std::string user_type {};
};

All fields have sensible default values and may be modified by OpenDHT when performing DHT operations. For instance, the ID will be changed to a random value when performing a put operation if it is not set by the user ; the owner is set to the public key when the value is signed, etc.

Making your class serializable by msgpack allows to define the way your structured data will be converted from/to a binary object to be transmitted over the network (the data field).

For most users that will be enough, but some others will want a finer control on how their data is converted from/to a dht::Value object (including metadata). For instance the user may want to use the user_type field, or to preserve the information about the signer of a value or if the value was encrypted.

Serialization from/to dht::Value is defined by the dht::Value::Serializable interface.

(TBD)