TidesDB v0.4.0b

[guycipher]

TidesDB v0.4.0b pre release.

Features

• ACID transactions are atomic, consistent, isolated, and durable. Transactions are tied to their respective column family.
• Concurrent multiple threads can read and write to the storage engine. Column families use a read-write lock thus allowing multiple readers and a single writer per column family. Transactions on commit block other threads from reading or writing to the column family until the transaction is completed. A transaction is thread safe.
• Column Families store data in separate key-value stores. Each column family has their own memtable and sstables.
• Atomic Transactions commit or rollback multiple operations atomically. When a transaction fails, it rolls back all operations.
• Cursor iterate over key-value pairs forward and backward.
• WAL write-ahead logging for durability. Column families replay WAL on startup. This reconstructs memtable if the column family did not reach threshold prior to shutdown.
• Multithreaded Compaction manual multi-threaded paired and merged compaction of sstables. When run for example 10 sstables compacts into 5 as their paired and merged. Each thread is responsible for one pair - you can set the number of threads to use for compaction.
• Bloom Filters reduce disk reads by reading initial blocks of sstables to check key existence.
• Compression compression is achieved with Snappy, or LZ4, or ZSTD. SStable entries can be compressed as well as WAL entries.
• TTL time-to-live for key-value pairs.
• Configurable column families are configurable with memtable flush threshold, data structure, if skip list max level, if skip list probability, compression, and bloom filters.
• Error Handling API functions return an error code and message.
• Easy API simple and easy to use api.
• Multiple Memtable Data Structures memtable can be a skip list or hash table.
• Multiplatform Linux, MacOS, and Windows support.

Building

Using cmake to build the shared library.

cmake -S . -B build
cmake --build build
cmake --install build

Requirements

You need cmake and a C compiler that supports C.
You also require the snappy, lz4, and zstd libraries.

Linux

sudo apt install libzstd-dev
sudo apt install liblz4-dev
sudo apt install libsnappy-dev

MacOS

brew install zstd
brew install lz4
brew install snappy

Windows

Windows using vcpkg

vcpkg install zstd
vcpkg install lz4
vcpkg install snappy

Dependencies

• Snappy
• LZ4
• Zstandard

Include

#include <tidesdb.h>

Usage

Each database method returns a tidesdb_err_t* which returns an error code and message. If no error, TidesDB returns NULL.

Example of error structure

typedef struct
{
    int code;
    char *message;
} tidesdb_err_t;

Opening a database

To open a database you pass the path to the database directory and a pointer to the database.

tidesdb_t *tdb = NULL;
tidesdb_err_t *e = tidesdb_open("your_tdb_directory", &tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* close the database */
e = tidesdb_close(tdb);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Creating a column family

In order to store data in TidesDB you need a column family. This is by design.

You pass

• the database you want to create the column family in. Must be open
• the name of the column family
• memtable flush threshold in bytes. Example below is 128MB
• skip list max level. Example below is 12 ( only if using TDB_MEMTABLE_SKIP_LIST ) pass 0 if using TDB_MEMTABLE_HASH_TABLE
• skip list probability. Example below is 0.24 ( only if using TDB_MEMTABLE_SKIP_LIST ) pass 0.0 if using TDB_MEMTABLE_HASH_TABLE
• whether column family sstable data is compressed
• the compression algorithm to use [TDB_NO_COMPRESSION, TDB_COMPRESS_SNAPPY, TDB_COMPRESS_LZ4, TDB_COMPRESS_ZSTD]
• whether to use bloom filters
• what data structure to use for the memtable [TDB_MEMTABLE_SKIP_LIST, TDB_MEMTABLE_HASH_TABLE]

/* create a column family with no compression and no bloom filters (slower reads) */
tidesdb_err_t *e = tidesdb_create_column_family(tdb, "your_column_family", (1024 * 1024) * 128, 12, 0.24f, false, TDB_NO_COMPRESSION, false, TDB_MEMTABLE_SKIP_LIST);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Using Snappy compression and bloom filters for column family sstables

/* create a column family with compression and bloom filter (the bloom filter provides fast read speed) */
tidesdb_err_t *e = tidesdb_create_column_family(tdb, "your_column_family", (1024 * 1024) * 128, 12, 0.24f, true, TDB_COMPRESS_SNAPPY, true, TDB_MEMTABLE_SKIP_LIST);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Dropping a column family

/* drop a column family */
tidesdb_err_t *e = tidesdb_drop_column_family(tdb, "your_column_family");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Listing column families

/* list column families
 * returns a char* of column family names separated by a newline
*/
char *column_families = tidesdb_list_column_families(tdb);
if (column_families == NULL)
{
    /* handle error */
}

/* in this example we just print and free the column families */
printf("%s\n", column_families);
free(column_families);

Putting a key-value pair

You pass

• the database you want to put the key-value pair in. Must be open
• the column family name you want to put the key-value pair in
• the key
• the key size
• the value
• the value size
• when the key-value pair should expire. If -1 then it never expires.

/* put a key-value pair */
uint8_t key[] = "key";
uint8_t value[] = "value";

tidesdb_err_t *e = tidesdb_put(tdb, "your_column_family", key, sizeof(key), value, sizeof(value), -1);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Putting a key-value pair with TTL

/* put a key-value pair with TTL */
uint8_t key[] = "key";
uint8_t value[] = "value";

time_t ttl = time(NULL) + 10; /* 10 seconds */
tidesdb_err_t *e  = tidesdb_put(tdb, "your_column_family", key, sizeof(key), value, sizeof(value), ttl);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Getting a key-value pair

You pass

• the database you want to get the key-value pair from. Must be open
• the column family name
• the key
• the key size
• a pointer to the value
• a pointer to the value size

size_t value_len = 0;
uint8_t *value_out = NULL;
uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_get(tdb, "your_column_family", key, sizeof(key), &value_out, &value_len);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Deleting a key-value pair

You pass

• the database you want to delete the key-value pair from. Must be open
• the column family name
• the key
• the key size

uint8_t key[] = "key";

tidesdb_err_t *e = tidesdb_delete(tdb, "your_column_family", key, sizeof(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Transactions

You can perform a series of operations atomically. This will block other threads from reading or writing to the column family until the transaction is committed or rolled back.

You begin a transaction by calling tidesdb_txn_begin.

You pass

• the database you want to perform the operations in. Must be open
• the transaction pointer
• the column family name you want to perform the operations in

tidesdb_txn_t *transaction;
tidesdb_err_t *e = tidesdb_txn_begin(tdb, &transaction, "your_column_family");
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

Now we can add operations to the transaction.

const uint8_t key[] = "example_key";
const uint8_t value[] = "example_value";
tidesdb_err_t *e = tidesdb_txn_put(transaction, key, sizeof(key), value, sizeof(value), -1); /* you can pass a ttl, similar to put */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* you can add delete operations as well */
e = tidesdb_txn_delete(transaction, key, sizeof(key));
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* now we commit */
e = tidesdb_txn_commit(transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* before you free, you can rollback */
tidesdb_err_t *e = tidesdb_txn_rollback(transaction);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* free the transaction */
tidesdb_txn_free(transaction);

Cursors

You can iterate over key-value pairs in a column family.

tidesdb_cursor_t *c;
tidesdb_err_t *e = tidesdb_cursor_init(tdb, "your_column_family", &c);
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
    return;
}

uint8_t *retrieved_key = NULL;
size_t key_size;
uint8_t *retrieved_value = NULL;
size_t value_size;

/* iterate forward */
while ((e = tidesdb_cursor_next(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &retrieved_key, &key_size, &retrieved_value, &value_size);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use retrieved_key and retrieved_value
     * .. */

    /* free the key and value */
    free(retrieved_key);
    free(retrieved_value);
}

if (e != NULL && e->code != TIDESDB_ERR_AT_END_OF_CURSOR)
{
    /* handle error */
    tidesdb_err_free(e);
}

/* iterate backward */
while ((e = tidesdb_cursor_prev(c)) == NULL)
{
    e = tidesdb_cursor_get(c, &retrieved_key, &key_size, &retrieved_value, &value_size);
    if (e != NULL)
    {
        /* handle error */
        tidesdb_err_free(e);
        break;
    }

    /* use retrieved_key and retrieved_value
     * .. */

    /* free the key and value */
    free(retrieved_key);
    free(retrieved_value);
}

if (e != NULL && e->code != TIDESDB_ERR_AT_START_OF_CURSOR)
{
    /* handle error */
    tidesdb_err_free(e);
}

tidesdb_cursor_free(c);

Compaction

You can manually compact sstables. This method pairs and merges column family sstables.
Say you have 100, after compaction you will have 50; Always half the amount you had prior. You can set the number of threads to use for compaction.

tidesdb_err_t *e = tidesdb_compact_sstables(tdb, "your_column_family", 10); /* use 10 threads */
if (e != NULL)
{
    /* handle error */
    tidesdb_err_free(e);
}

What's Changed

• multi-os compatibility header. The compat.h is is designed to provid… by @guycipher in multi-os compatibility header. The compat.h is is designed to provid… #187
• update cmake and read me by @guycipher in update cmake and read me #188
• add memtable_ds to tidesdb_create_column_family arguments by @toge in add memtable_ds to tidesdb_create_column_family arguments #190
• link dependent libraries to tidesdb library by @toge in link dependent libraries to tidesdb library #191
• Correct all build errors on linux(gcc version 12.3.0 (Ubuntu 12.3.0_1… by @guycipher in Correct all build errors on linux(gcc version 12.3.0 (Ubuntu 12.3.0_1… #192
• Add apple compatibility changes to CMakeLists and hash_table by @hp77-creator in Add apple compatibility changes to CMakeLists and hash_table #193
• cmake addition for hash table to be built with library as implementat… by @guycipher in cmake addition for hash table to be built with library as implementat… #194
• correct column family serialization pointer advancement. Was from co… by @guycipher in correct column family serialization pointer advancement. Was from co… #195
• minor cursor locking corrections by @guycipher in minor cursor locking corrections #196
• Further vcpkg implementation for windows support by @guycipher in Further vcpkg implementation for windows support #197
• Further compat.h for windows support by @guycipher in Further compat.h for windows support #198
• • Switch over to using compact.h in tides.h so we can now achieve cro… by @guycipher in - Switch over to using compact.h in tides.h so we can now achieve cro… #199
• use compact.h within block_manager and minor correction in tidesdb.h … by @guycipher in use compact.h within block_manager and minor correction in tidesdb.h … #200
• minor read me corrections by @guycipher in minor read me corrections #201
• hash table on collision use quadratic probing. add benchmark write a… by @guycipher in hash table on collision use quadratic probing. add benchmark write a… #202
• Minor test corrections to print statements by @guycipher in Minor test corrections to print statements #203

New Contributors

• @hp77-creator made their first contribution in Add apple compatibility changes to CMakeLists and hash_table #193

Full Changelog: v0.3.0b...v0.4.0b

---

This discussion was created from the release v0.4.0b.