Skip to content

Commit

Permalink
benchmarking: Improve docs on workload (#119)
Browse files Browse the repository at this point in the history
* benchmarking: Improve docs on workload

* benchmarking: Fix typo in docs
  • Loading branch information
ohsayan authored Apr 19, 2024
1 parent bfd5615 commit 5b79924
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 30 deletions.
72 changes: 42 additions & 30 deletions docs/14.benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,45 @@ id: benchmarking
title: Benchmarking
---

Due to Skytable having in-house implementations of almost everything, starting from the protocol, the storage engine and query
language — we have our own custom load testing tool called `sky-bench` that is distributed with the bundle.

## Setting up for benchmarking

**Quick notes**:
- The benchmark tool will create:
- a space called `bench`
- a model called `bench`
- **Be sure that these objects don't already exist!** But don't worry, if they do — the benchmark tool will error. You won't lose any data
- **Once the benchmark is complete, the `bench` space is removed!**
- **Do not use the `bench` space during the benchmark**: You might lose data

**The benchmark tool will:**
- Run a total of 4,000,000 queries (don't worry, they run pretty fast!):
- Run 1,000,000 `INSERT`s
- Run 1,000,000 `SELECT`s
- Run 1,000,000 `UPDATE`s
- Run 1,000,000 `DELETE`s
- The model used has the declaration `(un: string, pw: uint8)`
- The `SELECT` will select and return all fields
- The `UPDATE` will increment the value of `pw` like this `pw += 1`
- The `DELETE` removes the entire collection
- The default primary key size is 7 bytes. All generated keys will be padded with zeros from to 0 to 999,999 like this: `0000000` or `0999999`

## Off to the races

1. Start up the database server
2. Run `./sky-bench --password <your root password>`. We need your `root` password because only the root account can create, alter and drop models and the benchmark tool needs to run these queries
3. Wait for it to happen. You may not believe your eyes, so we recommend that you keep your eyes hydrated 🔥🚀✨💣
All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the `sky-bench` benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean.

The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors.

:::info
The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed.
:::

## Benchmark workload

The workload that the engine currently uses (as of v0.8.1) is the following:
- A model `bench.bench` is created with a primary key (of type `string` and with a column of type `uint8`)
- Multiple clients are created (simulating "application servers")
- Queries are run against **unique rows**. This means that unlike `redis-benchmark` **every query group touches a different row, as it would generally do in the real-world**
- The following queries (collectively a "query group") are run for each unique row:
- The row is first created with an `INSERT`
- All columns of the row are returned with a `SELECT`
- The integer column is incremented with an `UPDATE`
- The row is finally removed with a `DELETE`
- **By default, 1,000,000 rows are created and manipulated**
- **The time taken for each row to be sent, read back and decoded into a readable form is recorded** (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools
- In total 4,000,000 queries are run (by default)

:::caution
The benchmark tool will create a space `bench` and a model `bench` and will completely remove the space and associated data once the benchmark is complete. **Do not use this space!**
:::

## Running the benchmark

Now that you know how the benchmark engine works, go ahead and benchmark for yourself.

1. Run:
```sh
sky-bench --password <root_password>
```
**Note**: You can ignore the `--password` argument if you have already set it using the `SKYDB_PASSWORD` environment variable
2. The benchmark engine will run the full workload (as described earlier)
3. Witness 4,000,000 queries being executed in real-time. Good luck and enjoy the results!

:::tip
You can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting.
:::
6 changes: 6 additions & 0 deletions docs/2.installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ Getting started with Skytable involves choosing a mode of installation, download
- [**A Docker image**](#docker-image): We generally recommend using a Docker image for experimenting with Skytable on your local system during development and you want to keep your local system *clean*. If you want to use a Docker image for deployment, you're always free to do so!
> **Note:** You might experience slightly degraded performance from the storage engine due to Docker engine's abstractions.
:::tip
All client tools (such as `skysh` and `sky-bench`) *can* use the `SKYDB_PASSWORD` variable for authentication. If you're using Skytable in a testing environment and frequently need to use `skysh`, you may consider setting this variable to your password to avoid having to pass the `--password` argument every time.

However, we strongly recommend **not** using it outside testing environments.
:::

## Native binaries

To use native binaries you need to download a bundle which is simply a ZIP file with all the necessary binaries that you'll ever need to develop on and deploy Skytable.
Expand Down

0 comments on commit 5b79924

Please sign in to comment.