Example of configuration setup for a single bucket?

[beddari]

Hi there, we at Safespring were really intrigued when we saw this project released, it seems to solve a lot of the same use cases we also work with in our day-to-day operations. We have actually started to develop a very similar tool internally but you seem to have progressed further than we have, so far.

New to the project, what would be the easiest way to get a working replication setup for a single bucket s3->s3, in a standalone configuration?

Docker compose as the basis for the setup? Is there anything to go by that you have?

Looking forward to possibly collaborate!

[arttor]

Hi @beddari, thank you for your interest in our project. I will be happy to assist you and improve our documentation along the way!

The easiest setup for a try-out is the standalone binary. It contains all required components, including embedded Redis, making it easy to set up and install. However, it is not suitable for production because with the standalone option, you cannot have multiple workers or persist replication progress between restarts. In our organization, we use the Helm chart to run Chorus on K8s.

There is a docs for standalone setup but here is a short version (feel free to ask questions):

• Download standalone binary from docs page
• Run it with the print-config command to get a sample config and save into yaml file:

  chorus print-config > my-conf.yaml

• replace storage urls and user credentials in the config with your source and destination S3. Here is config docs but for a simple setup you just need to edit storage.storages.one url and user credentials as your source s3 and storage.storages.two as a destination.
• now run standalone again but with the edited config: chorus -config ./my-config.yaml
• Download the CLI from here (or checkuot repo and build it from soruce with cd tools/chorctl && go build .) and create bucket replication with the command:

  chorctl repl add -b <bucket name> -u <s3 user name from chorus config> -f one -t two

  where -f one -t two or --from=one --to=two are source and destination s3 aliases from the config.
• Alternatively, open the standalone web UI at http://127.0.0.1:9672/, go to replication tab, and create replication in wizard instead of CLI.
• you can monitor replication progress with CLI using chorctl dash command.

One of my next tasks is to create an example and docs for Docker-compose, but it is not done yet.

[beddari]

Thanks, this should be good to get us started, one of us will surely be in touch if we get stuck 👍🏼 The part that we couldn't figure out quickly ourselves was the cli tool example.

[arttor]

one more thing that may be useful: if you are familiar with Go, you can run Chorus e2e tests against your s3 storages with debug, breakpoints, and stuff from your IDE. To do so, you need:

1. Install Go
2. go to e2e migration test test/migration/
3. edit proxy and worker s3 storage config in test init code test/migration/init_test.go. Currently, test starts 3 fake s3 storages main, f1, and f2 and puts url and credentials to config, but you can remove fake s3 storages and just add your credentials into config Go struct proxyConf.Storage.Storages["main"], proxyConf.Storage.Storages["f1"], etc...
4. Then you can create a new test, or edit the existing one. The test is just a Go script that puts some data to src s3 with s3 go client and communicates with chorus directly over gRPC (instead of cli) to run replication and verify the results.

This is a little bit low-level and more relevant for developers but if you are fluent with Go, it might be easier to build all from source to run on your machine and use gRPC instead of CLI .