From 12c7a17fd8c264516a8f01927dd0f3b34799cc44 Mon Sep 17 00:00:00 2001 From: Marius Date: Fri, 2 Aug 2024 12:16:06 +0200 Subject: [PATCH] docs: Add example usage with Minio and Azurite (#1166) * docs: Add instructions on using Azurite * docs: Add detailed example for MinIO server --- docs/_storage-backends/aws-s3.md | 25 +++++++++++++++++++- docs/_storage-backends/azure-blob-storage.md | 24 +++++++++++++++++++ docs/minio.txt | 3 --- 3 files changed, 48 insertions(+), 4 deletions(-) delete mode 100644 docs/minio.txt diff --git a/docs/_storage-backends/aws-s3.md b/docs/_storage-backends/aws-s3.md index 5494308e8..bd0bd71ab 100644 --- a/docs/_storage-backends/aws-s3.md +++ b/docs/_storage-backends/aws-s3.md @@ -89,6 +89,29 @@ If [metadata](https://tus.io/protocols/resumable-upload#upload-metadata) is asso In addition, the metadata is also stored in the informational object, which can be used to retrieve the original metadata without any characters being replaced. -# Considerations +## Considerations When receiving a `PATCH` request, parts of its body will be temporarily stored on disk before they can be transferred to S3. This is necessary to meet the minimum part size for an S3 multipart upload enforced by S3 and to allow the AWS SDK to calculate a checksum. Once the part has been uploaded to S3, the temporary file will be removed immediately. Therefore, please ensure that the server running this storage backend has enough disk space available to hold these temporary files. + +## Usage with MinIO + +[MinIO](https://min.io/) is an object storage solution that provides an S3-compatible API, making it suitable as a replacement for AWS S3 during development/testing or even in production. To get started, please install the MinIO server according to their documentation. There are different installation methods available (Docker, package managers or direct download), but we will not go further into them. We assume that the `minio` (server) and `mc` (client) commands are installed. First, start MinIO with example credentials: + +```sh +$ MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY minio server ./minio-data +``` + +MinIO is available at `http://localhost:9000` by default and saves the associated data in `./minio-data`. Next, create a bucket called `mybucker` using the MinIO client: + +```sh +$ mc alias set myminio http://localhost:9000 AKIAIOSFODNN7EXAMPLE wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +$ mc mb myminio/mybucket +``` + +MinIO is now set up, and we can start tusd: + +```sh +$ AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY tusd -s3-bucket mybucket -s3-endpoint http://localhost:9000 +``` + +Tusd is then usable at `http://localhost:8080/files/` and saves the uploads to the local MinIO instance. diff --git a/docs/_storage-backends/azure-blob-storage.md b/docs/_storage-backends/azure-blob-storage.md index 95a43bed5..7b1dff1dc 100644 --- a/docs/_storage-backends/azure-blob-storage.md +++ b/docs/_storage-backends/azure-blob-storage.md @@ -58,3 +58,27 @@ By default, the objects are stored at the root of the container. For example the - `abcdef123.info`: Informational object - `abcdef123`: File object + +## Testing with Azurite + +With [Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=npm%2Cblob-storage), a local Azure Blob Storage service can be emulated for testing tusd without using the Azure services in the cloud. To get started, please install Azurite ([installation instructions](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=npm%2Cblob-storage#install-azurite)) and the Azure CLI ([installation instructions](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli#install)). Next, start the local Azurite application: + +```sh +$ azurite --location ./azurite-data +``` + +Azurite provides Blob Storage at `http://127.0.0.1:10000` by default and saves the associated data in `./azurite-data`. For testing, you can use the [well-known storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=npm%2Cblob-storage#well-known-storage-account-and-key) `devstoreaccount1` and its key. + +Next, create a container called `mycontainer` using the Azure CLI: + +```sh +$ az storage container create --name mycontainer --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;" +``` + +Azurite is now set up, and we can start tusd: + +```sh +$ AZURE_STORAGE_ACCOUNT=devstoreaccount1 AZURE_STORAGE_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== ./tusd -azure-storage=mycontainer -azure-endpoint=http://127.0.0.1:10000 +``` + +Tusd is then usable at `http://localhost:8080/files/` and saves the uploads to the local Azurite instance. diff --git a/docs/minio.txt b/docs/minio.txt deleted file mode 100644 index cdb8b9c65..000000000 --- a/docs/minio.txt +++ /dev/null @@ -1,3 +0,0 @@ -MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ./minio server data - -AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY go run cmd/tusd/main.go -s3-bucket tusdtest.transloadit.com -s3-endpoint http://127.0.0.1:9000 -expose-pprof