🚀 Feature: Add support for Cloudflare R2 Storage

[stnguyen90]

🔖 Feature description

Cloudflare R2 Storage is an S3-compatible distributed object storage with generous pricing and a free tier. It would be great if Appwrite could make use of this storage provider.

🎤 Pitch

A storage provider with a free tier would make switching to and testing a cloud storage provider easier.

Related issue:

• 🚀 Feature: Looking to add support for cloudflare r2 utopia-php/storage#52

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[gewenyu99]

If it's S3 compatible, can it be supported rn by using the configurable ENV Vars? (I'm genuinely curious as a sanity check).

[stnguyen90]

If it's S3 compatible, can it be supported rn by using the configurable ENV Vars? (I'm genuinely curious as a sanity check).

The way utopia storage works right now, a new Device needs to be implemented to change some parameters:

1. Backblaze
2. DOSpaces
3. Linode
4. Wasabi

The region constants for each provider are also a little nice to make sure you're not using the wrong region.

Then, the Appwrite code needs to be updated to use the right device based on the _APP_STORAGE_DEVICE env var.

That said, there is an issue for creating a generic S3 compatible device. And, I've seen a PR for a generic S3, but it doesn't look like it implements 1 S3 device that you can use for all S3 compatible providers.

[gewenyu99]

Cool, I always learn a lot from talking to you :P

[Meldiron]

@gewenyu99 I think we had PR to do that at some point before 0.14. Not sure why it was not used.

[runesign]

I have the same wish!

[doublevcodes]

Hello, I'd like to have a go at implementing this!

[stnguyen90]

@doublevcodes, thanks for your interest! 🙏 Happy hacking!

[doublevcodes]

So as I understood at it, I need to create 2 PRs? One to utopia-php/storage and one to this repo

[Meldiron]

@doublevcodes That is correct 😇 You can take a look at this Hacktoberfest issue that we created, it's on the same topic but different adapter. You can use it's description to learn more: #3991

[doublevcodes]

Hi, I'm a bit confused on how to implement this because of how Cloudflare R2 bucket URLs are structured.

So far I have something like this:

  public function __construct(string $root, string $accountId, string $accessKey, string $secretKey, string $bucket, string $region = self::AUTO, string $acl = self::ACL_PRIVATE)
  {
      parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
      $this->headers['host'] = $accountId . '.' . 'r2' . '.' . 'cloudflarestorage' . '.' . 'com' . '/'. $bucket;
  }

However I'm sure that this wouldn't form a valid Host header due to the slash.
How would I go about this?

[stnguyen90]

@doublevcodes, oye that's unfortunate Cloudflare does it differently...It looks like the implementation for Cloudflare won't be the same as the other S3 compatible providers like Linode. You'll need to figure out what else needs to be overridden in the base S3 class and implement it in the Cloudflare class.

[doublevcodes]

Ah. Thanks, I'll have a look at what I can do.

[stnguyen90]

@doublevcodes, FYI, virtual-hosted style paths should also be supported in Cloudflare

[pietrodicaprio]

Hi, I'm a bit confused on how to implement this because of how Cloudflare R2 bucket URLs are structured.

So far I have something like this:

  public function __construct(string $root, string $accountId, string $accessKey, string $secretKey, string $bucket, string $region = self::AUTO, string $acl = self::ACL_PRIVATE)
  {
      parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
      $this->headers['host'] = $accountId . '.' . 'r2' . '.' . 'cloudflarestorage' . '.' . 'com' . '/'. $bucket;
  }

However I'm sure that this wouldn't form a valid Host header due to the slash. How would I go about this?

Hello there, I was looking around for R2 support... You are not forced to keep the bucket at the end of the URL. You should use the URL without the bucket name and then use the name as usually done with standard S3 implementation.
Here a python example using the official boto3 library for AWS:
having as URL provided by Cloudflare https://blahblahblahblahblahblah.r2.cloudflarestorage.com/bucket-name-here

s3 = boto3.client('s3',
                      aws_access_key_id=key,
                      aws_secret_access_key=secret,
                      endpoint_url="https://blahblahblahblahblahblah.r2.cloudflarestorage.com")
 with open("some_file_name.png", "rb") as f:
        s3.upload_fileobj(f, "bucket-name-here", "some_file_name.png")

hope this helps a bit

[eldadfux]

Thank you everyone for celebrating Hacktoberfest 22 with us! This issue will now be closed as we're getting ready to celebrate Hacktoberfest 23.

[pietrodicaprio]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

[gewenyu99]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

[pietrodicaprio]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

Well, utopia-php/storage#76 is a thing and @stnguyen90 requested a review 3 weeks ago so it looks like to be almost ready to be merged?

[gewenyu99]

Dear @eldadfux I'm not sure this issue is related to an Hacktoberfest. It is a general request for the product's improvement

Our team opened this task for Hacktoberfest :P @stnguyen90 is a part of our team. AFAIK, he doesn't actually want this feature :D

Well, utopia-php/storage#76 is a thing and @stnguyen90 requested a review 3 weeks ago so it looks like to be almost ready to be merged?

Yep, that will be a generic adaptor, too. So this will be supported in some future version along with all S3 compatible storage services 🤞

[AndrewBucklin]

Could we re-open this? Cloudflare R2 provides much better costs than AWS and it's S3 compliant. It already use it in other projects by simply entering the Cloudflare R2 storage details where the AWS S3 information is usually entered.

[JosefJezek]

please re-open this

[Tushar98644]

@stnguyen90 @gewenyu99 any updates on my pr?

[tessamero]

@Tushar98644 the PR is still awaiting your follow up before we can merge

[docimin]

any update? I think those 3 days are kind of over

[docimin]

@tessamero Is this implemented yet?

[tessamero]

@docimin I closed the issue as the PR did not provide any proof it works as they wrote the code without Cloudflare access. They need to provide proof of tests passing for it to be merged and they are unable to. We are going to close the PR.

[schneidermr]

@tessamero I'd like to reopen this issue because here is the implementation that can solve also this issue: #7172
I've already tested it with R2 and working as expected.

[tessamero]

@schneidermr , reopened the issue, thank you for submitting a PR :)

[stnguyen90]

@pietrodicaprio, thanks for the insight. Does this work too?

https://bucket-name-here.blahblahblahblahblahblah.r2.cloudflarestorage.com/

[docimin]

@stnguyen90 It does not.
It has to be: https://accountID.r2.cloudflarestorage.com/bucketname
or https://accountID.eu.r2.cloudflarestorage.com/bucketname if you use EU juristiction

[stnguyen90]

@stnguyen90 It does not. It has to be: https://accountID.r2.cloudflarestorage.com/bucketname or https://accountID.eu.r2.cloudflarestorage.com/bucketname if you use EU juristiction

I forgot I actually researched this..According to this, it is supported.

[eweren]

Is there still anything that blocks the related PR from being merged? R2 support would be awesome 🚀

[docimin]

Is there still anything that blocks the related PR from being merged? R2 support would be awesome 🚀

Heyo,
This PR also still needs to be merged, but I think the waiting currently is on @schneidermr
utopia-php/storage#103

[schneidermr]

Is there still anything that blocks the related PR from being merged? R2 support would be awesome 🚀

Heyo, This PR also still needs to be merged, but I think the waiting currently is on @schneidermr utopia-php/storage#103

They will let me know if that's the case, but if you could understand the result of the unit test, you would see that there is also some problem with the local storage adapter test case. So my changes absolutely couldn't break any unit test with that extra function param what is optional and not required. I'm happy that you need this changes, but in the future please don't write these type of comments if you aren't senior enough to understand the changes in the code and the result of the unit tests. 🙂

[docimin]

Is there still anything that blocks the related PR from being merged? R2 support would be awesome 🚀

Heyo, This PR also still needs to be merged, but I think the waiting currently is on @schneidermr utopia-php/storage#103

They will let me know if that's the case, but if you could understand the result of the unit test, you would see that there is also some problem with the local storage adapter test case. So my changes absolutely couldn't break any unit test with that extra function param what is optional and not required. I'm happy that you need this changes, but in the future please don't write these type of comments if you aren't senior enough to understand the changes in the code and the result of the unit tests. 🙂

No need to be passive aggressive 😕. I'm just trying to help. The answer is based on the last comment on there.

I also wrote without assurance, which you most likely didn't read based on your answer. Just because you got mentioned doesn't automatically mean everyone is waiting for you and it's your fault. 😊

[adityaoberai]

Closing this issue. Thank you so much for participating in Hacktoberfest 2023! We can't wait to welcome you all during HF 2024! Stay tuned for a lot more amazing issues from the Appwrite team!

[pietrodicaprio]

Isn't utopia-php/storage#103 solving utopia-php/storage#28 that allows any S3-like storage solving also the mentioned utopia-php/storage#76 ?
With that merge it should be possible to use R2

[docimin]

Isn't utopia-php/storage#103 solving utopia-php/storage#28 that allows any S3-like storage solving also the mentioned utopia-php/storage#76 ? With that merge it should be possible to use R2

76 and 28 will not be merged, as 103 already does this I think, however due to test issues in 103, utopia-php/storage#106 got created. So I would say keep checking 106.

[pietrodicaprio]

Isn't utopia-php/storage#103 solving utopia-php/storage#28 that allows any S3-like storage solving also the mentioned utopia-php/storage#76 ? With that merge it should be possible to use R2

76 and 28 will not be merged, as 103 already does this I think, however due to test issues in 103, utopia-php/storage#106 got created. So I would say keep checking 106.

Correct, keep in mind that 106 is needed due to a repository configuration issue. I expect to be 103 the mentioned issue as reference (e.g. in release notes)

[eweren]

Hey there 👋
Any updates? The related PR utopia-php/storage#106 seems to have been merged.

[zengxs]

Hello,

It seems that the utopia-php/storage now supports custom S3 endpoints, and the feature has been released. To enable support for Cloudflare R2 Storage in Appwrite, I believe we just need to add a configuration option for custom S3 endpoints.

There is a related PR (#7172) that addresses this by adding support for custom endpoint URLs for S3. Could this PR be reviewed and merged soon to enable this functionality?

Thank you for your time and attention!