Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean Orphans from S3 Bucket #550

Closed
wants to merge 5 commits into from
Closed

Clean Orphans from S3 Bucket #550

wants to merge 5 commits into from

Conversation

noliveleger
Copy link
Contributor

@noliveleger noliveleger commented Apr 17, 2019

New management command: remove_s3_orphans
What it does:

  • List all files from the bucket AWS_STORAGE_BUCKET_NAME
  • Try to find a match in tables logger_attachments, view_exports, kpi_export_tasks based on the filename pattern.

Rules:

  • If filename does not match required patterns, continue
  • If filename does match one of the required patterns and it exists in the DB, continue
  • If filename does match one of the required patters and it doesn't exist in DB, delete

Because of LifeCycle rules on our buckets. The file will be mark as a deleted will be kept for another 30 days in previous versions. We should find a way to tell DjangoStorage to permanently delete the file (or add a tag to it, and set another rule on the bucket to delete those files after 1 day).
Edit: All versions are deleted now.

This management command covers one kpi table. I think we should add kpi_imports too and create another management command in kpi which does the same with its own table.

Closes kobotoolbox/tasks#301

@noliveleger noliveleger requested a review from jnm April 17, 2019 13:25
@noliveleger noliveleger self-assigned this May 22, 2019
@noliveleger
Copy link
Contributor Author

noliveleger commented Oct 2, 2019

Closed in favor of #579

@noliveleger noliveleger closed this Oct 2, 2019
@noliveleger noliveleger deleted the orphans_cleaner branch May 22, 2020 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant