FEAT: OPMI Box Loader #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updated gitignore to 1) ignore .DS_Store files generated by MacOS, 2) ignore the token file, and 3) no longer ignore jupyter notebook
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds a new ETL job to load files from the OPMI BOX account into the AWS OMPI Research Server RDS.
Job steps:
1. confirm structure of BOX_IMPORT_FOLDER_ID
1.a BOX_IMPORT_FOLDER_ID should contain folder named "aws_loaded" (where sucessfully processed files will be moved to)
1.b BOS_IMPORT_FOLDER_ID should contain folder named "aws_error" (where files with load errors will be moved to)
1.c if either expected folder does not exist, create them
2. Iterate through all .csv files in BOX_IMPORT_FOLDER_ID
2.a Files should follow specific naming convention targetSchema_targetTable.csv
2.b First row of file should be column names that match targetTable
3. Download file to local disk from Box
4. Load file into RDS targetSchema.targetTable via csv load function
4.a On successful load, move file to "aws_loaded" folder
4.b If load error occurs, move file to "aws_error" folder