This repository has been archived by the owner on Sep 26, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6
Add all collections and items for public cmip6 data #205
Closed
Closed
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
b923736
Rewrite ingest and discovery process for large datasets.
moradology 1a80ed0
Reduce concurrency in step functions to avoid lambda limits
moradology 8854ce9
Organize cmip6 ingests, throw with useful error when no files found
moradology ab00adb
Print insertion response after every new job
moradology 55cfa7f
Retry all lambdas to avoid transient TooManyRequest errors
moradology f02e625
Propagate through event parameters to file objects so date arguments …
moradology b0ec0ac
Lint
moradology ace5f5a
Add 'day' to event validation
moradology File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
41 changes: 41 additions & 0 deletions
41
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-crossover.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-crossover", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 2-degree average temperature crossover predictions", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "2015-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "Predicted year at which the average daily temperature has risen by 2-degrees", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": false, | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } | ||
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-hurs.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-hurs", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected near-surface relative humidity", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected near-surface relative humidity", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-huss.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-huss", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected near-surface specific humidity", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected near-surface specific humidity", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-pr.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-pr", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected precipitation", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected precipitation", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-rlds.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-rlds", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected surface downwelling longwave radiation", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected surface downwelling longwave radiation", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-rsds.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-rdsds", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected surface downwelling shortwave radiation", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected surface downwelling shortwave radiation", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-sfcwind.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-sfcwind", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected near-surface wind speed", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected near-surface wind speed", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-tas.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-tas", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected daily near-surface air temperature", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected daily near-surface air temperature", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
42 changes: 42 additions & 0 deletions
42
data/collections/nex-gddp-cmip6/nex-gddp-cmip6-daily/nex-gddp-cmip6-daily-tasmax.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "id": "nex-gddp-cmip6-daily-tasmax", | ||
| "type": "Collection", | ||
| "links": [], | ||
| "title": "CMIP6 historical and projected daily maximum near-surface air temperature", | ||
| "extent": { | ||
| "spatial": { | ||
| "bbox": [ | ||
| [ | ||
| -180, | ||
| -90, | ||
| 180, | ||
| 90 | ||
| ] | ||
| ] | ||
| }, | ||
| "temporal": { | ||
| "interval": [ | ||
| [ | ||
| "1950-01-01T00:00:00Z", | ||
| "2101-01-01T00:00:00Z" | ||
| ] | ||
| ] | ||
| } | ||
| }, | ||
| "license": "MIT", | ||
| "description": "CMIP6 historical and projected daily maximum near-surface air temperature", | ||
| "stac_version": "1.0.0", | ||
| "dashboard:is_periodic": true, | ||
| "dashboard:time_density": "day", | ||
| "item_assets": { | ||
| "cog_default": { | ||
| "type": "image/tiff; application=geotiff; profile=cloud-optimized", | ||
| "roles": [ | ||
| "data", | ||
| "layer" | ||
| ], | ||
| "title": "Default COG Layer", | ||
| "description": "Cloud optimized default layer to display on map" | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this simply not be here? When do we want to add
"dashboard:is_periodic": false? @anayeayeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we do want
dashboard:is_periodicbut it should betrue+ we need to adddashboard:time_density day. This will tell the summarizer to just return the first and last date and the dashboard UI will create a daily timestep picker. Not sure how nice 30 years of daily data will feel in the UI experience, though, if that's what you meanThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about this - the rasters in question have values which are years rather than, themselves, being indexed by years. In a sense, these two rasters are temporal indices. The rasters themselves are definitely not periodic in the normal sense
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking of COGs set one year apart. Now that I am looking at the crossing year data I see that there is only one COG for each scenario. Not a time series. Sorry about that.
Given that, I think the best thing to do is keep
dashboard:is_periodicfalseand adddashboard:time_densitynull.This will let the dashboard apply the default mosaic xyz logic here but we are definitely pushing the edges of the default cog dashboard display logic here.