move STAC service Magpie definition under its component #383

fmigneault · 2023-09-20T15:29:18Z

Overview

Move STAC service definitions under components/stac configuration.

Changes

Non-breaking changes

Move initial stac service Magpie definition under its component configuration.
- Before this change, optional-components/stac-public-access was mandatory since the stac service under
  Magpie was not created otherwise, leading to "service not found" error when requesting the /stac endpoint.
- Ensure that the first stac resource under stac service in Magpie is created by default.
  Without this resource being defined initially, it is very easy to forget creating it, which would not take into
  account the required /stac/stac request path to properly resolve the real endpoints where STAC API is served.
Remove optional-components/stac-public-access dependency under optional-components/all-public-access
to avoid indirectly enforcing components/stac when optional-components/all-public-access is enabled.
Users that desire using optional-components/stac-public-access will have to add it explicitly to the list
of EXTRA_CONF_DIRS.
Rename optional-components/stac-public-access/config/magpie/config.yml.template to
optional-components/stac-public-access/config/magpie/permissions.cfg in order to align
with permissions-specific contents as accomplished with other components.

Breaking changes

n/a

Related Issue / Discussion

Relates to Adding redirect to /stac/stac when using twitcher #382

Nazim-crim

Okay so we can merge this seperately from the fix with the redirect

mishaschwartz

I think we should also completely remove stac-public-access and put those permission changes under all-public-access. (see my concern here: #382 (comment))

See proposed changes from here (https://github.com/bird-house/birdhouse-deploy/compare/stac-permission-fixes)

fmigneault · 2023-09-20T16:40:19Z

@mishaschwartz
I believe stac-public-access should not be removed.
Instead, its dependency in all-public-access should be removed.
This is to avoid stac being enforced by all-public-access, while still providing a quick method to make it public as needed.

crim-jenkins-bot · 2023-09-20T16:42:10Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2076/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-magpie-provider-without-public-permissions
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1319/

NOTEBOOK TEST RESULTS

    
[2023-09-20T15:41:34.472Z] ============================= test session starts ==============================
[2023-09-20T15:41:34.472Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-20T15:41:34.472Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-09-20T15:41:34.472Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-20T15:41:34.472Z] collected 237 items
[2023-09-20T15:41:34.472Z] 
[2023-09-20T15:41:44.412Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-20T15:41:52.603Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-20T15:42:02.320Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-20T15:42:10.427Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .F......       [ 13%]
[2023-09-20T16:15:26.286Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-20T16:15:28.827Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 18%]
[2023-09-20T16:15:56.036Z] ...............                                                          [ 25%]
[2023-09-20T16:16:09.338Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-20T16:16:16.059Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 29%]
[2023-09-20T16:16:31.301Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-20T16:16:32.682Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-09-20T16:16:38.909Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-09-20T16:16:43.583Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-09-20T16:20:15.103Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-20T16:21:30.512Z] .............                                                            [ 49%]
[2023-09-20T16:21:33.254Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-20T16:21:35.574Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 54%]
[2023-09-20T16:21:52.725Z] .................                                                        [ 62%]
[2023-09-20T16:22:00.562Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-20T16:22:01.944Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 64%]
[2023-09-20T16:22:03.293Z] .FFFFFFFF                                                                [ 68%]
[2023-09-20T16:22:13.432Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-20T16:22:22.764Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 75%]
[2023-09-20T16:22:24.183Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-20T16:22:27.241Z] ......                                                                   [ 78%]
[2023-09-20T16:22:33.828Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-20T16:22:48.836Z] .............                                                            [ 84%]
[2023-09-20T16:22:58.832Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-20T16:23:33.556Z] ....s.                                                                   [ 87%]
[2023-09-20T16:23:41.688Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-20T16:23:56.453Z] ...                                                                      [ 89%]
[2023-09-20T16:24:08.682Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2023-09-20T16:24:31.013Z] ......                                                                   [ 92%]
[2023-09-20T16:24:33.859Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-20T16:27:13.452Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-09-20T16:27:13.452Z] 
[2023-09-20T16:27:13.452Z] =================================== FAILURES ===================================

mishaschwartz · 2023-09-20T17:08:22Z

Instead, its dependency in all-public-access should be removed.

Ok I'm fine with that compromise. Can you remove it from the default.conf file then?

birdhouse/optional-components/stac-public-access/config/magpie/config.yml.template

tlvu

LGTM as well, pending the comment from Misha.

birdhouse/components/stac/config/magpie/config.yml.template

crim-jenkins-bot · 2023-09-20T20:13:40Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2077/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-magpie-provider-without-public-permissions
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-133.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1320/

NOTEBOOK TEST RESULTS

    
[2023-09-20T19:29:32.859Z] ============================= test session starts ==============================
[2023-09-20T19:29:32.859Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-20T19:29:32.859Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-20T19:29:32.859Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-20T19:29:32.859Z] collected 237 items
[2023-09-20T19:29:32.859Z] 
[2023-09-20T19:29:40.733Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-20T19:29:49.687Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-20T19:29:59.399Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-20T19:30:06.570Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .F......       [ 13%]
[2023-09-20T19:41:18.256Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-20T19:44:09.787Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 18%]
[2023-09-20T19:44:09.787Z] ...............                                                          [ 25%]
[2023-09-20T19:44:18.272Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-20T19:44:25.180Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 29%]
[2023-09-20T19:44:41.534Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-20T19:44:43.189Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-09-20T19:44:50.859Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-09-20T19:44:55.182Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-09-20T19:49:42.546Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-20T19:51:03.966Z] .............                                                            [ 49%]
[2023-09-20T19:51:07.786Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-20T19:51:10.379Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 54%]
[2023-09-20T19:51:26.963Z] .................                                                        [ 62%]
[2023-09-20T19:51:35.057Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-20T19:51:36.450Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 64%]
[2023-09-20T19:51:37.801Z] .FFFFFFFF                                                                [ 68%]
[2023-09-20T19:51:48.628Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-20T19:51:57.980Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 75%]
[2023-09-20T19:51:59.360Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-20T19:52:02.660Z] ......                                                                   [ 78%]
[2023-09-20T19:52:09.232Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-20T19:52:25.020Z] .............                                                            [ 84%]
[2023-09-20T19:52:37.248Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-20T19:53:23.378Z] ....s.                                                                   [ 87%]
[2023-09-20T19:53:31.644Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-20T19:53:47.380Z] ...                                                                      [ 89%]
[2023-09-20T19:54:02.281Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2023-09-20T19:54:26.232Z] ......                                                                   [ 92%]
[2023-09-20T19:54:29.493Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-20T19:57:19.182Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-09-20T19:57:19.182Z] 
[2023-09-20T19:57:19.182Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-09-20T21:03:16Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2080/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : stac-magpie-provider-without-public-permissions
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1322/

NOTEBOOK TEST RESULTS

    
[2023-09-20T20:19:51.789Z] ============================= test session starts ==============================
[2023-09-20T20:19:51.789Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-20T20:19:51.789Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-20T20:19:51.789Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-20T20:19:51.789Z] collected 237 items
[2023-09-20T20:19:51.789Z] 
[2023-09-20T20:19:59.861Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-20T20:20:11.894Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-20T20:20:24.144Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-20T20:20:30.726Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .F......       [ 13%]
[2023-09-20T20:33:44.485Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-20T20:33:44.485Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 18%]
[2023-09-20T20:36:23.234Z] ...............                                                          [ 25%]
[2023-09-20T20:36:33.054Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-20T20:36:40.325Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 29%]
[2023-09-20T20:36:56.300Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-20T20:36:57.680Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-09-20T20:37:04.228Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-09-20T20:37:08.473Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-09-20T20:39:55.436Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-20T20:41:17.770Z] .............                                                            [ 49%]
[2023-09-20T20:41:21.856Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-20T20:41:24.078Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 54%]
[2023-09-20T20:41:39.902Z] .................                                                        [ 62%]
[2023-09-20T20:41:47.483Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-20T20:41:48.863Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 64%]
[2023-09-20T20:42:03.545Z] .........                                                                [ 68%]
[2023-09-20T20:42:12.861Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-20T20:42:21.828Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 75%]
[2023-09-20T20:42:23.207Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-20T20:42:26.859Z] ......                                                                   [ 78%]
[2023-09-20T20:42:34.998Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-20T20:42:49.463Z] .............                                                            [ 84%]
[2023-09-20T20:43:01.679Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-20T20:43:38.306Z] ....s.                                                                   [ 87%]
[2023-09-20T20:43:46.437Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-20T20:43:59.735Z] ...                                                                      [ 89%]
[2023-09-20T20:44:14.639Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2023-09-20T20:44:37.652Z] ......                                                                   [ 92%]
[2023-09-20T20:44:40.402Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-20T20:47:14.507Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-09-20T20:47:14.507Z] 
[2023-09-20T20:47:14.507Z] =================================== FAILURES ===================================

fmigneault added 2 commits September 20, 2023 11:04

move STAC service Magpie definition under its component

41bf891

add initial Mapige /stac/stac resource creation

db67535

fmigneault requested review from mishaschwartz, tlvu and Nazim-crim September 20, 2023 15:29

fmigneault self-assigned this Sep 20, 2023

github-actions bot added component/magpie Related to https://github.com/Ouranosinc/Magpie documentation Improvements or additions to documentation labels Sep 20, 2023

fmigneault mentioned this pull request Sep 20, 2023

Adding redirect to /stac/stac when using twitcher #382

Merged

Nazim-crim approved these changes Sep 20, 2023

View reviewed changes

mishaschwartz requested changes Sep 20, 2023

View reviewed changes

mishaschwartz reviewed Sep 20, 2023

View reviewed changes

birdhouse/optional-components/stac-public-access/config/magpie/config.yml.template Outdated Show resolved Hide resolved

tlvu reviewed Sep 20, 2023

View reviewed changes

fmigneault added 2 commits September 20, 2023 14:57

remove stac-public-access dependnecy under all-public-access

929faa2

rename stac magpie config.yml to permissions.cfg

89f5f60

fmigneault requested review from mishaschwartz and tlvu September 20, 2023 19:19

tlvu approved these changes Sep 20, 2023

View reviewed changes

mishaschwartz requested changes Sep 20, 2023

View reviewed changes

birdhouse/components/stac/config/magpie/config.yml.template Show resolved Hide resolved

rename to plain stac magpie permissions.cfg (no template)

e0a6e0d

fmigneault requested a review from mishaschwartz September 20, 2023 19:32

mishaschwartz approved these changes Sep 20, 2023

View reviewed changes

fmigneault merged commit 7515db4 into master Sep 21, 2023
3 checks passed

fmigneault deleted the stac-magpie-provider-without-public-permissions branch September 21, 2023 01:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

move STAC service Magpie definition under its component #383

move STAC service Magpie definition under its component #383

fmigneault commented Sep 20, 2023 •

edited

Loading

Nazim-crim left a comment

mishaschwartz left a comment •

edited

Loading

fmigneault commented Sep 20, 2023

crim-jenkins-bot commented Sep 20, 2023

mishaschwartz commented Sep 20, 2023

tlvu left a comment

crim-jenkins-bot commented Sep 20, 2023

crim-jenkins-bot commented Sep 20, 2023

move STAC service Magpie definition under its component #383

move STAC service Magpie definition under its component #383

Conversation

fmigneault commented Sep 20, 2023 • edited Loading

Overview

Changes

Related Issue / Discussion

Nazim-crim left a comment

Choose a reason for hiding this comment

mishaschwartz left a comment • edited Loading

Choose a reason for hiding this comment

fmigneault commented Sep 20, 2023

crim-jenkins-bot commented Sep 20, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

mishaschwartz commented Sep 20, 2023

tlvu left a comment

Choose a reason for hiding this comment

crim-jenkins-bot commented Sep 20, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Sep 20, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Sep 20, 2023 •

edited

Loading

mishaschwartz left a comment •

edited

Loading