Feature/dac 300 weaver compose

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 1.16.2
+current_version = 1.17.0
 commit = True
 tag = False
 tag_name = {new_version}

.gitignore

@@ -1,14 +1,21 @@
+## Project Files
 birdhouse/env.local
 birdhouse/certkey.pem
 birdhouse/cert.pem
 birdhouse/key.pem
+
+## Deployment
 vagrant_variables.yml
 .vagrant/
 ubuntu-bionic-18.04-cloudimg-console.log
+
+## IDEs
 .idea
 .vscode
-*.bak
-weaver/
 
-# vim temp files
+## Backup Files
+**/*.bak*
+**/*.backup*
+
+## vim temp files
 *.swp

CHANGES.md

@@ -16,6 +16,40 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[1.17.0](https://github.com/bird-house/birdhouse-deploy/tree/1.17.0) (2021-11-01)
+------------------------------------------------------------------------------------------------------------------
+
+  ### Changes
+
+  - Adds [Weaver](https://github.com/crim-ca/weaver) to the stack (optional) when ``./components/weaver`` 
+    is added to ``EXTRA_CONF_DIRS``. For more details, refer to 
+    [Weaver Component](https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/components/README.rst#Weaver)
+    Following happens when enabled:
+
+    * Service ``weaver`` (API) gets added with endpoints ``/twitcher/ows/proxy/weaver`` and ``/weaver``.
+
+    * All *birds* offering a WPS 1.x/2.x endpoint are automatically added as providers known by `Weaver`
+      (birds: ``catalog``, ``finch``, ``flyingpigeon``, ``hummingbird``, ``malleefowl`` and ``raven``).
+      This offers an automatic mapping of WPS 1.x/2.x requests of process descriptions and execution nested under
+      the *birds* to corresponding [OGC-API - Processes](https://github.com/opengeospatial/ogcapi-processes/) 
+      RESTful interface (and added functionalities). 
+
+    * New processes can be deployed and executed using 
+      Dockerized [Application Packages](https://pavics-weaver.readthedocs.io/en/latest/package.html).
+      Additionally, all existing processes (across *bird* providers and Dockerized Application Packages) 
+      can be chained into [Workflows](https://pavics-weaver.readthedocs.io/en/latest/processes.html#workflow)
+
+    * Images ``weaver-worker`` (`Weaver`'s job executor) and ``docker-proxy`` (sibling Docker container dispatcher)
+      are added to the stack to support above functionalities.
+
+    * Adds `Magpie` permissions and service for `Weaver` endpoints.
+
+    * Adds ``./optional-components/test-weaver`` for even more `Magpie` extended permissions for `Weaver` 
+      for getting access to resources for functionalities required by [Weaver Testing notebook][weaver-test-notebook].
+
+[weaver-test-notebook]: https://github.com/Ouranosinc/pavics-sdi/blob/master/docs/source/notebook-components/weaver_example.ipynb
+
+
 [1.16.2](https://github.com/bird-house/birdhouse-deploy/tree/1.16.2) (2021-10-27)
 ------------------------------------------------------------------------------------------------------------------
 

README.rst

@@ -14,13 +14,13 @@ for a full-fledged production platform.
     * - releases
       - | |latest-version| |commits-since|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.16.2.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.17.0.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.16.2...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/1.17.0...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-1.16.2-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-1.17.0-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.16.2
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/1.17.0
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

birdhouse/components/README.rst

@@ -61,7 +61,7 @@ How to Enable the Component
   - Add "./components/scheduler" to ``EXTRA_CONF_DIRS``.
   - Set ``AUTODEPLOY_EXTRA_REPOS``, ``AUTODEPLOY_DEPLOY_KEY_ROOT_DIR``,
     ``AUTODEPLOY_PLATFORM_FREQUENCY``, ``AUTODEPLOY_NOTEBOOK_FREQUENCY`` as desired,
-    full documentation in env.local.example_.
+    full documentation in `env.local.example`_.
   - Run once fix-write-perm_ (:download:`download <../deployment/fix-write-perm>`), see doc in script.
 
 
@@ -163,7 +163,7 @@ able to access them.
 How to Enable the Component
 ---------------------------
 
-- Edit ``env.local`` (a copy of env.local.example_ (:download:`download <../env.local.example>`))
+- Edit ``env.local`` (a copy of `env.local.example`_ (:download:`download <../env.local.example>`))
 
   - Add "./components/monitoring" to ``EXTRA_CONF_DIRS``
   - Set ``GRAFANA_ADMIN_PASSWORD`` to login to Grafana
@@ -241,8 +241,71 @@ Customizing the Component
   Slack or other services accepting webhooks), ``ALERTMANAGER_EXTRA_RECEIVERS``.
 
 
+Weaver
+======
+
+By enabling this component, the `Weaver`_ service will be integrated into the stack.
+
+This component offers `OGC API - Processes`_ interace to WPS components (a.k.a `WPS-REST bindings` and
+`WPS-T (Transactional)` support).
+This provides a RESTful JSON interface with asynchronous WPS processes execution over remote instances. 
+Other WPS components of the birdhouse stack (`finch`_, `flyingpigeon`_, etc.) will also all be registered
+under `Weaver`_ in order to provide a common endpoint to retrieve all available processes, and dispatch 
+their execution to the corresponding service.
+Finally, `Weaver`_ also adds `Docker` image execution capabilities as a WPS process, allowing deployment 
+and execution of custom applications and workflows.
+
+
+Usage
+-----
+
+Once this component is enabled, `Weaver`_ will be accessible at ``https://<PAVICS_FQDN_PUBLIC>/weaver`` endpoint,
+where ``PAVICS_FQDN_PUBLIC`` is defined in your ``env.local`` file.
+
+Full process listing (across WPS providers) should be available using request:
+
+.. code-block::
+
+    GET https://<PAVICS_FQDN_PUBLIC>/weaver/processes?providers=true
+
+Please refer to the `Weaver OpenAPI`_ for complete description of available requests.
+This description will also be accessible via ``https://<PAVICS_FQDN_PUBLIC>/weaver/api`` once the instance is started.
+
+For any specific details about `Weaver`_ configuration parameters, functionalities or questions, please refer to its
+`documentation <https://pavics-weaver.readthedocs.io/en/latest/>`_.
+
+How to Enable the Component
+---------------------------
+
+- Edit ``env.local`` (a copy of `env.local.example`_)
+
+  - Add ``"./components/weaver"`` to ``EXTRA_CONF_DIRS``.
+
+  - Component ``birdhouse/optional-components/all-public-access`` should also be enabled to ensure that `Weaver`_
+    can request ``GetCapabilities`` of every WPS provider to be registered. Publicly inaccessible services will not
+    succeed registration and will not provide the WPS-REST interface.
+
+
+Customizing the Component
+-------------------------
+
+- Edit ``env.local`` (a copy of `env.local.example`_)
+
+  - Optionally, set any additional environment variable overrides amongst values defined in `weaver/default.env`_.
+
+  - Optionally, mount any additional `Weaver`_-specific configuration files
+    (see contents of ``birdhouse/components/weaver/config/weaver``) if extended functionalities need to be defined.
+    Further ``docker-compose-extra.yml`` could be needed to define
+    any other ``volumes`` entries where these component would need to be mounted to.
+
 
 
+.. _finch: https://github.com/bird-house/finch
+.. _flyingpigeon: https://github.com/bird-house/flyingpigeon
+.. _Weaver: https://github.com/crim-ca/weaver
+.. _Weaver OpenAPI: https://pavics-weaver.readthedocs.io/en/latest/api.html
+.. _weaver/default.env: ./weaver/default.env
+.. _OGC API - Processes: https://github.com/opengeospatial/ogcapi-processes
 .. _env.local.example: ../env.local.example
 .. _fix-write-perm: ../deployment/fix-write-perm
 .. _deploy.sh: ../deployment/deploy.sh

birdhouse/components/weaver/.gitignore

@@ -0,0 +1,6 @@
+conf.extra-service.d/weaver.conf
+config/magpie/config.yml
+config/weaver/data_sources.yml
+config/weaver/request_options.yml
+config/weaver/wps_processes.yml
+config/weaver/weaver.ini

birdhouse/components/weaver/celery-healthcheck

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+INFO="[INFO]  "
+ERROR="[ERROR] "
+
+echo "${INFO}Starting Celery Healthcheck"
+result=$(celery inspect registered -A pyramid_celery.celery_app --ini "${APP_CONFIG_DIR}/weaver.ini" --timeout 5)
+if [[ -z "${result}" ]]; then
+  echo "${ERROR}Empty Celery response."
+  exit 1
+fi
+
+# should look like:
+#   [... weaver-logs ...]
+#   -> celery@<uuid>: OK
+#       * weaver.processes.execution.execute_process
+result=$(echo "${result}" | tail -n 2)
+status=$(echo "${result}" | head -n 1)
+task=$(echo "${result}" | tail -n 1)
+
+echo "${INFO}Celery inspect results dump:"
+echo "${result}"
+
+if [[ $(echo "${status}" | grep -c "OK") -ne 1 ]]; then
+  echo "${ERROR}Celery inspect did not return expected 'OK' status."
+  exit 2
+fi
+if [[ $(echo "${task}" | grep -c "weaver.processes.execution.execute_process") -ne 1 ]]; then
+  echo "${ERROR}Celery inspect did not retrieve expected task."
+  exit 3
+fi
+
+echo "${INFO}Success. Celery Task is ready."
+exit 0

birdhouse/components/weaver/conf.extra-service.d/weaver.conf.template

@@ -0,0 +1,25 @@
+
+    # NOTE:
+    #   Redirect to internal network of twitcher with Weaver root endpoint and alias allows to set
+    #   the same 'magpie' permissions on the 'weaver' service defined by "WEAVER_MANAGER_NAME".
+    #   This allows verification of the same service user/group permissions references regardless
+    #   whether the *shortcut* Weaver endpoint, the alias or the explicit 'twitcher' proxy route is used.
+    # redirect EMS/ADES to actual secured Weaver path
+    #location /${WEAVER_CONFIG} {
+    #    return 302 https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/${WEAVER_MANAGER_NAME};
+    #}
+
+    location /${WEAVER_MANAGER_NAME} {
+        proxy_pass https://${PAVICS_FQDN_PUBLIC}${TWITCHER_PROTECTED_PATH}/${WEAVER_MANAGER_NAME};
+        proxy_set_header Host $host;
+        proxy_buffering off;
+        include /etc/nginx/conf.d/cors.include;
+    }
+
+    # NOTE:
+    #   this is needed only if not using the location already provided by the core configuration
+    #   see 'birdhouse/config/proxy/conf.d/all-services.include.template'
+    # location where process job outputs will be accessible
+    #location ^~ ${WEAVER_WPS_OUTPUTS_PATH}/ {
+    #    alias ${WEAVER_WPS_OUTPUTS_DIR}/;
+    #}

birdhouse/components/weaver/config/canarie-api/weaver_config.py.template

@@ -0,0 +1,9 @@
+SERVICES['node']['monitoring'].update({
+    'Weaver': {
+        'request': {
+            'url': 'https://${PAVICS_FQDN_PUBLIC}/weaver'
+        },
+    },
+})
+
+# vi: tabstop=8 expandtab shiftwidth=4 softtabstop=4 syntax=python

birdhouse/components/weaver/config/magpie/config.yml.template

@@ -0,0 +1,138 @@
+## see reference example for details:
+# https://github.com/Ouranosinc/Magpie/blob/master/config/config.yml
+
+providers:
+  # definition of Weaver service
+  ${WEAVER_MANAGER_NAME}:
+    url: http://${PAVICS_FQDN}:4001
+    title: Weaver (${WEAVER_CONFIG})
+    public: true
+    c4i: false
+    type: api   # FIXME: 'ades' when https://github.com/Ouranosinc/Magpie/issues/360 implemented
+    sync_type: api
+
+  # FIXME: remove when https://github.com/Ouranosinc/Magpie/issues/360 implemented, see 'default.env'
+  ${WEAVER_WPS_NAME}:
+    url: http://${PAVICS_FQDN}:4001/wps
+    title: Weaver (WPS)
+    public: true
+    c4i: false
+    type: wps
+    sync_type: wps
+
+permissions:
+  # Administrator permissions over whole service
+  - service: ${WEAVER_MANAGER_NAME}
+    permission: read
+    group: administrators
+    action: create
+  - service: ${WEAVER_MANAGER_NAME}
+    permission: write
+    group: administrators
+    action: create
+
+  # Entrypoint
+  - service: ${WEAVER_MANAGER_NAME}
+    permission: read-match
+    group: anonymous
+    action: create
+
+  # OpenAPI JSON schema
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /json
+    permission: read
+    group: anonymous
+    action: create
+
+  # OpenAPI Swagger
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /api
+    permission: read
+    group: anonymous
+    action: create
+
+  # Version details
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /versions
+    permission: read
+    group: anonymous
+    action: create
+
+  # OGC-API conformance
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /conformance
+    permission: read
+    group: anonymous
+    action: create
+
+  # Process deployment (write) and listing (read)
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /processes    # GET is processes listing, POST is deploy: only allow view by anonymous
+    permission: read        # under '/processes/...', JSON 'DescribeProcess', POST job submit, GET results, etc.
+    group: anonymous
+    action: create
+
+  # convenience group that should be allowed deployment of Application Packages provided as OGC-API REST-JSON WPS
+  # NOTE: all users member of this group will be allowed to deploy/undeploy **any** app in Weaver
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /processes
+    permission: write
+    group: app_producers
+    action: create
+
+  # Providers listing
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers
+    permission: read-match
+    group: anonymous
+    action: create
+
+  # Provider processes listing
+  # (each provider in './components/weaver/config/weaver/wps_processes.yml.template')
+  # catalog
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/catalog
+    permission: read
+    group: anonymous
+    action: create
+  # finch
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/finch
+    permission: read
+    group: anonymous
+    action: create
+  # flyingpigeon
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/flyingpigeon
+    permission: read
+    group: anonymous
+    action: create
+  # hummingbird
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/hummingbird
+    permission: read
+    group: anonymous
+    action: create
+  # malleefowl
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/malleefowl
+    permission: read
+    group: anonymous
+    action: create
+  # raven
+  - service: ${WEAVER_MANAGER_NAME}
+    resource: /providers/raven
+    permission: read
+    group: anonymous
+    action: create
+
+  # FIXME: remove when https://github.com/Ouranosinc/Magpie/issues/360 implemented
+  # for the moment, align WPS processes to allow public GetCapabilities and DescribeProcess like REST endpoint does
+  - service: ${WEAVER_WPS_NAME}
+    permission: getcapabilities
+    group: anonymous
+    action: create
+  - service: ${WEAVER_WPS_NAME}
+    permission: describeprocess
+    group: anonymous
+    action: create