Skip to content

Latest commit

 

History

History
374 lines (248 loc) · 14.8 KB

INSTALL.rst

File metadata and controls

374 lines (248 loc) · 14.8 KB
Raw

Installation

1. Install B2SHARE for evaluation, using Docker

1.0. Prerequisite: clone dockerize

The B2SHARE Docker image can be used as is and is available on Docker Hub as eudatb2share/b2share. However, a full B2SHARE installation requires many different services (e.g. elasticsearch, postgresql, redis, rabbitmq), thus we provide a standard deployment procedure that creates a demonstration environment. The next steps will describe how to use this deployment.

First clone the dockerize repository:

$ git clone https://github.com/EUDAT-B2SHARE/dockerize.git

1.1. Prerequisite: B2ACCESS configuration

B2SHARE requires B2ACCESS for user management. For this purpose you must create a new B2ACCESS OAuth client, providing as the 'return URL' the address of the local B2SHARE server and the authorization path: https://$FQDN/api/oauth/authorized/b2access/ (where $FQDN must be replaced with the domain of the B2SHARE server).

After successfully registering the B2ACCESS account please set the following environment variables with the username and password provided for the B2ACCESS account:

Configuration before version v2.2.5:

$ export B2ACCESS_CONSUMER_KEY=...    # the username used for registration
$ export B2ACCESS_SECRET_KEY=...      # the password used for registration

For demonstration purposes it's recommended to use the staging B2ACCESS server. In this case make sure to define and export the USE_STAGING_B2ACCESS variable:

$ export USE_STAGING_B2ACCESS=1

Configuration from v2.3.0:

With B2SHARE configuration loader introduced in v2.3.0, a B2SHARE_ prefix needs to be added in front of variables. A new variable for enabling B2ACCESS was introduced:

$ export B2SHARE_B2ACCESS_CONSUMER_KEY=...    # the username used for registration
$ export B2SHARE_B2ACCESS_SECRET_KEY=...      # the password used for registration
$ export B2SHARE_USE_STAGING_B2ACCESS=...     # Configure B2ACCESS
$ export B2SHARE_CONFIG_ENABLE_B2ACCESS=1     # Enable B2ACCESS feature

1.2. Prerequisite: B2SHARE secret key

The B2SHARE_SECRET_KEY environment variable (a secret string used for encrypting user sessions) must also be set as an environment variable. It is recommended to use a randomly generated string for this purpose. E.g.:

$ export B2SHARE_SECRET_KEY=mfTpGpQBwTTOmWCoZrYSuiJ1UzXq1d2L

1.3. Prerequisite: configuring B2SHARE

B2SHARE needs connection to a list of external services: redis, elasticsearch, a database (postgresql or mysql), and rabbitmq. To simplify the creation of this necessary execution environment, the Docker application engine is used. The default Docker configuration for B2SHARE will try to mount a folder from the host machine. This host directory is designated by the environment variable $B2SHARE_DATADIR which should be an absolute path. This folder is used to store all the data uploaded into B2SHARE and is therefore very important.

B2SHARE has a set of configuration parameters, which should be inspected and changed when necessary by the site administrator. On the first run of the service, a configuration file will be created in the B2SHARE data location (which also by default is mounted to the host folder ${B2SHARE_DATADIR}/b2share-data/ as described above). The name of this configuration file is b2share.cfg. Please note that after each update of this file the service needs to be restarted (with e.g.``docker-compose restart``) for the changes to take effect.

By default, the B2SHARE service will use a newly created postgresql database. This database's data is persisted in ${B2SHARE_DATADIR}/postgresql-data/. In case you want to use an external database, one can be configured by editing the b2share.cfg file and defining the SQLALCHEMY_DATABASE_URI variable. For example, for a postresql database, use SQLALCHEMY_DATABASE_URI = "postgresql://db_username:db_password@db_host/db_name") Note that you also need to remove the B2SHARE_SQLALCHEMY_DATABASE_URI variable in the docker-compose.yml file.

B2SHARE uses nginx as the front-end HTTP server, responsible for establishing secure connections with clients over https. By default an autogenerated security certificate is used, which triggers security warnings in the client browsers. For a production server it is recommended to use a valid certificate. To use it with nginx, place the certificate files in an empty folder on the host machine (e.g. /b2share-etc/ssl) and mount the folder in the nginx container by updating the volumes declaration in the docker-compose.yml file. Make sure that the following files are present in the mounted folder:

ssl_certificate         /b2share-etc/ssl/b2share.crt
ssl_certificate_key     /b2share-etc/ssl/b2share.key

1.4. Run B2Share with Docker

After all the configuration parameters have been set go into the recently downloaded dockerize folder and run docker-compose, e.g.:

Before v2.2.5:

$ cd dockerize

## mandatory environment variables
$ export B2ACCESS_CONSUMER_KEY=...           # the username used for registration
$ export B2ACCESS_SECRET_KEY=...             # the password used for registration
$ export B2SHARE_SECRET_KEY=...              # a secret string used for encrypting user sessions
$ export B2SHARE_JSONSCHEMAS_HOST='<FQDN>'   # replace <FQDN> with the domain of the B2SHARE server
$ export B2SHARE_POSTGRESQL_DBNAME=...       # name of the postgresql database used by b2share
$ export B2SHARE_POSTGRESQL_PASSWORD=...     # password used by b2share when accessing the postgresql database
$ export B2SHARE_POSTGRESQL_USER=...         # username used by b2share when accessing the postgresql database (default=b2share)
$ export B2SHARE_RABBITMQ_USER=...           # username used by b2share when accessing RabbitMQ service
$ export B2SHARE_RABBITMQ_PASS=...           # password used by b2share when accessing RabbitMQ service
$ export B2SHARE_DATADIR='<PATH>'            # path on the host which will be mounted and contain all b2share related data, including postgresql, elasticsearch, redis, rabbitmq, nginx and b2share itself.

## optional environment variables
$ export USE_STAGING_B2ACCESS=1              # to run with staging (testing) b2access (unity install)
$ export INIT_DB_AND_INDEX=1                 # when run the first time, initialize the database and indices
$ export LOAD_DEMO_COMMUNITIES_AND_RECORDS=1 # when run the first time, load some communities and records for demonstration

## run it
$ docker-compose up -d

From v2.3.0:

$ cd dockerize

## mandatory environment variables
$ export B2SHARE_B2ACCESS_CONSUMER_KEY=...           # the username used for registration
$ export B2SHARE_B2ACCESS_SECRET_KEY=...             # the password used for registration
$ export B2SHARE_SECRET_KEY=...              # a secret string used for encrypting user sessions
$ export B2SHARE_JSONSCHEMAS_HOST='<FQDN>'   # replace <FQDN> with the domain of the B2SHARE server
$ export B2SHARE_POSTGRESQL_DBNAME=...       # name of the postgresql database used by b2share
$ export B2SHARE_POSTGRESQL_PASSWORD=...     # password used by b2share when accessing the postgresql database
$ export B2SHARE_POSTGRESQL_USER=...         # username used by b2share when accessing the postgresql database (default=b2share)
$ export B2SHARE_RABBITMQ_USER=...           # username used by b2share when accessing RabbitMQ service
$ export B2SHARE_RABBITMQ_PASS=...           # password used by b2share when accessing RabbitMQ service
$ export B2SHARE_DATADIR='<PATH>'            # path on the host which will be mounted and contain all b2share related data, including postgresql, elasticsearch, redis, rabbitmq, nginx and b2share itself.
$ export B2SHARE_LOGGING_LEVEL=...           # set logging level, must be int or allowed string

## optional environment variables
$ export B2SHARE_USE_STAGING_B2ACCESS=1              # to run with staging (testing) b2access (unity install)
$ export B2SHARE_INIT_DB_AND_INDEX=1                 # when run the first time, initialize the database and indices
$ export B2SHARE_LOAD_DEMO_COMMUNITIES_AND_RECORDS=1 # when run the first time, load some communities and records for demonstration
$ export B2SHARE_CONFIG_ENABLE_B2ACCESS=1     # Enable B2ACCESS feature
$ export B2SHARE_CONFIG_USE_VAULT_AGENT=...   # If a secret provider is used, such as Vault

## run it
$ docker-compose up -d

After the docker image is built and running, B2SHARE will be available at https://<FQDN>. If B2SHARE does not become available, you can try inspecting the container logs to find the cause:

$ docker-compose logs -f b2share

1.5. Stopping all services

The services can be stopped just like any other docker-compose deployment.

$ docker-compose down

This command will stop and remove all containers. The data will still be available in the directory pointed by $B2SHARE_DATADIR.

1.6. Restarting all services

The services can be started as before. If the directory pointed by the environment variable $B2SHARE_DATADIR already contains data from a previous run, the data will be reused. See section 1.7 for removing all previously persisted data if you want to start from scratch.

$ docker-compose up -d

1.7. Removing all data

In order to remove all persisted data one can just delete the directory pointed by $B2SHARE_DATADIR. All containers must be stopped before removing the directory. Remember to also delete the containers.

2. Running the service in production

The provided docker-compose file is not production ready. The database has no backup mechanism running. The system administrator can comment the "postgres" service and set the SQLALCHEMY_DATABASE_URI as described in section 1.3. It is also possible to enhance the postgresql container by building a custom one.

By default files uploaded by users are written in ${B2SHARE_DATADIR}/b2share-data/files. This location must be backuped or written on a safe filesystem.

If the containers need to run on different servers it is recommended to use native docker volumes instead of mounted host directories.

3. Integration with other services

The records metadata in your local instance of B2SHARE can be harvested by various other repositories by using the included endpoint for the OAI-PMH protocol. For the records to be visible in B2FIND, please create a ticket in the B2FIND support queue (https://eudat.eu/support- request?service=B2FIND), by describing your instance and specifying the OAI-PMH url (http://<FQDN>/oai2d)

4. Install B2SHARE for development

Before installing B2Share you will need the following software:

  • python3
  • virtualenv and virtualenvwrapper
$ # on OSX, with brew:
$ brew install python --framework --universal
$ pip install virtualenv virtualenvwrapper

and a working installation of the following services:

  • Elasticsearch 2.4
  • Redis 2.10.5
  • RabbitMQ 3.6
  • Celery 4.1
  • PostgreSQL 9.6 and psycopg2

Another option, which requires a bit more RAM, is to use a docker-compose.yml file to manage the starting and stopping of the services, and just have a bare metal installation of B2Share for development.

Now to install B2Share, first create a new virtualenv with:

$ virtualenv -p python3 b2share

and clone the B2Share repository from github:

$ git clone https://github.com/eudat-b2share/b2share.git

Now we can install b2share for development with pip:

$ pip install -r requirements.txt
$ pip install -e .[all]

and start all services.

B2Share requires some configuration as mentioned in section 1.1 of this document so make sure to follow the steps exporting the enviroment variables, and the additional ones required for local installations:

$ export B2SHARE_UI_PATH=<path-to-b2share-repository>/webui/app
$ export B2SHARE_SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://$B2SHARE_POSTGRESQL_USER:$B2SHARE_POSTGRESQL_PASSWORD@localhost:5432/$B2SHARE_POSTGRESQL_DBNAME

Finally, create the DB:

$ b2share upgrade run

create the Elasticsearch indices and load the templates:

$ b2share index init
$ b2share schemas init

and start B2Share with:

$ b2share run

If you wish to add some demo records and communities, then install the b2share demo application:

$ cd <path-to-b2share-repository>/demo/
$ pip install .

and load the demo data:

$ b2share demo load_data -vv

Please note that a custom B2ACCESS configuration is also needed, as described above. The 'return URL' of the B2ACCESS configuration in this case can be set to http://localhost:5000/api/oauth/authorized/b2access/

If working on the web UI, see also: https://github.com/EUDAT-B2SHARE/b2share/wiki/Developer's-corner.

FAQ:

  • There is a timeout when trying to create a new record from the UI, or the server hangs when trying to create a draft from the REST API.

This could indicate that the JSONSCHEMAS_HOST is misconfigured. If you are running B2SHARE locally then it should be set to localhost:5000.

  • When trying to login the following error is shown:

    ERROR OAuth Authorization Server got an invalid request.

    If you are a user then you can be sure that the web application you was using previously is either misconfigured or buggy.

    If you are an administrator or developer the details of the error follows:

    The client '<your-b2access-consumer-key>' is unknown

This means that your B2ACCESS credentials are incorrect, or you are working with the staging server and you didn't set USE_STAGING_B2ACCESS=1.

  • There is the following error when you run b2share upgrade run: sqlalchemy.exc.InvalidRequestError: Naming convention including %(constraint_name)s

If your alembic recipes are unchanged and you also didn't change any of your models.py, this could be due to not having set the B2SHARE_SQLALCHEMY_DATABASE_URI to use Postgresql, and you are using the default which is SQLite, which doesn't support DB migrations with alembic.