Skip to content

ESGFNode|ManagedInstallation

Jump to bottom Edit New page
Stephen Pascoe edited this page Apr 9, 2014 · 10 revisions
Wiki Reorganisation
This page has been classified for reorganisation. It has been given the category MOVE.
The content of this page will be revised and moved to one or more other pages in the new wiki structure.

ESGF Datanode - Manual Minimum Installation

(It is NOT recommended that you do this installation by hand, use the esg-node script to coordinate the installation!!!)
If you ignore the recommendation, you do so at your own risk! This document is not updated to match esg-node installer updates/changes.
This page was not written by the installation script (esg-node) author.

-gavin m bell

Work in Progress : See TODO sections for places that need expanding.

Contents

  1. ESGF Datanode - Manual Minimum Installation 1. Prerequisites 2. Part 1: THREDDS Data Server 3. Part 2: ESG Publisher 4. Part 3: ESGF Security
    1. Enable Tomcat SSL security
    2. Install ESGF OpenID Relying Party
    3. Install TDS security filters 5. References

This document describes how to install a minimum ESGF datanode without using the esg-node script. This method is "Managed" in the sense that the person installing must manage the process of installing each component; the advantage being that there is more control of how the datanode is installed at the expense of automating many routine tasks.

At the end of the process you will be able to:

  1. Serve NetCDF data through the THREDDS Data Server with THREDDS metadata compatible with ESGF.
  2. Manage your datasets using the ESG Publisher tool
  3. Publish your data to an ESGF Gateway
  4. Secure your published data with ESGF security

Some features of the ESGF system are not covered with by this document. They include:

  1. The ESGF node-manager component for metrics and logging

  2. P2P services like search, web-frontend, LAS

  3. Security services provided by an ESGF Gateway: MyProxy

  4. GridFTP

The installation divides into 3 parts.

'''THREDDS Data Server Installation'''

  • The TDS is a front-end application that serves NetCDF data over HTTP and via OPeNDAP. TDS also serves THREDDS catalogs that advertise datasets with accompanying metadata. '''ESG Publisher Installation'''

  • Next we install ESG Publisher (a.k.a the esgcet package). The ESG Publisher maintains a database of published datasets and configures TDS to serve this data. It allows you to publish datasets to an ESGF Gateway where it can be discovered by users. ESG Publish also allows you to unpublish datasets or supercede datasets with new versions. '''ESGF Security Installation'''

  • The final step is to configure ESGF security. This requires configuration changes to several other components and the deployment of the OpenidRelyingParty webapp which allows users to download secured data via a browser by supplying their OpenID. The ESGF Security filters filter HTTP requests to TDS and calls an external AuthorsationService (usually situated at a Gateay) to determine whether the request is allowed.

Prerequisites

Before you begin you will need

  1. NetCDF4 libraries and headers. We recommend you build NetCDF4 in full NetCDF4-format mode with OPeNDAP support which requires HDF5 curl. See [NETCDF4].

  2. A Python 2.5+ environment (2.6 recommended) with development headers (may require the python-devel rpm) and one of the Python package managers setuptools or pip [SETUPTOOLS], [PIP]

  3. Apache-tomcat TODO:version [TOMCAT]

  4. A Postgres 8.4+ installation. It is sufficient to have all privileges on a single logical database. The target system will need the postgres client libraries but the database server can be external to the node

Part 1: THREDDS Data Server

Download the TDS war from the THREDDS website and follow the installation instructions there. At the end of the process you should be able to see TDS serving it's default content by visiting the webapp's URL.

In order for esgpublish to automatically reinitialise TDS after publication an administrator password has to be set in tomcat's default authorisation system. Usually this can be done by editing the $CATALINA_HOME/conf/tomcat-users.xml file. You should add the roles tdsConfig and tdrAdmin and create a user with both of these roles. E.g.

<?xml version='1.0' encoding='utf-8'?>
<tomcat-users>
  <role rolename="manager"/>
  <role rolename="tdsConfig"/>
  <user username="dnode_user" password="<password-hash>" roles="tdsConfig"/>
</tomcat-users>

TODO: Are both these roles needed?

The <password-hash> value above should be generated using the digest.sh script provided by Tomcat:

$ $CATALINA_HOME/bin/digest.sh -a SHA foobar
foobar:8843d7f92416211de9ebb963ff4ce28125932878

Just paste the hash following the ":" into the tomcat-users.xml . You will have to insert the username (in this example dnode_user into esg.ini when configuring ESG Publisher.

Part 2: ESG Publisher

First you will need a CDAT installation. cdat-lite provides a quickly installable subset of CDAT that will suffice. cdat-lite requires NetCDF and numpy to be installed first. Although easy_install sometimes works with numpy it isn't reliable so it is recommended to download the source and build manually. You can still use easy_install to download the latest code:

$ eash_install -eb . numpy
$ cd numpy
$ python setup.py install

Now if you have numpy and NetCDF installed you can use easy_install or pip to install cdat-lite. Provided nc-config is available on your path the installer should detect all dependencies:

$ easy_install cdat_lite

If this completes without errors check you can import the cdms2 module from python

$ python
>>> import cdms2
>>>

If you want to be sure install nose and run the unit tests

$ easy_install nose
$ nosetests cdat_lite

Now install ESG publisher

$ easy_install -f http://rainbow.llnl.gov/dist/externals/ esgcet

This will download and install several dependencies including the postgresql database driver psycopg2, SQLAlchemy and lxml.

Next we will configure the ESG publisher database. Create a logical database on the database server and a user which has full privileges on that database. This user's password will be put in a configuration file so it shouldn't be a superuser.

Generate an example esg.ini file using the esgsetup command:

$ esgsetup --config

NOTE: you cant set the destination esg.ini file with the $ESGINI environment variable.

Edit $HOME/.esgcet/esg.ini . At a minimum you should set the dburl option to refer to your database. You may also want to configure the root_id option and the thredds_* options to refer to your TDS installation. For instance

  • thredds_username

  • thredds_password

  • thredds_dataset_roots

  • thredds_root

Many other options can be revisited after installation.

You can also edit $HOME/.esgcet/esgcet_models_table.txt to add models to the database. Every time you add models or projects you will need to update the database schema so it is convenient to put reasonable values there now.

Initialise the ESG publisher database. The publisher was designed to create a database and user for you but in this case we have them already so instead of running esgsetup&#160;\--db and esgcreate_tables you should reset database versioning with esgdrop_tables then migrate the schema to the latest version with esginitialise

$ esgdrop_tables
$ esginitialize -c

Provided you have configured the right dburl this should create the schema in the database and initialise it with your projects and models.

Initialise TDS for publishing. This will create the top-level catalogs used by ESG Publisher and configure it to reinitialise TDS. You will need the username and password set in tomcat-users.xml .

$ esgsetup --thredds

TODO:test-thredds-generation

Part 3: ESGF Security

You can now serve unsecured data from your ode. Before you publish your test dataset to a gateway you need to install the various security components and filters needed to support ESGF security.

Enable Tomcat SSL security

We assume you have obtained an SSL certificate/key pair from a Certificate Authority such as Verisign. ESGF Security doesn't work smoothly without certificates signed by established authorities since users will see a security warning when they visit your datanode. However, if you are simply making a test deployment guides exist on the web about how to create your own certificate/key pair [TODO: link].

Download the ESGF truststore in Java Keystore format from http://esgf.org /esg-certs . This truststore contains the public certificate of certificate authorities in the federation. You must then add your server's certificate to the truststore.

[TODO: creating a keystore from PEM certificate chain and key]

$ keytool -import -v -trustcacerts -alias tomcat -file <your-cert> -keystore <keystore> -storepass <store-password> -noprompt

Where <your-cert> is a X509 certificate in PEM format. is the password for the truststore (default "changeit").

Your Tomcat installation probably has a Connector declaration for SSL already. Configure it match the following:

<Connector port="443" SSLEnabled="true" maxHttpHeaderSize="8192"
          maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
          enableLookups="false" disableUploadTimeout="true"
          acceptCount="100" scheme="https" secure="true"
          clientAuth="want" sslProtocol="TLS"
          ciphers="TLS_DHE_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_RC4_128_MD5,
SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA"
          keystoreFile="<keystore>" keystorePass="<keystore-password>"
          truststoreFile="<truststore>" truststorePass="<truststore-password>"
/>

Install ESGF OpenID Relying Party

Download the esgf-security WAR from the ESG repository

$ wget http://rainbow.llnl.gov/dist/esg-orp/esg-orp.war

Stop tomcat and unpack the WAR file into your webapps directory. E.g.

$ cd /usr/local/tomcat/webapps
$ mkdir OpenidRelyingParty
$ cd OpenidRelyingParty
$ jar xf esg-orp.war

Edit <application-dir>/WEB-INF/classes/esg-orp.properties setting the keystoreFile, keystorePassword and keystoreAlias parameters.

Restart tomcat and navigate to http://localhost/OpenidRelyingParty . Try loging in with an ESGF OpenID.

Install TDS security filters

For this part you will need the host of the Gateway you will publish to.

Add the following XML to the top of your stack of filters in <TDS_WEB_CONTEXT>/WEB-INF/web.xml :

  <filter>
    <filter-name>authenticationFilter</filter-name>
    <filter-class>esg.orp.app.AuthenticationFilter</filter-class>
    <init-param>
      <param-name>policyServiceClass</param-name>
      <param-value>esg.orp.app.tds.TDSPolicyService</param-value>
    </init-param>
    <init-param>
      <param-name>openidRelyingPartyUrl</param-name>
      <param-value>https://@@YOUR_HOST@@/OpenidRelyingParty/home.htm</param-value>
    </init-param>
    <init-param>
      <param-name>trustoreFile</param-name>
      <param-value>@@TRUSTSTORE_FILE@@</param-value>
    </init-param>
    <init-param>
      <param-name>trustorePassword</param-name>
      <param-value>@@TRUSTSTORE_PASSWORD@@</param-value>
    </init-param>
  </filter>
  <filter-mapping>
    <filter-name>authenticationFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

  <filter>
    <filter-name>authorizationFilter</filter-name>
    <filter-class>esg.orp.app.AuthorizationFilter</filter-class>
    <init-param>
      <param-name>authorizationServiceClass</param-name>
      <param-value>esg.datanode.security.app.NoAuthorizationServiceFilterCollaborator</param-value>
      <!-- <param-value>esg.orp.app.SAMLAuthorizationServiceFilterCollaborator</param-value> -->
    </init-param>
    <init-param>
      <param-name>authorizationServiceUrl</param-name>
      <param-value>https://@@GATEWAY_HOST@@/saml/soap/secure/authorizationService.htm</param-value>
    </init-param>
  </filter>
  <filter-mapping>
    <filter-name>authorizationFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

Replace all @@...@@@ items with your configuration. Notice this configuration uses the NoAuthorizationServiceFiltercollaborator . This setting doesn't do authorization and will allow you to test the filters without a Gateway. Later we will enable the SAMLAuthorizationServiceFilterCollaborator .

Also change the definition of the RestrictedDataset Servlet to use the ORP servlet:

<servlet>
   <servlet-name>RestrictedDataset</servlet-name>
   <servlet-class>thredds.servlet.restrict.RestrictedDatasetServlet</servlet-class>

   <init-param>
     <param-name>Authorizer</param-name>
     <param-value>esg.orp.app.tds.TDSAuthorizer</param-value>
   </init-param>

   <load-on-startup>2</load-on-startup>
</servlet>

TODO: OPeNDAP url patterns

TODO: details on each item below

  1. Install TDS security filters
  2. Install MyProxyClient

References

TODO: resolve references

[NETCDF4]

[SETUPTOOLS]

[PIP]

[TOMCAT]

[CDAT-LITE]

Add a custom footer
Add a custom sidebar
Clone this wiki locally