Integrate MDTranslator into Datagov Harvesting Logic

[btylerburton]

User Story

In order to successfully transform datasets from one schema to another, datagovteam would like to use the MDTranslator library, via a Rails application, to do so.

As an interim step, while the DCAT-US writer is still in active development, datagovteam would like to transform an FGDC/CSDGM source into ISO, in order to validate that the mdTranslator is functioning correctly.

Depends on:

• Harvester 2.0: Deploy mdTranslator on cloud.gov #4467

Acceptance Criteria

• GIVEN a harvest source with an FGDC CSDGM schema, stored in a WAF file format
  WHEN that source has been loaded into a variation of the Airflow ETL Pipeline, which does not include the validate and load steps
  THEN datagov-harvesting-logic will utilize the MDTranslator Rails application to to transform the source into valid ISO 19115 format.

Background

This will require work in both the datagov-harvesting-logic repo to integrate the MDTranslator Rails application, and will also potentially require work in the datagov-harvester (our Airflow / Orchestration repo) to allow for deferred operations.

At present, this additional work to support deferred operations is not necessarily a given, and could be a future enhancement to unblock pipeline workers while the transform is processing.

Security Considerations (required)

None. All work will reside within the Cloud.gov boundary and no external routes should be necessary.

Sketch

• Integrate the MDTranslator rails application into a local Docker Compose container network
• Test that local integration fully before proceeding to Cloud.gov integration
• Implement the same locally tested solution in Cloud.gov
  • Note that this will involve the creation of new internal routes and network mappings, and it is encouraged to keep these considerations in mind while developing the local service
• Configure a WAF Harvest source using FGDC/CSDGM standard metadata
• Extract that WAF using datagov-harvesting-logic WAF extract task
• Transform datasets into ISO 19115 using the MDTranslator application
• Check for errors

[rshewitt]

#4940 reminded me to configure the rails app for production ( e.g. rails server -e production ). i'll look into this later.

[rshewitt]

spoke with james and tyler and we've decided to update the values for schema_type & source_type. schema_type can be "iso19115_1", "iso19115_2", "csdgm", "dcatus" and source_type can be "document", "waf".

[Bagesary]

MOved to Sprint backlog to focus on non-harvester stories