Adding AWS HealthOmics as a Module in "Play" tools

[ironspur5]

Feature or Bugfix

• Feature

Background

Currently, data.all has integrations to AWS services such as SageMaker, Athena, and QuickSight through the “Play” modules/tools Notebooks, Worksheets, and Dashboards, respectively. This is valuable and convenient for end users who want to process and share their data but may not want to or know how to use these services in the AWS Console.

Researchers, scientists, and bioinformaticians often fall into this end user category. One such AWS service that is popular amongst the research community is AWS HealthOmics. HealthOmics helps users process and generate insights from genomics and other biological data stored in the cloud. Raw genomic data can be sent to a HealthOmics managed workflow (aka Ready2Run workflow) that can perform various tasks such as quality control, read alignment, transcript assembly, and gene expression quantification. The output can then be stored in a manageable format for querying/visualizing/sharing.

AWS HealthOmics Integration

This feature contains both modularized backend and frontend changes for adding HealthOmics as a “Play” module/tool to data.all. It specifically adds the capability to view and instantiate HealthOmics Ready2Run workflows as runs that can output and save omic data as data.all Datasets.

Consumption Patterns

• data.all Worksheets: Users can use Worksheets to make data easier to query and combine with other forms of health data.
• data.all Notebooks/Studio: Users can Notebooks and Studio to build, train, and deploy novel machine learning algorithms on the multiomic and multimodal data.
• data.all Dashboards: Users can use the transformed data in Dashboards for advanced analytics and visualizations.

Considerations

• Linked Environment and Dataset Region: The HealthOmics run must be performed in a data.all Linked Environment that is located in an AWS Region that supports AWS HealthOmics. Similarly, the data.all source and destination Dataset must live in same AWS Region as where the user will perform the HealthOmics run.
• Ready2Run Workflow Support: Currently, only Ready2Run workflows are supported. Ready2Run are pre-built workflows designed by industry leading third-party software companies like Sentieon, Inc. and NVIDIA, as well as common open-source pipelines such as AlphaFold for protein structure prediction. Ready2Run workflows do not require the management of software tools or workflow scripts. Bring your own, also known as Private, workflows where you bring a custom workflow script, are not yet supported. Please note that some Ready2Run workflows require a subscription/license from the software provider to run.

User Journey

This example user journey depicts an end-to-end process from viewing available HealthOmics Ready2Run workflows to instantiating a run and viewing its output in a data.all Worksheet.

• Initiation:
  • User navigates to the "Omics" section within data.all and browses Ready2Run workflows

• User can also search for a specific workflow directly

• After clicking on a workflow, users see a detailed view of it with a full description of what it does

• Creation: After clicking on a workflow, users see a detailed view and hit “Create Run”. Users fill in the run creation form with the following parameters:

• Workflow ID: Immutable ID of the Ready2Run workflow
  * Run Name: Customizable name of the run user will submit
  * Environment: data.all Environment AWS Account where the HealthOmics run will be (NOTE: the Environment must be in an AWS Region supported by HealthOmics, ex: N. Virginia or London)
  * Region: Pre-populated from the Environment and immutable Region where the run will be
  * Owners: data.all group who owns the run
  * Select S3 Output Destination: data.all Dataset where the output omics data will reside (NOTE: please create this prior to kicking off a run)
  * Run Parameters: JSON parameter input in the format expected by the Ready2Run workflow. It will be pre-populated with the correct fields, and users will paste in their data in the appropriate fields. For example, the raw input data in S3 that will be processed in the run. (NOTE: the input data does not have to be in a data.all Dataset, as long as it is accessible. For example, raw genomic data may be hosted publicly on the AWS Registry of Open Data, and the S3 URI can be provided in a field here)

• History:

  • Users navigate to Run tab at the top to view a history of the data.all-initiated Ready2Run workflows they’ve kicked off. (NOTE: run history deletion is still in progress)

• Data Consumption:

  • In Worksheets:
    • Users can select (or create) a new Worksheet.

• Users can then query the data using SQL

Relates

• Github Issue - HealthOmics module in "Play" tools #563

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

• Does this PR introduce or modify any input fields or queries - this includes
  fetching data from storage outside the application (e.g. a database, an S3 bucket)? Yes
  • Is the input sanitized? Yes
  • What precautions are you taking before deserializing the data you consume? N/A
  • Is injection prevented by parametrizing queries? N/A
  • Have you ensured no eval or similar functions are used? N/A
• Does this PR introduce any functionality or component that requires authorization? Yes
  • How have you ensured it respects the existing AuthN/AuthZ mechanisms? Yes
  • Are you logging failed auth attempts? N/A
• Are you using or adding any cryptographic features? No
  • Do you use a standard proven implementations?
  • Are the used keys controlled by the customer? Where are they stored?
• Are you introducing any new policies/roles/users? Yes
  • Have you used the least-privilege principle? How? Yes, through scoped policies added to the role

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[noah-paige]

First a foremost - this is a great contribution from @ironspur5 and team! All of your work and collaboration is super appreciated and I think this is an amazing starting point for health science applications and data consumption patterns in data.all :)

I left some comments throughout the PR - most are small changes or touch ups but let me know if any questions or disagreements. Additionally, some high-level notes I too:

Notes:

• For existing DA Deployment (a heads up on what we need to call out during release @dlpzx)
  • Existing Groups need to be given MANAGE_OMICS permission from tenant
  • Need to update envs after enabling omics module
  • Need to run omics workflow fetcher at least 1 time before can start runs
• Any need for an Omits VPC Endpoint? Similar to how we have VPC interface endpoints for other supported services

Enhancement Ideas (can be picked up after this PR)

• Permissions
  • Is there any need for more in depth permission model? Currently we have:
    • MANAGE_OMICS_RUNS (Tenant Level)
    • CREATE_OMICS_RUNS (on Env Record)
    • DELETE_OMICS_RUN (on OmicsRun Record)
  • Especially if we move to supporting Private Workflows May Need more verbose permissions for update, get, list, for one if not both of omics workflow and omics runs
• Private Workflow Types, RunGroups, etc.
• Search + Filters on Omics Run History
• Better UX JSON editing for parameter template
• Provide Output Locations specific to Table Location or Folder Location rather than entire Dataset

Tests I performed:

• CICD Pipeline Completes
• Env Stack Update - adds Omics permissions to pivot role and env group roles (if they have omics env group permission)
• Omics Permissions Show (both in Admin Tenant Permissions and Env Group Permissions + can edit appropriately for each)
• ECS Workflow Fetch Task Successful (IMPORTANT: Omics Module needs to be enabled - left comment on file)
• Workflows Paginated and showing on Omics Workflow Tab
• Workflow Card with Additional Description and Parameter Template Shows Properly
• Create Run from Workflow (Tested ESM Fold Workflow - mimiced Video)
• Run History Shows OmicsRuns for the Users Teams (other users in separate team cannot view Omics Runs)
• Run Status Updates when Job Completes
• Output Data to correct S3 Output (tested both same dataset and a different owned dataset)
• Omics Run Creator Team can Delete Job Run as well
• Test with Shared Dataset as Input
• Crawl + Query Output metrics in Worksheets

[noah-paige]

A final suggestion left on omics_workflow_fetcher task

And 1 last thought - should we restrict omics runs to follow a pattern using NamingConventionService similar to how we do with other resources?

• If so think we can also then restrict the pivot role permissions to arn:aws:omics:{self.region}:{self.account}:run/{self.env_resource_prefix}* (right now it is wildcard on name)

If we do not implement naming convention like above - we may want to remove resource prefix restriction on env group role
arn:aws:omics:{self.region}:{self.account}:run/{self.resource_prefix}* (currently implemented as such in backend/dataall/modules/omics/cdk/env_role_omics_policy.py

[ironspur5]

A final suggestion left on omics_workflow_fetcher task

And 1 last thought - should we restrict omics runs to follow a pattern using NamingConventionService similar to how we do with other resources?

* If so think we can also then restrict the pivot role permissions to `arn:aws:omics:{self.region}:{self.account}:run/{self.env_resource_prefix}*` (right now it is wildcard on name)

If we do not implement naming convention like above - we may want to remove resource prefix restriction on env group role arn:aws:omics:{self.region}:{self.account}:run/{self.resource_prefix}* (currently implemented as such in backend/dataall/modules/omics/cdk/env_role_omics_policy.py

I am not 100% certain, but I also don't think the self.env_resource_prefix is actually doing anything. Run ARNs don't include any naming/prefix in their format, just the Run ID. I think I put this in there to match what other modules looked like. But the tag key condition to prevent unauthorized run access is really what I was intending for.

[noah-paige]

thanks again for all you hard work here @ironspur5 @dlpzx + team

Final changes look good - approving!

[anmolsgandhi]

Great work on this initiative team - the collaboration and coordination across the teams, all focused on delivering the best possible experience for our customers, has been awesome. Let's continue this collaboration for other initiatives as well.

[ironspur5]

Thank you for all your support in getting this added, team!!