feature/es-phase-2

CHANGELOG.md

@@ -4,6 +4,95 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
+## Phase 2 Release
+
+### Breaking Changes
+
+- **CUMULUS-3698**
+  - GranuleSearch retrieving files/execution is toggled
+      by setting "includeFullRecord" field to 'true' in relevant api endpoint params
+  - GranuleSearch does *not* retrieve files/execution by default unless includeFullRecord is set to 'true'
+  - @cumulus/db function getExecutionArnByGranuleCumulusId is removed. To replace this function use getExecutionInfoByGranuleCumulusId with parameter executionColumns set to ['arn'] or unset (['arn'] is the default argument)
+
+### Migration Notes
+
+#### CUMULUS-3833 Migration of ReconciliationReports from DynamoDB to Postgres after Cumulus is upgraded.
+
+To invoke the Lambda and start the ReconciliationReport migration, you can use the AWS Console or CLI:
+
+```bash
+aws lambda invoke --function-name $PREFIX-ReconciliationReportMigration $OUTFILE
+```
+
+- `PREFIX` is your Cumulus deployment prefix.
+- `OUTFILE` (**optional**) is the filepath where the Lambda output will be saved.
+
+
+#### CUMULUS-3967
+
+External tooling making use of `searchContext` in the `GET` `/granules/` endpoint will need to update to make use of standard pagination via `limit` and `page` scrolling, as `searchContext` is no longer supported/is an ES specific feature.
+
+### Replace ElasticSearch Phase 2
+
+- **CUMULUS-3967**
+  - Remove `searchContext` from API granules GET `/granules` endpoint.
+  - Update relevant tests to validate expected behavior utilizing postgres pagination
+- **CUMULUS-3229**
+  - Remove ElasticSearch queries from Rule LIST endpoint
+- **CUMULUS-3230**
+  - Remove ElasticSearch dependency from Rule Endpoints
+- **CUMULUS-3231**
+  - Updated API `pdrs` `LIST` endpoint to query postgres
+- **CUMULUS-3232**
+  - Update API PDR endpoints `DEL` and `GET` to not update Elasticsearch
+- **CUMULUS-3233**
+  - Updated `providers` list api endpoint and added `ProviderSearch` class to query postgres
+  - Removed Elasticsearch dependency from `providers` endpoints
+- **CUMULUS-3235**
+  - Updated `asyncOperations` api endpoint to query postgres
+- **CUMULUS-3236**
+  - Update API AsyncOperation endpoints `POST` and `DEL` to not update
+    Elasticsearch
+  - Update `@cumlus/api/ecs/async-operation` to not update Elasticsearch index when
+    reporting status of async operation
+- **CUMULUS-3698**
+  - GranuleSearch now can retrieve associated files for granules
+  - GranuleSearch now can retrieve latest associated execution for granules
+- **CUMULUS-3806**
+  - Update `@cumulus/db/search` to allow for ordered collation as a
+    dbQueryParameter
+  - Update `@cumulus/db/search` to allow `dbQueryParameters.limit` to be set to
+    `null` to allow for optional unlimited page sizes in search results
+  - Update/add type annotations/logic fixes to `@cumulus/api` reconciliation report code
+  - Annotation/typing fixes to `@cumulus/cmr-client`
+  - Typing fixes to `@cumulus/db`
+  - Re-enable Reconciliation Report integration tests
+  - Update `@cumulus/client/CMR.getToken` to throw if a non-launchpad token is requested without a username
+  - Update `Inventory` and `Granule Not Found` reports to query postgreSQL
+    database instead of elasticsearch
+  - Update `@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery` to
+    allow order by collation to be optionally specified
+  - Update `@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery` to
+    be parameterized and include a modifier on `temporalBoundByCreatedAt`
+  - Remove endpoint call to and all tests for Internal Reconciliation Reports
+    and updated API to throw an error if report is requested
+  - Update Orca reconciliation reports to pull granules for comparison from
+    postgres via `getGranulesByApiPropertiesQuery`
+- **CUMULUS-3837**
+  - Added `reconciliation_reports` table in RDS, including indexes
+  - Created pg model, types, and translation for `reconciliationReports` in `@cumulus/db`
+- **CUMULUS-3833**
+  - Created api types for `reconciliation_reports` in `@cumulus/types/api`
+  - Updated reconciliation reports lambda to write to new RDS table instead of Dynamo
+  - Updated `@cumulus/api/endpoints/reconciliation-reports` `getReport` and `deleteReport` to work with the new RDS table instead of Dynamo
+- **CUMULUS-3718**
+  - Updated `reconciliation_reports` list api endpoint and added `ReconciliationReportSearch` class to query postgres
+  - Added `reconciliationReports` type to stats endpoint, so `aggregate` query will work for reconciliation reports
+- **CUMULUS-3859**
+  - Updated `@cumulus/api/bin/serveUtils` to no longer add records to ElasticSearch
+  - Removed ElasticSearch from local API server code
+  - Updated CollectionSearch to filter granule fields in addition to time frame for active collections
+
 ## [Unreleased]
 
 ### Breaking Changes

audit-ci.json

@@ -3,12 +3,6 @@
   "pass-enoaudit": true,
   "retry-count": 20,
   "allowlist": [
-    {
-      "GHSA-776f-qx25-q3cc": {
-        "active": true,
-        "expiry": "1 July 2023 11:00"
-      }
-    },
     "jsonpath-plus",
     "semver"
   ]

example/cumulus-tf/variables.tf

@@ -350,7 +350,7 @@ variable "rds_admin_access_secret_arn" {
 variable "async_operation_image_version" {
   description = "docker image version to use for Cumulus async operations tasks"
   type = string
-  default = "52"
+  default = "53"
 }
 
 variable "cumulus_process_activity_version" {

example/spec/helpers/granuleUtils.js

@@ -238,8 +238,6 @@ const waitForGranuleRecordUpdatedInList = async (stackName, granule, additionalQ
       'beginningDateTime',
       'endingDateTime',
       'error',
-      'execution', // TODO remove after CUMULUS-3698
-      'files', // TODO -2714 this should be removed
       'lastUpdateDateTime',
       'productionDateTime',
       'updatedAt',
@@ -255,7 +253,8 @@ const waitForGranuleRecordUpdatedInList = async (stackName, granule, additionalQ
     });
     const results = JSON.parse(resp.body).results;
     if (results && results.length === 1) {
-      // TODO - CUMULUS-2714 key sort both files objects for comparison
+      results[0].files.sort((a, b) => a.cumulus_id - b.cumulus_id);
+      granule.files.sort((a, b) => a.cumulus_id - b.cumulus_id);
       const granuleMatches = isEqual(omit(results[0], fieldsIgnored), omit(granule, fieldsIgnored));
 
       if (!granuleMatches) {