UAT E2E Testing

[viviant100]

Once UAT is deployed to service-uat account, and UAT Ingestion #82 starts, UAT testing will begin.

[frankinspace]

@davidcolemanjpl @viviant100 See below for the initial test plan. Let me know if anything is unclear.

Collection short names in question:
SWOT_L2_HR_PIXCVec_2.0
SWOT_L2_HR_PIXC_2.0
SWOT_XDF_ORBIT_REV_FILE_2.0

High level steps for testing:

1. From swot-cumulus-sit, ingest (at a minimum) 8 SWOT_L2_HR_PIXCVec_2.0 and 8 SWOT_L2_HR_PIXC_2.0 and 1 SWOT_XDF_ORBIT_REV_FILE_2.0. The pixc and pixcvec files ingested should be files required to generate an L2 raster product.
2. Verify in service-uat that all files are ingested by the ingest to sds step function (work with @joshgarde if there are question about how to verify)
3. Using the service-uat swodlr UI, create a custom L2 raster product for the cycle/pass/scene associated with the pixc/pixcvec files ingested
4. Verify new raster product is available for download in podaac-swot-uat-swodlr-protected s3 bucket

[viviant100]

The test data will now use v2 collection:

• SWOT_L2_HR_PIXCVec_2.0
• SWOT_L2_HR_PIXC_2.0
• SWOT_XDF_ORBIT_REV_FILE_2.0

[viviant100]

@davidcolemanjpl Below are the user stories for SWODLR UI testing:

• (1) As a user of SWODLR, I want to be able to select SDP (Standard Data Product) with different configurations (e.g., projection, resolution, granule extent) to customize SWOT L2 Raster products.

• (2) As a user of SWODLR, I want the be able to choose a specific area on the map (i.e., draw a bounding box) with a defined time range so that I can customize SWOT L2 Raster products according to my specific geographical and temporal preferences.

• (3) As a user of SWODLR, I want to receive email notifications about the status of my raster creation request (e.g. new, in progress, ready to download, etc.).

• (4) As a user of SWODLR, I want to be able to select specific granules with the option of band and zone adjustment (i.e., +1/0/-1) associated with each granule so that I have more granular control over the customization of SWOT L2 Raster products.

[davidcolemanjpl]

SWODLR-UI (SIT) (v1.0 Pre-Alpha)
podaac-application-services-sit-3404 (service-swodlr-sit-raster-create)
gsfc-esdis-edc-podaac-app-swot-sit-1128 (s3 archive - podaac-swot-sit-swodlr-protected /L2_HR_Raster)

note: SWODLR (UAT) UI - currently non-functional today - ..."issues with data."

note: the User story (3) noted above for SWODLR UI testing is not currently implemented this way (e.g. new, in progress, ready to download, etc.) - Only Email status currently implemented = "Product Download Ready"
note: Unable to locate cycles/passes/scenes for SWOT_XDF_ORBIT_REV_FILE_2.0 collection (in CMR / EDSC)

Some SWODLR-UI Front-End issues observed:

• Limited combos (c/p/s) worked in SWODLR UI (for manually added scenes = 1/414/22-23 (note: unable to locate respective c/p/s in EDSC); Scenes retrieved from map = 7/72/13-15)
• Map in UI has sizing/resizing issues. The size of map should remain stable, in place. It should not randomly re-size (e.g. when navigating to tabs in navbar)
• Configure Options - all radio button do not work as expected when configuring SDPs (7/72/13-15) - User unable to toggle or select all of the row options for UTM Zone Adjust or MGRS Band Adjust
• Customization page in UI - user unable to delete Added Scenes. When refreshing the page after added scenes removed/deleted, the deleted scenes re-appear in SWODLR UI. (can use c/p/s combo 1/414/22)
• (Firefox and Chrome) 'My Data' page - table header has formatting issue - missing columns
• 'MyData' page in UI: user currently Unable to see or invoke the 'Unavailable' and 'Available' statuses. Suggest updating Generated Products Data statuses to be 'Queued', 'Running', 'FAIL', 'COMPLETE' (similar to the statuses listed in Cumulus UI)
• The Tutorial has issues (added to UI 1/18/24). E.g. User only able to access the Tutorial link from Navbar from the Customization page in swodlr-UI. User can NOT access the Tutorial link when using the 'About' window and/or 'MyData' Page. The 'Back' function does not work as expected for tutorial pages (12/18), (16/18). The tutorial crashes/exits. For tutorial page (17/18), try the 'Back' and then 'Next' to see the issue

Some SWODLR Back-End issues observed:

• False-positive raster-create executions. Generated Products Data shows ERROR status in SWODLR UI, but the associated raster-create execution succeeds w/ "SDS threw an Error" message. User also unable to efficiently trace the Generated Products Data created in UI to the raster create executions in AWS. User must search through False-positive raster executions in AWS to see which one actually had an issue
• AWS S3 archive bucket could be better sorted/organized with timestamps and raster products grouped together in Main bucket location (see S3 = podaac-swot-sit-swodlr-protected/L2_HR_Raster)
• (Performance issue) When Generating Products successfully in UI, the performance was very sluggish (using the raster-create workflow) – executions hung at the TimeoutRaster step function. The longest observed execution was approx. 21+ hrs in duration. Shortest execution was approx. 2+ hours

[davidcolemanjpl]

SWODLR-UI (UAT) (v1.0 Pre-Alpha): FAIL
podaac-app-services-uat-1858
https://swodlr.podaac.uat.earthdatacloud.nasa.gov/
(this issue is related to /issues/82)

Observed SWODLR-UI Front-End issues:

• When generating products using valid CPS combos, Generated Products Data statuses currently display as 'GENERATING' and 'ERROR' in the SWODLR (UAT) UI - My Data page
• User currently unable to locate and retrieve footprint scenes using Spatial Search Options on MAP in UI
• User never received an email (Product Download Ready) as expected after executions completed in UAT UI

SWODLR Back-End issues observed:

• service-swodlr-uat-raster-create executions fail with attribute errors ("module 'sys' has no attribute 'last_type'")
  E.g. see recent uat-raster-create executions ebf664ba-a785-4b6c-9249-b71bb8d94a37 or d234740d-5cb1-46bd-a2c7-e0bf424951b0

{
"errorMessage": "module 'sys' has no attribute 'last_type'",
"errorType": "AttributeError",
"requestId": "3129f281-0ad2-446a-8a86-79b1ffe71713",
"stackTrace": [
" File "/var/task/podaac/swodlr_common/decorators.py", line 158, in lambda_handler\n output_jobs = handler(input_jobs)\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 132, in default_bulk_job_handler\n output_jobs = [\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 133, in \n try_handler(input_job) for input_job in input_jobs\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 120, in try_handler\n sys.last_type, sys.last_value, sys.last_traceback # pylint: disable=no-member # noqa: E501\n"
]
}

• service-swodlr-uat-raster-create executions currently fails with error- "SDS threw an error" (false-positive successful executions):
  E.g. see uat-raster-create executions 9e4fe62f-7c05-4f5e-a381-4bc13e92de83 or 28da583a-1aac-41b3-a590-3090249feb07

  "job_id": "99159a67-87db-4fe9-93cf-c924777bc831",
  "job_status": "job-failed",
  "traceback": "Traceback (most recent call last):\n File "/home/ops/verdi/ops/hysds-1.2.12/hysds/job_worker.py", line 1412, in run_job\n post_processor_sigs.append(func(job, context))\n File "/home/ops/verdi/ops/hysds-1.2.12/hysds/triage.py", line 138, in triage\n prod_json = publish_dataset(triage_dir, ds_file, job, ctx)\n File "/home/ops/verdi/ops/hysds-1.2.12/hysds/dataset_ingest.py", line 214, in publish_dataset\n metrics, prod_json = ingest(\n File "/home/ops/verdi/ops/hysds-1.2.12/hysds/dataset_ingest.py", line 595, in ingest\n write_to_object_store(\n File "/home/ops/verdi/ops/hysds-1.2.12/hysds/dataset_ingest.py", line 366, in write_to_object_store\n osaka.main.put(abs_path, dest_url, params=params, noclobber=True)\n File "/home/ops/verdi/ops/osaka-1.2.3/osaka/main.py", line 38, in put\n transfer(\n File "/home/ops/verdi/ops/osaka-1.2.3/osaka/main.py", line 118, in transfer\n transfer.transfer(\n File "/home/ops/verdi/ops/osaka-1.2.3/osaka/transfer.py", line 128, in transfer\n raise err\n File "/home/ops/verdi/ops/osaka-1.2.3/osaka/transfer.py", line 87, in transfer\n raise osaka.utils.NoClobberException(error)\nosaka.utils.NoClobberException: Destination, s3://s3-us-west-2.amazonaws.com:80/swodlr-ops-triage-ondemand/2024/01/31/triaged_job-job-SCIFLO_L2_HR_Raster__pcm-v5.0.0-pge-v5.0.4-20240131T164658.129029Z_task-92755568-9dea-47ab-88de-39a4d6549924/triaged_job-job-SCIFLO_L2_HR_Raster__pcm-v5.0.0-pge-v5.0.4-20240131T164658.129029Z_task-92755568-9dea-47ab-88de-39a4d6549924.dataset.json, already exists and no-clobber is set\n",
  "errors": [
  "SDS threw an error"
  ]
  }

[davidcolemanjpl]

SWODLR UI (UAT) –
podaac-app-services-uat-1858

The AWS raster create workflow executions are still not completing as expected.

note: I'm also still unable to currently locate footprint scenes on the map in SWODLR (UAT) UI (using Spatial Search Options)

see raster create workflow execution aa0cf897-0873-4aa7-a6d7-1ca7db01eecf:
{
"errorMessage": "module 'sys' has no attribute 'last_type'",
"errorType": "AttributeError",
"requestId": "f6807fad-05a1-43cc-b436-09111f44b2c0",
"stackTrace": [
" File "/var/task/podaac/swodlr_common/decorators.py", line 158, in lambda_handler\n output_jobs = handler(input_jobs)\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 132, in default_bulk_job_handler\n output_jobs = [\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 133, in \n try_handler(input_job) for input_job in input_jobs\n",
" File "/var/task/podaac/swodlr_common/decorators.py", line 120, in try_handler\n sys.last_type, sys.last_value, sys.last_traceback # pylint: disable=no-member # noqa: E501\n"
]
}

False-positive - successful raster create execution - RuntimeError:
see raster create workflow execution 5c19c558-857d-4184-8f05-c66e760ff42f:

  "job_status": "job-failed",
  "traceback": "Traceback (most recent call last):\n  File \"/home/ops/verdi/ops/hysds-1.2.12/hysds/job_worker.py\", line 1237, in run_job\n    raise RuntimeError(\"Got non-zero exit code: {}\".format(status))\nRuntimeError: Got non-zero exit code: 255\n",
  "errors": [
    "SDS threw an error"
  ]

This issue was observed while reviewing / testing: (podaac/swodlr-ui#72)

note: No test data (SIT/UAT) for the SWOT_XDF_ORBIT_REV_FILE_2.0 collection

[davidcolemanjpl]

SWODLR UI (UAT) – Re-TEST: (v1.0.0-rc.8 Pre-Alpha)
SWODLR API: v0.0.1-alpha48
podaac-app-services-uat-1858
podaac-app-swot-uat-3235 (s3)

• Certain service-swodlr-uat-raster-create executions show error ("SDS threw an error")  – note: this type of SDS error is OK (Per DEV) - (see recent service-swodlr-uat-raster-create executions: 7485b8b1-2f84-4f37-9c0f-cbc00253e34a
  e596e504-d3ce-4d8f-b39c-bb378eaada4d)

• User currently unable to download generated products via SWODLR UI (UAT); 404 (PAGE NOT FOUND) error displays after selecting the download icon (Download URL column (MyData page)) - (see attachment below)

• Download Ready email - User currently unable to access or download product from SWODLR UAT Download Ready email

UPDATE: 02/26/24:
Issue still happens today:
Certain service-swodlr-uat-raster-create executions show error ("SDS threw an error") – note: this type of SDS error is OK (Per DEV) - (see recent successful (false-positive) service-swodlr-uat-raster-creater executions - 114774e6-ff49-4a36-be97-c109aef51532 or c0342ae4-2439-4e71-8826-03f9efa14b90 )

Issue still happens today:
User currently unable to download generated products via SWODLR UI (UAT); 404 (PAGE NOT FOUND) error displays after selecting the download icon (Download URL column (MyData page))

OKAY / PASS today:
Download Ready email - User currently able to download product as expected from the SWODLR UAT Download Ready email (using user's SIT EDL credentials)

[davidcolemanjpl]

Re-TEST:
SWODLR UI (UAT) (v1.0.0-rc.8 Pre-Alpha)
SWODLR API: v0.0.1-alpha48
podaac-app-services-uat-1858
podaac-app-swot-uat-3235 (s3)

SWODLR Front-End issues:

• User currently unable to download generated products via SWODLR UI; 404 (Page Not Found) error displays after selecting the download icon ('My Data' page, 'Download URL' column)
• User must zoom in on Map in UI to properly locate footprint scenes (locating scenes via MAP polygon intermittent ('Customization' page))

SWODLR Back-end: OKAY

• service-swodlr-uat-raster-create executions currently seem to be Succeeding as expected when generating raster products

[davidcolemanjpl]

Re-TEST:
SWODLR UI (UAT) (v1.0.0-rc.8 Pre-Alpha)
SWODLR API: v0.0.1-alpha48
podaac-app-services-uat-1858
podaac-app-swot-uat-3235 (s3)

SWODLR Front-End issues:
Okay now/PASS - User currently unable to download generated products via SWODLR UI; 404 (Page Not Found) error displays after selecting the download icon ('My Data' page, 'Download URL' column)

Still an issue - User must zoom in on Map in UI to properly locate footprint scenes (locating scenes via MAP polygon intermittent ('Customization' page))