Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CMR-10257: [Fix] Updating API Docs to simplify language #2195

Merged
merged 5 commits into from
Nov 26, 2024
Merged

CMR-10257: [Fix] Updating API Docs to simplify language #2195

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d50a3cb
update api docs fo ingest app
jmaeng72 Nov 20, 2024
327e682
update search api
jmaeng72 Nov 20, 2024
247205c
rename subscriptions and reword some phrases
jmaeng72 Nov 21, 2024
a12101f
update search and ingest to nrt and batch notification
jmaeng72 Nov 21, 2024
1e9727a
re-order sub types desc
jmaeng72 Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 104 additions & 42 deletions ingest-app/docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1081,67 +1081,114 @@ Tool metadata can be deleted by sending an HTTP DELETE to the URL `%CMR-ENDPOINT
#### Successful Response in JSON

{"concept-id":"TL1200000015-PROV1","revision-id":2}
## <a name="subscription"></a> Subscription
### <a name="create-subscription"></a> Create a Subscription
#### <a name="subscription-endpoint"></a> /subscriptions

NOTE: The `%CMR-ENDPOINT%/providers/<provider-id>/subscriptions` API routes for subscription ingest are deprecated. Please switch to the new `%CMR-ENDPOINT%/subscriptions` API routes. All the examples below are using the new routes.

Subscription allows a user to register some query conditions in the CMR and be notified when collections/granules matching the conditions are created, updated, or deleted in the CMR. By the end of 2024 there are two kinds of subscriptions. Those that are run by search queries and those that are processed on ingest. The ingest subscriptions are only for granules and notifications are sent out to a provided AWS SQS endpoint. The search subscriptions are run periodically using CMR search and notifiy users by email using the email address that is registered in Earthdata Login. For the search subscriptions, there are two types of subscriptions (identified by the `Type` field of the subscription):

- collection subscription for users to be notified when collections are created/updated.
- granule subscription for users to be notified when granules are created/updated.
## <a name="subscription"></a> Subscriptions
A subscription allows a user to be notified when specific collections/granules are created, updated, or deleted. The collections/granules specified can be filtered by query conditions.

Subscription metadata is in JSON format and conforms to [UMM-Sub Schema](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/subscription). There is a background job that processes the search subscriptions periodically (configurable), to see if there are any collections/granules that are created/updated since the last time the subscription has been processed and notify the subscription user with any matches.
johnwteague marked this conversation as resolved.
Show resolved Hide resolved
johnwteague marked this conversation as resolved.
Show resolved Hide resolved
There are two kinds of subscriptions: Search and Ingest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than describe the subscription types as "search" and "ingest" we should give a more user-facing description. Something like: "batch notification" and "near-real-time notification".


Subscription concepts can be created by sending an HTTP POST or PUT with the metadata sent as data to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`. The response will include the [concept id](#concept-id) ,the [revision id](#revision-id), and a [native-id](#native-id).
- Ingest subscriptions are processed on ingest and are only for granules. When a user subscribes, notifications are sent out to a provided AWS SQS endpoint.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sent out via the provided notification endpoint, such as an AWS SQS messaging queue.


`Type` is a required field in subscription request body. The valid values of `Type` are: `"collection"` or `"granule"`. It indicates if the subscription is a collection subscription or granule subscription. Subscriptions of type granule must supply a requisite CollectionConceptId, and subscriptions of type collection cannot have a CollectionConceptId field. When the CollectionConceptId field is used, the subscription subscriber must have read access to the collection, otherwise the ingest of the subscription will fail.
- Search subscriptions run periodically via CMR search queries and notify users by the email address that is registered in Earthdata Login. There are two types of subscriptions (identified by the `Type` field of the subscription):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batch subscription processing is run periodically and execute CMR search queries, as defined by the Subscription definition, and notify users ...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...two types of batch process subscriptions...


If a native-id is not provided it will be generated. This is only supported for POST requests.
POST requests may only be used for creating subscriptions.
- collection subscription for users to be notified when collections are created/updated.
- granule subscription for users to be notified when granules are created/updated.

If a SubscriberId is not provided, then the user ID associated with the token used to ingest the subscription will be used as the SubscriberId.
There is a configurable background job that processes the search subscriptions periodically, to see if there are any collections/granules that are created/updated since the last time the subscription has been processed and notify the subscription user with any matches.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batch subscription notification processing is executed periodically...
Notification of updates is via the email address associated with the SubscriberId's EarthData Login (URS).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do see that the parameter description section describes this but if feels like a good thing to mention as an overview also, since that is the notification mechanism.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this is not configurable by the client, so no need to include it in the API docs


EmailAddress was previously a required field, but this field is now deprecated. Instead, the email address associated with the SubscriberId's EarthData Login (URS) account will be used as the EmailAddress. If an EmailAddress is specified at subscription creation it will be ignored.

For ingest subscriptions to be used there are three new fields that are requird in addition to the other fields already described. First though, the Type field must be set to `granule` as ingest subscriptions are only for granules. Because this is a granule subscription the CollectionConceptId must be set. The new fields are:
### <a name="create-subscription"></a> Create a Subscription
#### <a name="subscription-endpoint"></a> /subscriptions

Method is a new field and it designates between search subscriptions and ingest subscriptions. Thus the valid values for this field are either `search` or `ingest`. If this field is not used, then search is assumed.
NOTE: The `%CMR-ENDPOINT%/providers/<provider-id>/subscriptions` API routes for subscription ingest are deprecated. Please switch to the new `%CMR-ENDPOINT%/subscriptions` API routes. All the examples below are using the new routes.

EndPoint describes to where notifications get sent. At this time only Amazon Web Services (AWS) Simple Query Service (SQS) Amazon Resource Names (ARN) are allowed. Ingest of subscriptions that do not use an AWS SQS ARN will fail. If search subscriptions are desired, do not use this field.
##### Method Options
Subscription concepts can be created by sending an HTTP POST or PUT with the metadata sent as data to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`.

Mode describes whether the notification is for New (ingested for the first time into the CMR) granules, Updated granules, or Deleted granules. The valid values are New, Update, Delete. Any combination of these values are valid and they are set using a json array. Here are some examples:
[`New`]
[`New`, `Update`]
[`Update`, `Delete`]
[`Update`]
The response will include the [concept id](#concept-id) ,the [revision id](#revision-id), and a [native-id](#native-id).

POST only may be used without a native-id at the following URL.
`%CMR-ENDPOINT%/subscriptions`
- POST or PUT may be used with the following URL: `%CMR-ENDPOINT%/subscriptions/<native-id>`
- When using POST:
- May only be used for creating subscriptions
- Can use without a native-id at the following URL: `%CMR-ENDPOINT%/subscriptions`
- The native-id will be auto-generated

POST or PUT may be used with the following URL.
`%CMR-ENDPOINT%/subscriptions/<native-id>`
##### Query Params for Search Subscriptions

Query values are currently only for search subscriptions and should not be URL encoded. Instead, the query should consist of standard granule search parameters, separated by '&'. For example, a valid query string might look like:

instrument=MODIS&sensor=1B&polygon=-18,-78,-13,-74,-16,-73,-22,-77,-18,-78

If the query provided is invalid for granule searching, subscription creation will fail with HTTP status response of 400, and an error message detailing which query parameters were invalid.

### <a name="update-subscription"></a> Update a Subscription
#### <a name="subscription-native-id-endpoint"></a> /subscriptions/&lt;native-id&gt;
Subscription concept can be updated by sending an HTTP POST or PUT with the metadata sent as data to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`. The response will include the [concept id](#concept-id) and the [revision id](#revision-id).

If a native-id is provided in a POST, and a subscription already exists for that provider with the given native-id, the request will be rejected.
##### Data Fields
The metadata sent in the body of the request is in JSON format and conforms to [UMM-Sub Schema](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/subscription). This metadata data body consists of specific fields below:

- `Name`: [required] Name of subscription
- `Type`: [required] Indicates if the subscription is a collection subscription or granule subscription.
- Valid values: `"collection"`,`"granule"`.
- "Type": "granule": must supply a requisite CollectionConceptId field. The subscriber must have read access to the collection here, or the subscription will fail.
- "Type": "collection": cannot have a CollectionConceptId field.
- `Method`: [required] designates between search subscriptions and ingest subscriptions
- Valid values: `search` or `ingest`.
- Default: `search`
- `SubscriberId`: [optional] If a SubscriberId is not provided, then the user ID associated with the token used to ingest the subscription will be used as the SubscriberId.
- `EmailAddress`: [deprecated] was previously a required field, but is now deprecated. Instead, the email address associated with the SubscriberId's EarthData Login (URS) account will be used as the EmailAddress. If an EmailAddress is specified at subscription creation it will be ignored.
- `MetadataSpecification`: [required] Specifies which metadata version schema you are using for this subscription. Currently, that version is 1.1.1.

##### Additional Data Fields for Ingest Subscriptions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...for Near Real-time Subscriptions

Basically, rename "Ingest subscription" everywhere to NRT or whatever we land on.

For ingest subscriptions to be used there are three new fields that are required in addition to the other fields already described.
- `Type`: [required] Must be set to `granule` because ingest subscriptions are only for granules.
- `CollectionConceptId`: [required] Because type must be `granule`, we must set `CollectionConceptId` as well, as indicated in Data Fields section
- `EndPoint`: describes where notifications get sent. At this time only AWS SQS ARN's are allowed. Ingest subscriptions that do not use an AWS SQS ARN will fail. If search subscriptions are desired, do not use this field.
- `Mode`: describes whether the notification is for New (ingested for the first time into the CMR) granules, Updated granules, or Deleted granules.
- Valid values: `New`, `Update`, `Delete`. Any combination of these values are valid and they are set using a json array.
- Examples:
[`New`] or
[`New`, `Update`] or
[`Update`, `Delete`] or
[`Update`]

##### Ingest Subscription POST Request

```
curl --request POST '%CMR-ENDPOINT%/ingest/subscriptions/my-native-id-of-my-subscription' \
--header 'Content-Type: application/vnd.nasa.cmr.umm+json' \
--header 'Authorization: ••••••' \
--data-raw '{
"Name": "Ingest-Subscription-Test",
"Type": "granule",
"SubscriberId": "My_sub_id",
"EmailAddress": "[email protected]",
"CollectionConceptId": "MyCollectionID",
"EndPoint": "MyAWSSQSARN",
"Mode": ["New","Update"],
"Method":"ingest",
"MetadataSpecification": {
"URL": "https://cdn.earthdata.nasa.gov/umm/subscription/v1.1.1",
"Name": "UMM-Sub",
"Version": "1.1.1"
}
}
'
```

PUT requests should be used for updating subscriptions. Creation of subscriptions using PUT may be deprecated in the future. All PUT requests require a native-id to be part of the request URL.
Following are two examples that show how to update and create a search subscription. The third example shows how to create an ingest subscription using PUT.
##### Ingest Subscription Successful Response in XML
```
curl -XPUT \
<?xml version="1.0" encoding="UTF-8"?>
<result>
<concept-id>SUB12345678-MyCollectionID</concept-id>
<native-id>my-native-id-of-my-subscription</native-id>
<revision-id>1</revision-id>
</result>
```

##### Create search subscription with %CMR-ENDPOINT%/subscriptions URL (native-id will be autogenerated)
```
curl -XPOST \
-H "Content-type: application/vnd.nasa.cmr.umm+json" \
-H "Authorization: XXXX" \
%CMR-ENDPOINT%/subscriptions/subscription123 \
%CMR-ENDPOINT%/subscriptions \
-d \
"{\"Name\": \"someSubscription\",
\"Type\": \"granule\",
Expand All @@ -1150,11 +1197,23 @@ curl -XPUT \
\"Query\": \"polygon=-18,-78,-13,-74,-16,-73,-22,-77,-18,-78\"}"
```


### <a name="update-subscription"></a> Update a Subscription
#### <a name="subscription-native-id-endpoint"></a> /subscriptions/&lt;native-id&gt;
Subscription concept can be updated by sending an HTTP POST or PUT request with the metadata sent as data to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`.

The response will include the [concept id](#concept-id) ,the [revision id](#revision-id), and a [native-id](#native-id).

If a native-id is provided in a POST, and a subscription already exists for that provider with the given native-id, the request will be rejected.

PUT requests should be used for updating subscriptions. Creating subscriptions with PUT may be deprecated in the future. All PUT requests require a native-id to be part of the request URL.

##### Create search subscription with %CMR-ENDPOINT%/subscriptions/<native-id> URL
```
curl -XPOST \
curl -XPUT \
-H "Content-type: application/vnd.nasa.cmr.umm+json" \
-H "Authorization: XXXX" \
%CMR-ENDPOINT%/subscriptions \
%CMR-ENDPOINT%/subscriptions/subscription123 \
-d \
"{\"Name\": \"someSubscription\",
\"Type\": \"granule\",
Expand All @@ -1163,6 +1222,7 @@ curl -XPOST \
\"Query\": \"polygon=-18,-78,-13,-74,-16,-73,-22,-77,-18,-78\"}"
```

##### Update ingest subscription with PUT
```
curl -XPUT \
-H "Content-type: application/vnd.nasa.cmr.umm+json" \
Expand All @@ -1188,7 +1248,7 @@ curl -XPUT \
<?xml version="1.0" encoding="UTF-8"?>
<result>
<concept-id>SUB1200000015-PROV1</concept-id>
<revision-id>1</revision-id>
<revision-id>2</revision-id>
<native-id>subscription123</native-id>
</result>
```
Expand All @@ -1197,11 +1257,13 @@ curl -XPUT \
By passing the option `-H "Accept: application/json"` to `curl`, one may
get a JSON response:

{"concept-id":"SUB1200000015-PROV1","revision-id":1,"native-id":"subscription123"}
{"concept-id":"SUB1200000015-PROV1","revision-id":2,"native-id":"subscription123"}

### <a name="delete-subscription"></a> Delete a Subscription

Subscription metadata can be deleted by sending an HTTP DELETE to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`. The response will include the [concept id](#concept-id) and the [revision id](#revision-id) of the tombstone.
Subscription metadata can be deleted by sending an HTTP DELETE to the URL `%CMR-ENDPOINT%/subscriptions/<native-id>`.

The response will include the [concept id](#concept-id) and the [revision id](#revision-id) of the tombstone.

```
curl -XDELETE \
Expand Down
14 changes: 10 additions & 4 deletions search-app/docs/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5122,12 +5122,18 @@ On occasions when tool dissociation cannot be processed at all due to invalid in

### <a name="subscription"></a> Subscription

Subscription allows a user to register some query conditions in the CMR and be notified when collections/granules matching the conditions are created, updated, or deleted in the CMR. By the end of 2024 there are two kinds of subscriptions. Those that are run by search queries and those that are processed on ingest. The ingest subscriptions are only for granules and notifications are sent out to a provided AWS SQS endpoint. The search subscriptions are run periodically using CMR search and notifiy users by email using the email address that is registered in Earthdata Login. For the search subscriptions, there are two types of subscriptions (identified by the `Type` field of the subscription):
A subscription allows a user to be notified when specific collections/granules are created, updated, or deleted. The collections/granules specified can be filtered by query conditions.

- collection subscription for users to be notified when collections are created/updated.
- granule subscription for users to be notified when granules are created/updated.
There are two kinds of subscriptions: Search and Ingest

Subscription metadata is in JSON format and conforms to [UMM-Sub Schema](https://git.earthdata.nasa.gov/projects/EMFD/repos/unified-metadata-model/browse/subscription). Subscriptions of type `granule` must supply a requisite CollectionConceptId, and subscriptions of type `collection` cannot have a CollectionConceptId field. There is a background job that processes the search subscriptions periodically (configurable), to see if there are any collections/granules that are created/updated since the last time the subscription has been processed and notify the subscription user with any matches.
- Ingest subscriptions are processed on ingest and are only for granules. When a user subscribes, notifications are sent out to a provided AWS SQS endpoint.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar comments for the search doc, replace "ingest subscriptions" and "search subscriptions"


- Search subscriptions run periodically via CMR search queries and notify users by the email address that is registered in Earthdata Login. There are two types of subscriptions (identified by the `Type` field of the subscription):

- collection subscription for users to be notified when collections are created/updated.
- granule subscription for users to be notified when granules are created/updated.

There is a configurable background job that processes the search subscriptions periodically, to see if there are any collections/granules that are created/updated since the last time the subscription has been processed and notify the subscription user with any matches.

#### <a name="searching-for-subscriptions"></a> Searching for Subscriptions

Expand Down