[ti_anomali] Support the ThreatStream API (#11309)

A new data stream is added that uses the CEL input to fetch data from
the Anomali ThreatStream API's intelligence endpoint.

This new data stream should be preferred over the existing data stream
for new users, because the existing data stream relies on the
additional Elastic Extension software, which sends data to the
integration via the HTTP Endpoint input.

The two data sources are roughly equivalent, although there are
significant differences in the data format and some other details.

Support for fetching remote indicators is added as an advanced option.
Using this on the test instance shows no differences in results, but
it may be useful for on-prem users.

The `date_first` and `date_last` (indicator first and last seen) fields
are not available from the API endpoint and the time fields differ.

Variables and fields related to the IOC expiry transform have been
changed to be more clear.

The old pipeline handles a smaller number of `itype` values. The new
pipeline extends this to list all documented values, but they remain
unhandled in some cases where the mapping to a STIX indicator type is
unclear. All indicator types for which example data exists have been
covered in pipeline tests.

For fields that aren't mapped to ECS, the new data stream uses field
names that match the API response rather than matching them to the
legacy field names.

There was some system test setup and dashboard filtering for a
datastream removed earlier (#4220), and that is now removed.

Dashboard filters were updated and deprecated controls were switched to
current equivalents. Kibana tags were removed. Screenshots have been
added.

packages/ti_anomali/_dev/build/docs/README.md

@@ -1,42 +1,53 @@
 # Anomali Integration
 
-The Anomali integration supports the following datasets.
+The Anomali integration can fetch indicators from [Anomali ThreatStream](https://www.anomali.com/products/threatstream), a commercial Threat Intelligence service.
 
-- `threatstream` dataset: Support for [Anomali ThreatStream](https://www.anomali.com/products/threatstream), a commercial Threat Intelligence service.
+It has the following data streams:
 
-## Logs
-
-### Anomali Threatstream
+- **`intelligence`** Indicators retrieved from the Anomali ThreatStream API's intelligence endpoint.
+- **`threatstream`** Indicators received from the Anomali ThreatStream Elastic Extension, which is additional software. This is deprecated.
 
-This integration requires additional software, the _Elastic_ _Extension,_
-to connect the Anomali ThreatStream with this integration. It's available
-at the [ThreatStream download page.](https://ui.threatstream.com/downloads)
-
-Please refer to the documentation included with the Extension for a detailed
-explanation on how to configure the Anomali ThreatStream to send indicator
-to this integration.
+## Logs
 
 ### Expiration of Indicators of Compromise (IOCs)
-The ingested IOCs expire after certain duration. An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to faciliate only active IOCs be available to the end users. This transform creates a destination index named `logs-ti_anomali_latest.threatstream-2` which only contains active and unexpired IOCs. The destination index also has an alias `logs-ti_anomali_latest.threatstream`. When setting up indicator match rules, use this latest destination index to avoid false positives from expired IOCs. Please read [ILM Policy](#ilm-policy) below which is added to avoid unbounded growth on source `.ds-logs-ti_anomali.threatstream-*` indices.
+
+An [Elastic Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) is created to provide a view of active indicators for end users. The transform creates destination indices that are accessible via the alias of the form `logs-ti_anomali_latest.<datastreamname>`. When querying for active indicators or setting up indicator match rules, use the alias to avoid false positives from expired indicators. The dashboards show only the latest indicators.
 
 #### Handling Orphaned IOCs
-When an IOC expires, Anomali feed contains information about all IOCs that got `deleted`. However, some Anomali IOCs may never expire and will continue to stay in the latest destination index `logs-ti_anomali_latest.threatstream`. To avoid any false positives from such orphaned IOCs, users are allowed to configure `IOC Expiration Duration` parameter while setting up the integration. This parameter deletes all data inside the destination index `logs-ti_anomali_latest.threatstream` after this specified duration is reached. Users must pull entire feed instead of incremental feed when this expiration happens so that the IOCs get reset. 
 
-**NOTE:** `IOC Expiration Duration` parameter does not override the expiration provided by the Anomali for their IOCs. So, if Anomali IOC is expired and subsequently such `deleted` IOCs are sent into the feed, they are deleted immediately. `IOC Expiration Duration` parameter only exists to add a fail-safe default expiration in case Anomali IOCs never expire.
+Indicator data from Anomali can contain information about deletion or expiry times. However, some Anomali IOCs may never expire and will continue to stay in the latest destination index. To avoid any false positives from such orphaned IOCs, users are allowed to configure an "IOC Expiration Duration" or "IOC Duration Before Deletion" parameter while setting up a policy. The value set there will limit the time that indicators are retained before deletion, but indicators may be removed earlier based on information from Anomali.
 
 ### Destination index versioning and deleting older versions
-The destination indices created by the transform are versioned with an integer suffix such as `-1`, `-2`. Example index name - `logs-ti_anomali_latest.threatstream-1`. 
 
-Due to schema changes on destination index, the versioning on it could be bumped. For example, in integration version `1.15.1`, the destination index  is changed to `logs-ti_anomali_latest.threatstream-2` from `logs-ti_anomali_latest.threatstream-1`. 
+The destination indices created by the transform are versioned with an integer suffix such as `-1`, `-2`, for example, `logs-ti_anomali_latest.intelligence-1`.
+
+Due to schema changes in the destination index, its version number may be incremented.
+
+When this happens, the transform does not have the functionality to auto-delete the old index, so users must delete this old index manually. This is to ensure that duplicates are not present when using wildcard queries such as `logs-ti_anomali_latest.intelligence-*`. To delete an old index, follow the steps below (either for `intelligence` as below, or for the older `threatstream` equivalents):
+
+1. After upgrading the integration to the latest version, check the current transform's destination index version by navigating to: `Stack Management -> Transforms -> logs-ti_anomali.latest_intelligence-default -> Details`. Check the `destination_index` value.
+2. Run `GET _cat/indices?v` and check if any older versions exist. Such as `logs-ti_anomali_latest.intelligence-1`
+3. Run `DELETE logs-ti_anomali_latest.intelligence-<OLDVERSION>` to delete the old index.
+
+### ILM Policies
+
+To prevent unbounded growth of the source data streams `logs-ti_opencti.<datastreamname>-*`, index lifecycle management (ILM) policies will deletes records 5 days after ingestion.
+
+### Anomali ThreatStream API
+
+The Anomali ThreatStream API's intelligence endpoint is the preferred source of indicators. This data will be be accessible using the alias `logs-ti_anomali_latest.intelligence`.
+
+{{event "intelligence"}}
+
+{{fields "intelligence"}}
+
+### Anomali ThreatStream via the Elastic Extension
 
-Since the transform does not have the functionality to auto-delete the old index, users must to delete this old index manually. This is to ensure duplicates are not present when using wildcard queries such as `logs-ti_anomali_latest.threatstream-*`. Please follow below steps:
-1. After upgrading the integration to latest, check the current transform's destination index version by navigating via: `Stack Management -> Transforms -> logs-ti_anomali.latest_ioc-default -> Details`. Check `destination_index` value.
-2. Run `GET _cat/indices?v` and check if any older versions exist. Such as `logs-ti_anomali_latest.threatstream-1`
-3. Run `DELETE logs-ti_anomali_latest.threatstream-<OLDVERSION>` to delete the old index.
+This source of indicators is deprecated. New users should instead use the API source above. This source requires additional software, the _Elastic_ _Extension,_ to connect Anomali ThreatStream to this integration. It's available on the [ThreatStream download page](https://ui.threatstream.com/downloads).
 
-### ILM Policy
-To facilitate IOC expiration, source datastream-backed indices `.ds-logs-ti_anomali.threat-*` are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn't lead to unbounded growth. This means data in these source indices will be deleted after `5 days` from ingested date. 
+Please refer to the documentation included with the extension for a detailed explanation on how to configure Anomali ThreatStream to send indicators to this integration.
 
+Indicators ingested in this way will become accessible using the alias `logs-ti_anomali_latest.threatstream`.
 
 {{event "threatstream"}}
 

packages/ti_anomali/_dev/deploy/docker/docker-compose.yml

@@ -1,6 +1,6 @@
 version: '2.3'
 services:
-  limo-http:
+  intelligence-api:
     image: docker.elastic.co/observability/stream:v0.15.0
     ports:
       - 8080
@@ -11,7 +11,7 @@ services:
     command:
       - http-server
       - --addr=:8080
-      - --config=/files/config.yml
+      - --config=/files/intelligence-api-config.yml
   threatstream-webhook-http:
     image: docker.elastic.co/observability/stream:v0.15.0
     volumes:
@@ -32,6 +32,6 @@ services:
   threatstream-integrator-test:
     image: docker.io/adrisr/filebeat-anomali-integrator-test:latest
     volumes:
-      - ./files:/files:ro
+      - ./sample_logs:/sample_logs:ro
     command:
-      - /files/test-intel.ndjson
+      - /sample_logs/test-intel.ndjson