Improve DGA/LotL doc's Installation Procedures (#10476)

* Update README.md

1 - Noted that one of the listed ingest pipeline methods should be used
2 - added the missing mapping that is required for the ML job card to appear as an option
3 - added optional step of creating a data view that is specific to the windows process data (rather than selecting `logs-*` which hits impertinent indices)
4 - noted that the ML job card can also be missing if the mapping is not in place

* remove non-functional field mappings

* add correct index name for elastic defend and remove windows agent reference

* add runtime mapping install for DGA

* change lotl instructions to use runtime mappings

* add changelog, bump manifest version

* update lotl docs

* small fixes

* add dga instructions

* fix wording

* finish network log sentence

* fix field mappings in dga

* add custom pipeline instructions

* formatting fixes

* formatting try 2

* formatting fix 3

* formatting fix 4

* fix typo

* fix tab indentation for numbering

* clarify UI vs instruction steps

* remove problemchild reference from dga

---------

Co-authored-by: Gus Carlock <[email protected]>

packages/dga/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.0.3"
+  changes:
+    - description: Add mapping instructions
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/10476
 - version: "2.0.2"
   changes:
     - description: Improve package installation documentation

packages/dga/docs/README.md

@@ -11,18 +11,75 @@ For more detailed information refer to the following blogs:
 1. **Upgrading**: If upgrading from a version below v2.0.0, see the section v2.0.0 and beyond.
 1. **Add the Integration Package**: Install the package via **Management > Integrations > Add Domain Generation Algorithm Detection**. Configure the integration name and agent policy. Click Save and Continue.
 1. **Install assets**: Install the assets by clicking **Settings > Install Domain Generation Algorithm Detection assets**.
-1. **Configure the ingest pipeline**: Once you’ve installed the package you can ingest your data using the ingest pipeline via the ingest pipeline. This will enrich your incoming data with its predictions from the machine learning model.
-    - If using an Elastic Beat such as Packetbeat, add the ingest pipeline to it by adding a simple configuration [setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#pipelines-for-beats) to `packetbeat.yml`.
-    - If adding the ingest pipeline to an existing pipeline, use a [pipeline processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline-processor.html). For example, you can check if Packetbeat (default index pattern `packetbeat-*`), Elastic Defend (`logs-endpoint.events.*`), or Elastic Agent (the default index pattern being `logs-endpoint.events.network-default`), already has an ingest pipeline by navigating to **Stack Management > Data > Index Management**, finding the index (sometimes you need to toggle "Include hidden indices"), and checking the index's settings for a default or final [pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#set-default-pipeline). 
-    - To enable the enrichment policy as the default pipeline on an index, you can use this example and replace `INDEX_NAME` with the desired index:
-    ```
-    POST INDEX_NAME/_settings
-    {
-      "index" : {
-        "default_pipeline" : "<VERSION>-ml_dga_ingest_pipeline"
+1. **Configure the pipeline**: To configure the pipeline you can use one of the following steps:
+    - If using Elastic Defend, add a custom pipeline to the data stream. Go to **Stack Management > Ingest Pipelines**, and check if the pipeline `logs-endpoint.events.network@custom` exists. 
+    ![Component Templates](../img/custom-pipeline.png)
+    If it does not exist, you can create it by running the following command in the Dev Console. Be sure to replace `<VERSION>` with the current package version.
+      ```
+      PUT _ingest/pipeline/logs-endpoint.events.network@custom
+      {
+        "processors": [
+          {
+            "pipeline": {
+              "name": "<VERSION>-ml_dga_ingest_pipeline",
+              "ignore_missing_pipeline": true,
+              "ignore_failure": true
+            }
+          }
+        ]
+      }
+      ```
+    - If `logs-endpoint.events.process@custom` already exists, select the three dots next to it and choose **Edit**. Click **Add a processor**. Select **Pipeline** for Processor, enter `<VERSION>-ml_dga_ingest_pipeline` for name (replacing `<VERSION>` with the current package version), and check **Ignore missing pipeline** and **Ignore failures for this processor**. Select **Add Processor**.
+    - If using an Elastic Beat such as Packetbeat, add the ingest pipeline to it by adding a simple configuration [setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#pipelines-for-beats) to `packetbeat.yml` and skip to the **Add preconfigured anomaly detection jobs** section in these instructions.
+1. **Add the required mappings to the index or component template**: Go to **Stack Management > Index Management > Component Templates**. Templates that can be edited to add custom components will be marked with a `@custom` suffix. For instance, the custom component template for Elastic Defend network events is `logs-endpoint.events.network@custom`. **Note:** Do not attempt to edit the `@package` template.
+    ![Component Templates](../img/component-templates.png)
+    - If the `@custom` component template does not exist, you can execute the following command in the Dev Console to create it and then continue to the **Rollover** section in these instructions.
+      ```
+      PUT _component_template/{COMPONENT_TEMPLATE_NAME}@custom
+      {
+        "template": {
+          "mappings": {
+            "properties": {
+              "ml_is_dga": {
+                "type": "object",
+                "properties": {
+                  "malicious_prediction": {
+                    "type": "long"
+                  },
+                  "malicious_probability": {
+                    "type": "float"
+                  }
+                }
+              }
+            }
+          }
+        }
       }
-    }
+      ```
+    - If the `@custom` component template already exists, you will need to edit it to add mappings for data to be properly enriched. Click the three dots next to it and select **Edit**. 
+    ![Component Templates](../img/component-templates-edit.png)
+    - On the index settings step, add the following. Be sure to change `<VERSION>` to the current package version.
+      ```
+      {
+        "index": {
+          "default_pipeline": "<VERSION>-ml_dga_ingest_pipeline"
+        }
+      }
+      ```
+    - Proceed to the mappings step in the UI. Click **Add Field** at the bottom of the page and create an an `Object` field for `ml_is_dga`. 
+    ![Component Templates](../img/field1.png)
+    - Finally create two properties under `ml_is_dga`.
+    ![Component Templates](../img/field1a.png)
+    - The first for `malicious_prediction` of type `Long` and then for `malicious_probability` or type `Float`.
+    ![Component Templates](../img/field2.png)
+    - Your component mappings should look like the following:
+    ![Component Templates](../img/fields-complete.png)
+    - Click **Review** then **Save Component Template**.
+1. **Rollover** Depending on your environment, you may need to [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html) in order for these mappings to get picked up. The deault index pattern for Elastic Defend is `logs-endpoint.events.network-default`.
+    ```
+    POST INDEX_NAME/_rollover
     ```
+1. **(Optional) [Create a data view](https://www.elastic.co/guide/en/kibana/current/data-views.html)** for your network logs.
 1. **Add preconfigured anomaly detection jobs**: In **Machine Learning > Anomaly Detection**, when you create a job, you should see an option to `Use preconfigured jobs` with a card for `DGA`. When you select the card, you will see a pre-configured anomaly detection job that you can enable depending on what makes the most sense for your environment. Note this job is only useful for indices that have been enriched by the ingest pipeline.
 1. **Enable detection rules**: You can also enable detection rules to alert on DGA activity in your environment, based on anomalies flagged by the above ML jobs. As of version 2.0.0 of this package, these rules are available as part of the Detection Engine in **Security > Rules**, and can be found using the tag `Use Case: Domain Generated Algorithm Detection`. See this [documentation](https://www.elastic.co/guide/en/security/current/prebuilt-rules-management.html#load-prebuilt-rules) for more information on importing and enabling the rules.
 

packages/dga/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 2.2.0
 name: dga
 title: "Domain Generation Algorithm Detection"
-version: 2.0.2
+version: 2.0.3
 source:
   license: "Elastic-2.0"
 description: "ML solution package to detect domain generation algorithm (DGA) activity in your network data."

packages/problemchild/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.1.4"
+  changes:
+    - description: Add mapping instructions
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/10476
 - version: "2.1.3"
   changes:
     - description: Improve package installation documentation

packages/problemchild/docs/README.md

@@ -12,19 +12,86 @@ For more detailed information refer to the following blogs and webinar:
 1. **Upgrading**: If upgrading from a version below v2.0.0, see the section v2.0.0 and beyond.
 1. **Add the Integration Package**: Install the package via **Management > Integrations > Add Living off the Land Detection**. Configure the integration name and agent policy. Click Save and Continue.
 1. **Install assets**: Install the assets by clicking **Settings > Install Living off the Land Detection assets**.
-1. **Configure the ingest pipeline**: Once you’ve installed the package you can ingest your data using the ingest pipeline via the ingest pipeline. This will enrich your incoming data with its predictions from the machine learning model.
+1. **Configure the pipeline**: To configure the pipeline you can use one of the following steps:
+    - If using Elastic Defend, add a custom pipeline to the data stream. Go to **Stack Management > Ingest Pipelines**, and check if the pipeline `logs-endpoint.events.process@custom` exists. 
+    ![Component Templates](../img/custom-pipeline.png)
+    If it does not exist, you can create it by running the following command in the Dev Console. Be sure to replace `<VERSION>` with the current package version.
+      ```
+      PUT _ingest/pipeline/logs-endpoint.events.process@custom
+      {
+        "processors": [
+          {
+            "pipeline": {
+              "name": "<VERSION>-problem_child_ingest_pipeline",
+              "ignore_missing_pipeline": true,
+              "ignore_failure": true
+            }
+          }
+        ]
+      }
+      ```
+    - If `logs-endpoint.events.process@custom` already exists, select the three dots next to it and choose **Edit**. Click **Add a processor**. Select **Pipeline** for Processor, enter `<VERSION>-problem_child_ingest_pipeline` for name (replacing `<VERSION>` with the current package version), and check **Ignore missing pipeline** and **Ignore failures for this processor**. Select **Add Processor**.
     - If using an Elastic Beat such as Winlogbeat, add the ingest pipeline to it by adding a simple configuration [setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#pipelines-for-beats) to `winlogbeat.yml`.
-    - If adding the ingest pipeline to an existing pipeline, use a [pipeline processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline-processor.html). For example, you can check if winlogbeat, default index pattern `winlogbeat-*`, or Elastic Defend, the default index pattern being `logs-endpoint*`, already has an ingest pipeline by navigating to **Stack Management > Data > Index Management**, finding the index (sometimes you need to toggle "Include hidden indices"), and checking the index's settings for a default or final [pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#set-default-pipeline).
-    - To enable the enrichment policy as the default pipeline on an index, you can use this example and replace `INDEX_NAME` with the desired index:
-    ```
-    POST INDEX_NAME/_settings
-    {
-      "index" : {
-        "default_pipeline" : "<VERSION>-problem_child_ingest_pipeline"
+1. **Add the required mappings to the component template**: Go to **Stack Management > Index Management > Component Templates**. Templates that can be edited to add custom components will be marked with a `@custom` suffix. For instance, the custom component template for Elastic Defend process events is `logs-endpoint.events.process@custom`. **Note:** Do not attempt to edit the `@package` template.
+    ![Component Templates](../img/component-templates.png)
+    - If the `@custom` component template does not exist, you can execute the following command in the Dev Console to create it and then continue to the **Rollover** section in these instructions. Be sure to change `<VERSION>` to the current package version.
+      ```
+      PUT _component_template/{COMPONENT_TEMPLATE_NAME}@custom
+      {
+        "template": {
+          "settings": {
+            "index": {
+              "default_pipeline": "<VERSION>-problem_child_ingest_pipeline"
+            }
+          },
+          "mappings": {
+            "properties": {
+              "blocklist_label": {
+                "type": "long"
+              },
+              "problemchild": {
+                "type": "object",
+                "properties": {
+                  "prediction": {
+                    "type": "long"
+                  },
+                  "prediction_probability": {
+                    "type": "float"
+                  }
+                }
+              }
+            }
+          }
+        }
       }
-    }
+      ```
+    - If the `@custom` component template already exists, you will need to edit it to add mappings for data to be properly enriched. Click the three dots next to it and select **Edit**. 
+    ![Component Templates](../img/component-templates-edit.png)
+    - On the index settings step, add the following. Be sure to change `<VERSION>` to the current package version.
+      ```
+      {
+        "index": {
+          "default_pipeline": "<VERSION>-problem_child_ingest_pipeline"
+        }
+      }
+      ```
+    - Proceed to the mappings step in the UI. Click **Add Field** at the bottom of the page and create a `blocklist_label` field of type `Long`:
+    ![Component Templates](../img/field1.png)
+    - Then create an `Object` field for `problemchild`. 
+    ![Component Templates](../img/field2.png)
+    - Finally create two properties under ProblemChild.
+    ![Component Templates](../img/field2a.png)
+    - The first for `prediction` of type `Long` and then for `prediction_probability` or type `Float`.
+    ![Component Templates](../img/field3.png)
+    - Your component mappings should look like the following:
+    ![Component Templates](../img/fields-complete.png)
+    - Click **Review** then **Save Component Template**.
+1. **Rollover** Depending on your environment, you may need to [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html) in order for these mappings to get picked up. The deault index pattern for Elastic Defend is `logs-endpoint.events.process-default`.
+    ```
+    POST INDEX_NAME/_rollover
     ```
-1. **Add preconfigured anomaly detection jobs**: In **Machine Learning > Anomaly Detection**, when you create a job, you should see an option to `Use preconfigured jobs` with a card for `Living off the Land Attack Detection`. When you select the card, you will see several pre-configured anomaly detection jobs that you can enable depending on what makes the most sense for your environment. **Warning**: if the ingest pipeline hasn't run for some reason, such as no eligible data has come in yet, _you won't be able to see this card yet_. If that is the case, try troubleshooting the ingest pipeline, and if any predictions have been populated yet.
+1. **(Optional) [Create a data view](https://www.elastic.co/guide/en/kibana/current/data-views.html) specificially for your windows process logs (index pattern or data stream name)**
+1. **Add preconfigured anomaly detection jobs**: In **Machine Learning > Anomaly Detection**, when you create a job, you should see an option to `Use preconfigured jobs` with a card for `Living off the Land Attack Detection`. When you select the card, you will see several pre-configured anomaly detection jobs that you can enable depending on what makes the most sense for your environment. **Warning**: if the ingest pipeline hasn't run for some reason, such as no eligible data has come in yet, or the required mapping has not been added, _you won't be able to see this card yet_. If that is the case, try troubleshooting the ingest pipeline, and if any predictions have been populated yet.
 1. **Enable detection rules**: You can also enable detection rules to alert on LotL activity in your environment, based on anomalies flagged by the above ML jobs. As of version 2.0.0 of this package, these rules are available as part of the Detection Engine, and can be found using the tag `Use Case: Living off the Land Attack Detection`. See this [documentation](https://www.elastic.co/guide/en/security/current/prebuilt-rules-management.html#load-prebuilt-rules) for more information on importing and enabling the rules.
 
 ![Domain Generation Detection Detection Rules](../img/lotlrules.png)
@@ -64,4 +131,4 @@ In version 2.1.1, the package ignores data in cold and frozen data tiers to redu
 
 ## Licensing
 
-Usage in production requires that you have a license key that permits use of machine learning features.
+Usage in production requires that you have a license key that permits use of machine learning features.

packages/problemchild/manifest.yml

@@ -1,7 +1,7 @@
 format_version: 3.0.0
 name: problemchild
 title: "Living off the Land Attack Detection"
-version: 2.1.3
+version: 2.1.4
 source:
   license: "Elastic-2.0"
 description: "ML solution package to detect Living off the Land (LotL) attacks in your environment. Requires a Platinum subscription."