[symantec_endpoint_security] Merge Symantec EDR Cloud into Symantec E…

…ndpoint Security (#10530)

* Add incident data-stream of Symantec EDR Cloud into Symantec Endpoint
  Security.
* Update Table of Contents of all dashboards.

packages/symantec_endpoint_security/_dev/build/docs/README.md

@@ -1,16 +1,12 @@
 # Symantec Endpoint Security
 
-This Symantec Endpoint Security integration enables your security team to securely stream event data to Elastic Security, via AWS S3, AWS SQS or GCS. When integrated with Elastic Security, this valuable data can be leveraged within Elastic.
-Symantec Endpoint Security (SES) delivers comprehensive protection for all your traditional and mobile devices across the entire attack chain. Symantec endpoint innovations include behavioral isolation, Active Directory security, and Threat Hunter technologies to protect your endpoints against sophisticated threats and targeted attacks.
+Symantec Endpoint Security (SES), is fully cloud-managed version of the on-premises Symantec Endpoint Protection (SEP), which delivers multilayer protection to stop threats regardless of how they attack your endpoints. You manage SES through a unified cloud console that provides threat visibility across your endpoints and uses multiple technologies to manage the security of your organization.
 
-The Symantec Endpoint Security integration can be used in three different modes to collect data:
-- AWS S3 polling mode: Symantec Endpoint Security writes data to S3, and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
-- AWS S3 SQS mode: Symantec Endpoint Security writes data to S3, S3 sends a notification of a new object to SQS, the Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple agents can be used in this mode.
-- GCS polling mode: Symantec Endpoint Security writes data to GCS bucket, and Elastic Agent polls the GCS bucket by listing its contents and reading new files.
+This SES Integration enables user to stream Events and EDR incidents data to Elastic, via Data Storage(AWS S3, AWS SQS or GCS) and API endpoint respectively.
 
 ## Data streams
 
-The Symantec Endpoint Security integration collects logs for different events that Integrated Cyber Defense Schema organizes into following categories:
+The Symantec Endpoint Security integration collects logs via Amazon S3 and SQS, and Google GCP for different events that The Integrated Cyber Defense Schema organizes into following categories:
 
 **Security [1]**
 
@@ -87,7 +83,7 @@ The Symantec Endpoint Security integration collects logs for different events th
 |----------------------------------------------------|
 | 1000 - Status                                      |
 
-**NOTE**: The Symantec Endpoint Security integration collects logs for the above mentioned events, but we have combined all of those in one data stream named `event`.
+The Symantec Endpoint Security integration can also retrieve **EDR incidents** via a REST API. See more details in the API documentation [here](https://apidocs.securitycloud.symantec.com/#/doc?id=edr_incidents).
 
 ## Requirements
 
@@ -112,8 +108,8 @@ With this approach, you install Elastic Agent and manually configure the agent l
 You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
 
 There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
-
-The minimum **kibana.version** required is **8.12.0**.
+ 
+This module has been tested against **Symantec Integrated Cyber Defense Exchange 1.4.7** for events, and **Symantec Endpoint Security API Version v1** for EDR Incidents.   
 
 ## Setup
 
@@ -168,6 +164,16 @@ A sample JSON Credentials file looks as follows:
 3. Configure event notifications for an S3 bucket. Follow this [link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html).
    - While creating `event notification` select the event type as s3:ObjectCreated:*, destination type SQS Queue, and select the queue name created in Step 2.
 
+
+### Steps to obtain Client ID and Client Secret to collect data from EDR Incident API:
+
+1. Login to your [Symantec EDR Cloud console](https://sep.securitycloud.symantec.com/v2/landing).
+2. Click Integration > Client Applications.
+3. Click Add for adding Client Application.
+4. Enter Client Application Name and press the Add button.
+5. Select Client Secret from the top.
+6. Copy the Client ID and Client Secret.
+
 ### Enabling the integration in Elastic:
 
 1. In Kibana go to Management > Integrations
@@ -191,13 +197,24 @@ A sample JSON Credentials file looks as follows:
    - Project ID
    - Buckets
    - Service Account Key/Service Account Credentials File
+
+   or if you want to collect logs via the REST API, then you have to put the following details:
+   - Client ID
+   - Client Secret
+   - URL
+   - Token URL
+
 6. Save the integration.
 
 **NOTE**:
 
 1. There are other input combination options available for the AWS S3 and AWS SQS, please check [here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html).
 2. There are other input combination options available for the GCS, please check [here](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-gcs.html).
 
+### Troubleshooting
+
+If the user stops integration and starts integration again after 30 days, then user will not be able to collect data and will get an error as Symantec EDR Cloud only collects data for the last 30 days. To avoid this issue, create a new integration instead of restarting it after 30 days.
+
 ## Logs reference
 
 ### Event
@@ -209,3 +226,13 @@ This is the `Event` dataset.
 {{event "event"}}
 
 {{fields "event"}}
+
+### Incident
+
+This is the `Incident` dataset.
+
+#### Example
+
+{{event "incident"}}
+
+{{fields "incident"}}

packages/symantec_endpoint_security/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.3.0"
+  changes:
+    - description: Merge Symantec EDR Cloud into Symantec Endpoint Security.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/10530
 - version: "0.2.0"
   changes:
     - description: Removed import_mappings. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

packages/symantec_endpoint_security/data_stream/incident/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,15 @@
+version: '2.3'
+services:
+  ses-incident:
+    image: docker.elastic.co/observability/stream:v0.15.0
+    hostname: ses-incident
+    ports:
+      - 8090
+    volumes:
+      - ./files:/files:ro
+    environment:
+      PORT: '8090'
+    command:
+      - http-server
+      - --addr=:8090
+      - --config=/files/config-incident.yml

packages/symantec_endpoint_security/data_stream/incident/_dev/deploy/docker/files/config-incident.yml

@@ -0,0 +1,36 @@
+rules:
+  - path: /v1/oauth2/tokens
+    methods: ['POST']
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - 'application/json'
+        body: |
+          {"access_token":"xxxx","expires_in":3600,"token_type":"Bearer"}
+  - path: /v1/incidents
+    methods: ['POST']
+    request_headers:
+      Authorization:
+        - 'Bearer xxxx'
+    request_body: /.*"limit":1,"next":0,.*/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |-
+          {"total":2,"next":1,"incidents":[{"customer_uid":"TEST-JvOsaJktSS-eyL-dXhxOvA","incident_uid":"8e7edfb1-27d2-4837-98ca-e7d794119c3b","type":"INCIDENT_CREATION","resolution_id":1,"conclusion":"Suspicious Activity","remediation":"Investigate further activity at the endpoint by downloading a full dump of the endpoint's recorded data. Give particular attention to activities performed by cmd.exe.","priority_id":4,"category_id":1,"modified":"2023-04-26T22:01:58.648+00:00","state_id":1,"id":4,"product_uid":"31B0C880-0229-49E8-94C5-48D56B1BD7B9","device_time":1682545570400,"ref_incident_uid":102110,"rule_name":"Advanced Attack Technique","created":"2023-04-26T21:46:10.400+00:00","type_id":8075,"incident_url":"https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details","message":"Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution","version":"1.0","product_name":"Symantec Integrated Cyber Defense Manager","event_id":8075004,"domain_uid":"TEST-ZBg_IqnyTAijNjP2BOOcuw","detection_type":"Advanced Analytics","severity_id":4,"time":1682545570400,"suspected_breach":"Yes"}]}
+  - path: /v1/incidents
+    methods: ['POST']
+    request_headers:
+      Authorization:
+        - 'Bearer xxxx'
+    request_body: /.*"limit":1,"next":1,.*/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |-
+          {"total":2,"next":2,"incidents":[{"customer_uid":"TEST-JvOsaJktSS-eyL-dXhxOvA","incident_uid":"1e7edfb1-27d2-4837-98ca-e7d794119c3b","type":"INCIDENT_CREATION","resolution_id":1,"conclusion":"Suspicious Activity","remediation":"Investigate further activity at the endpoint by downloading a full dump of the endpoint's recorded data. Give particular attention to activities performed by cmd.exe.","priority_id":4,"category_id":1,"modified":"2023-04-26T22:01:58.648+00:00","state_id":1,"id":4,"product_uid":"31B0C880-0229-49E8-94C5-48D56B1BD7B9","device_time":1682545570400,"ref_incident_uid":102110,"rule_name":"Advanced Attack Technique","created":"2023-04-26T21:46:10.400+00:00","type_id":8075,"incident_url":"https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details","message":"Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution","version":"1.0","product_name":"Symantec Integrated Cyber Defense Manager","event_id":8075004,"domain_uid":"TEST-ZBg_IqnyTAijNjP2BOOcuw","detection_type":"Advanced Analytics","severity_id":4,"time":1682545570400,"suspected_breach":"Yes"}]}

packages/symantec_endpoint_security/data_stream/incident/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,4 @@
+fields:
+  tags:
+    - preserve_original_event
+    - preserve_duplicate_custom_fields

packages/symantec_endpoint_security/data_stream/incident/_dev/test/pipeline/test-incident.log

@@ -0,0 +1 @@
+{"customer_uid":"TEST-JvOsaJktSS-eyL-dXhxOvA","incident_uid":"8e7edfb1-27d2-4837-98ca-e7d794119c3b","type":"INCIDENT_CREATION","resolution_id":1,"conclusion":"Suspicious Activity","remediation":"Investigate further activity at the endpoint by downloading a full dump of the endpoint's recorded data. Give particular attention to activities performed by cmd.exe.","priority_id":4,"category_id":1,"modified":"2023-04-26T22:01:58.648+00:00","state_id":1,"id":4,"product_uid":"31B0C880-0229-49E8-94C5-48D56B1BD7B9","device_time":1682545570400,"ref_incident_uid":102110,"rule_name":"Advanced Attack Technique","created":"2023-04-26T21:46:10.400+00:00","type_id":8075,"incident_url":"https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details","message":"Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution","version":"1.0","product_name":"Symantec Integrated Cyber Defense Manager","event_id":8075004,"domain_uid":"TEST-ZBg_IqnyTAijNjP2BOOcuw","detection_type":"Advanced Analytics","severity_id":4,"time":1682545570400,"suspected_breach":"Yes"}

packages/symantec_endpoint_security/data_stream/incident/_dev/test/pipeline/test-incident.log-expected.json

@@ -0,0 +1,81 @@
+{
+    "expected": [
+        {
+            "@timestamp": "2023-04-26T21:46:10.400Z",
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "malware"
+                ],
+                "created": "2023-04-26T21:46:10.400Z",
+                "id": "8e7edfb1-27d2-4837-98ca-e7d794119c3b",
+                "kind": "alert",
+                "original": "{\"customer_uid\":\"TEST-JvOsaJktSS-eyL-dXhxOvA\",\"incident_uid\":\"8e7edfb1-27d2-4837-98ca-e7d794119c3b\",\"type\":\"INCIDENT_CREATION\",\"resolution_id\":1,\"conclusion\":\"Suspicious Activity\",\"remediation\":\"Investigate further activity at the endpoint by downloading a full dump of the endpoint's recorded data. Give particular attention to activities performed by cmd.exe.\",\"priority_id\":4,\"category_id\":1,\"modified\":\"2023-04-26T22:01:58.648+00:00\",\"state_id\":1,\"id\":4,\"product_uid\":\"31B0C880-0229-49E8-94C5-48D56B1BD7B9\",\"device_time\":1682545570400,\"ref_incident_uid\":102110,\"rule_name\":\"Advanced Attack Technique\",\"created\":\"2023-04-26T21:46:10.400+00:00\",\"type_id\":8075,\"incident_url\":\"https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details\",\"message\":\"Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution\",\"version\":\"1.0\",\"product_name\":\"Symantec Integrated Cyber Defense Manager\",\"event_id\":8075004,\"domain_uid\":\"TEST-ZBg_IqnyTAijNjP2BOOcuw\",\"detection_type\":\"Advanced Analytics\",\"severity_id\":4,\"time\":1682545570400,\"suspected_breach\":\"Yes\"}",
+                "provider": "Symantec Integrated Cyber Defense Manager",
+                "reason": "Suspicious Activity",
+                "severity": 4,
+                "type": [
+                    "info"
+                ],
+                "url": "https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details"
+            },
+            "http": {
+                "version": "1.0"
+            },
+            "message": "Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution",
+            "rule": {
+                "name": "Advanced Attack Technique"
+            },
+            "ses": {
+                "incident": {
+                    "category": "Security",
+                    "category_id": "1",
+                    "conclusion": "Suspicious Activity",
+                    "created": "2023-04-26T21:46:10.400Z",
+                    "customer_uid": "TEST-JvOsaJktSS-eyL-dXhxOvA",
+                    "detection_type": "Advanced Analytics",
+                    "device_time": "2023-04-26T21:46:10.400Z",
+                    "domain_uid": "TEST-ZBg_IqnyTAijNjP2BOOcuw",
+                    "event": "Incident Creation: Logged",
+                    "event_id": "8075004",
+                    "id": "4",
+                    "incident_uid": "8e7edfb1-27d2-4837-98ca-e7d794119c3b",
+                    "incident_url": "https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details",
+                    "message": "Victim-2:Signed Binary Proxy Execution, Deobfuscate/Decode Files or Information, Command and Scripting Interpreter: PowerShell, System Services: Service Execution",
+                    "modified": "2023-04-26T22:01:58.648Z",
+                    "outcome": "Logged",
+                    "priority": "Critical",
+                    "priority_id": "4",
+                    "product_name": "Symantec Integrated Cyber Defense Manager",
+                    "product_uid": "31B0C880-0229-49E8-94C5-48D56B1BD7B9",
+                    "ref_incident_uid": "102110",
+                    "remediation": "Investigate further activity at the endpoint by downloading a full dump of the endpoint's recorded data. Give particular attention to activities performed by cmd.exe.",
+                    "resolution": "Insufficient data",
+                    "resolution_id": "1",
+                    "rule_name": "Advanced Attack Technique",
+                    "severity": "Major",
+                    "severity_id": 4,
+                    "state": "New",
+                    "state_id": "1",
+                    "suspected_breach": true,
+                    "time": "2023-04-26T21:46:10.400Z",
+                    "type": "INCIDENT_CREATION",
+                    "type_id": "8075",
+                    "version": "1.0"
+                }
+            },
+            "tags": [
+                "preserve_original_event",
+                "preserve_duplicate_custom_fields"
+            ],
+            "url": {
+                "domain": "sep.securitycloud.symantec.com",
+                "original": "https://sep.securitycloud.symantec.com/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details",
+                "path": "/v2/incidents/incidentListing/8e7edfb1-27d2-4837-98ca-e7d794119c3b/details",
+                "scheme": "https"
+            }
+        }
+    ]
+}

packages/symantec_endpoint_security/data_stream/incident/_dev/test/system/test-default-config.yml

@@ -0,0 +1,14 @@
+input: cel
+service: ses-incident
+vars:
+  url: http://{{Hostname}}:{{Port}}
+  client_id: xxxx
+  client_secret: xxxx
+  token_url: http://{{Hostname}}:{{Port}}/v1/oauth2/tokens
+data_stream:
+  vars:
+    preserve_original_event: true
+    preserve_duplicate_custom_fields: true
+    batch_size: 1
+assert:
+  hit_count: 2

packages/symantec_endpoint_security/data_stream/incident/agent/stream/cel.yml.hbs

@@ -0,0 +1,102 @@
+config_version: 2
+interval: {{interval}}
+{{#if enable_request_tracer}}
+resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson"
+resource.tracer.maxbackups: 5
+{{/if}}
+{{#if proxy_url}}
+resource.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+resource.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+resource.timeout: {{http_client_timeout}}
+{{/if}}
+resource.url: {{url}}
+auth.oauth2:
+  client.id: {{client_id}}
+  client.secret: {{client_secret}}
+  token_url: {{token_url}}
+  endpoint_params:
+    grant_type: client_credentials
+state:
+  initial_interval: {{initial_interval}}
+  want_more: false
+  limit: {{batch_size}}
+  next: 0
+program: |
+  (
+    state.want_more ?
+      state
+    :
+      state.with({
+        "limit": state.limit,
+        "start_date": state.?cursor.last_timestamp.orValue(string(now - duration(state.initial_interval))),
+        "end_date": now,
+        "next": state.next,
+      })
+  ).as(state,
+    post_request(
+      state.url.trim_right("/") + "/v1/incidents",
+      "application/json",
+      {
+        "limit": state.limit,
+        "start_date": state.start_date,
+        "end_date": state.end_date,
+        "next": state.next
+      }.encode_json()
+    ).do_request().as(resp, resp.StatusCode == 200 ?
+        bytes(resp.Body).decode_json().as(body, 
+          (body.?next.orValue(body.total) != body.total).as(want_more, {
+            "events": body.incidents.map(e, {
+              "message": e.encode_json(),
+            }),
+            "next": want_more ? body.next : 0,
+            "want_more": want_more,
+            "limit": state.limit,
+            "start_date": string(state.start_date),
+            "end_date": string(state.end_date),
+            "cursor": {
+              ?"last_timestamp": want_more ?
+                  state.?cursor.last_timestamp
+                :
+                  optional.of(state.end_date),
+            }
+          })
+        )
+      :
+        {
+          "events": {
+            "error": {
+              "code": string(resp.StatusCode),
+              "id": string(resp.Status),
+              "message": "POST:"+(
+                size(resp.Body) != 0 ?
+                  string(resp.Body)
+                :
+                  string(resp.Status) + ' (' + string(resp.StatusCode) + ')'
+              ),
+            },
+          },
+          "want_more": false,
+        }
+    )
+  )
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#if preserve_duplicate_custom_fields}}
+  - preserve_duplicate_custom_fields
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}