[New Integration] blacklens.io (#11664)

Create new integration blacklens.io

[blacklens.io](https://blacklens.io) is a powerful Attack Surface Management (ASM) platform designed to help organizations monitor and secure their external digital footprint. It provides in-depth insights into potential threats, including vulnerabilities and darknet activities, giving you a comprehensive view of your attack surface.

.github/CODEOWNERS

@@ -119,6 +119,7 @@
 /packages/beat @elastic/stack-monitoring
 /packages/bitdefender @elastic/security-service-integrations
 /packages/bitwarden @elastic/security-service-integrations
+/packages/blacklens @elastic/security-service-integrations
 /packages/bluecoat @elastic/sec-deployment-and-devices
 /packages/box_events @elastic/security-service-integrations
 /packages/canva @elastic/security-service-integrations

packages/blacklens/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/blacklens/_dev/build/docs/README.md

@@ -0,0 +1,55 @@
+# blacklens.io
+
+The [blacklens.io](https://blacklens.io) integration allows you to monitor alerts. blacklens.io is a comprehensive Attack Surface Management platform that helps businesses understand and secure their external attack surface. By combining automated security analysis, continuous monitoring, and penetration testing, it identifies and addresses vulnerabilities early. With features like Darknet Monitoring, Vulnerability Scanning, and XDR Response, blacklens.io provides a proactive defense strategy to protect companies from cyber threats while offering a clear view of their security posture at all times.
+
+Use the blacklens.io integration to fetch all related alerts about your Attack Surface. Then visualize that data in Kibana and create further alerts or enrich the data with other security solutions.
+
+## Data streams
+
+The blacklens.io integration collects one type of data streams: logs
+
+**Alerts** returns a list of blacklens.io alerts (The API Docs are referenced within the portal)
+## Requirements
+
+You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
+You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
+You will require the `alerts:read` permission in order to fetch the Alerts via the API.
+
+You need an active blacklens.io subscription and a user with the `alerts:read` permission to retrieve alerts via the API.
+
+## Setup
+
+### Copy blacklens.io required configuration properties
+
+1. Login to your blacklens.io Portal (This URL will be used for the Instance URL: 'https://portal-(ID).blacklens.io')
+2. Go to **Profile → Generate API Key** and copy it. 
+3. Go to **Settings → Company**.
+4. Copy **ws_id** and **tenant_id**.
+
+### Enable the blacklens.io Integration in Elastic
+
+1. In Kibana go to Management > Integrations.
+2. In "Search for integrations" search bar, type blacklens.io.
+3. Click on the "blacklens.io" integration from the search results.
+4. Click on the "Add blacklens.io" button to add the integration.
+5. Configure all required integration parameters. 
+    - Alert data requires following parameters:
+        - `URL`
+        - `Tenant ID (tenant_id)`
+        - `Workspace ID (ws_id)`
+        - `API Key`
+6. Save the integration.
+
+For detailed setup instructions, please refer to the blacklens.io Knowledge Base (The link is referenced within the portal).
+
+## Logs reference
+
+### alerts
+
+This is the `alerts` dataset
+
+#### Example
+
+{{ event "alerts"}}
+
+{{ fields "alerts"}}

packages/blacklens/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Initial draft of the package
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11664

packages/blacklens/data_stream/alerts/_dev/test/pipeline/test-alerts.log

@@ -0,0 +1 @@
+{"updated_date":"2024-11-12T09:39:58.489Z","created_date":"2024-11-12T09:39:58.489Z","id":1001,"details":{"id":100,"engine":"Port Scanner","title":"New Open Port"},"severity":"medium","affected_entities":2,"alert_outcome":"affected","alert_status":"resolved","customer_state":"open","alert_payload":[],"type_id":100}

packages/blacklens/data_stream/alerts/_dev/test/pipeline/test-alerts.log-expected.json

@@ -0,0 +1,32 @@
+{
+    "expected": [
+        {
+            "@timestamp": "2024-11-12T09:39:58.489Z",
+            "blacklens": {
+                "alert": {
+                    "id": 1001,
+                    "outcome": "affected",
+                    "severity": "medium",
+                    "status": "resolved",
+                    "title": "New Open Port",
+                    "type": "Port Scanner",
+                    "type_id": 100,
+                    "updated_date": "2024-11-12T09:39:58.489Z"
+                }
+            },
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "category": [
+                    "threat"
+                ],
+                "id": "1001",
+                "original": "{\"updated_date\":\"2024-11-12T09:39:58.489Z\",\"created_date\":\"2024-11-12T09:39:58.489Z\",\"id\":1001,\"details\":{\"id\":100,\"engine\":\"Port Scanner\",\"title\":\"New Open Port\"},\"severity\":\"medium\",\"affected_entities\":2,\"alert_outcome\":\"affected\",\"alert_status\":\"resolved\",\"customer_state\":\"open\",\"alert_payload\":[],\"type_id\":100}",
+                "type": [
+                    "indicator"
+                ]
+            }
+        }
+    ]
+}

packages/blacklens/data_stream/alerts/agent/stream/httpjson.yml.hbs

@@ -0,0 +1,67 @@
+config_version: 2
+interval: {{interval}}
+
+request.method: GET
+request.url: {{url}}/api/v1/{{ws_id}}/alert
+
+{{#if enable_request_tracer}}
+request.tracer.filename: "../../logs/httpjson/http-request-trace-*.ndjson"
+request.tracer.maxbackups: 5
+{{/if}}
+
+{{#if proxy_url }}
+request.proxy_url: {{proxy_url}}
+{{/if}}
+{{#if ssl}}
+request.ssl: {{ssl}}
+{{/if}}
+{{#if http_client_timeout}}
+request.timeout: {{http_client_timeout}}
+{{/if}}
+request.transforms:
+  - set:
+      target: header.X-API-Key
+      value: '{{api_key}}'
+  - set:
+      target: header.X-Tenant
+      value: '{{tenant_id}}'
+  - set:
+      target: url.params.per_page
+      value: '50'
+  - set:
+      target: url.params.sort_by
+      value: 'created_date'
+  - set:
+      target: url.params.sort
+      value: 'asc'
+  - set:
+      target: url.params.q
+      value: 'created_date>[[.cursor.last_created_date]]'
+      default: 'created_date>[[formatDate (now (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05Z"]]'
+
+response.pagination:
+  - set:
+      target: url.params.page
+      value: '[[if (and .last_response.body.pagination.next (ne .last_response.body.pagination.next ""))]][[add (toInt .last_response.body.pagination.page) 1]][[end]]'
+      fail_on_template_error: true
+
+response.split:
+  target: body.results
+  ignore_empty_value: true
+cursor:
+  last_created_date:
+    value: '[[.last_event.created_date]]'
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+  {{processors}}
+{{/if}}

packages/blacklens/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml

@@ -0,0 +1,104 @@
+---
+description: Pipeline for processing alert logs.
+processors:
+  - set:
+      field: ecs.version
+      value: '8.11.0'
+  - rename:
+      field: message
+      target_field: event.original
+      ignore_missing: true
+      if: ctx.event?.original == null
+  - remove:
+      field: message
+      ignore_missing: true
+      if: 'ctx.event?.original != null'
+  - append:
+      field: event.category
+      value: threat
+      allow_duplicates: false
+  - append:
+      field: event.type
+      value: indicator
+      allow_duplicates: false
+  - json:
+      field: event.original
+      target_field: json
+  - fingerprint:
+      fields:
+        - json.id
+      target_field: _id
+      ignore_missing: true
+  - date:
+      field: json.created_date
+      formats:
+        - ISO8601
+  - date:
+      field: json.updated_date
+      target_field: blacklens.alert.updated_date
+      formats:
+        - ISO8601
+  - convert:
+      field: json.id
+      target_field: event.id
+      type: string
+      ignore_missing: true
+  - rename:
+      field: json.id
+      target_field: blacklens.alert.id
+      ignore_missing: true
+  - rename:
+      field: json.severity
+      target_field: blacklens.alert.severity
+      ignore_missing: true
+  - rename:
+      field: json.alert_status
+      target_field: blacklens.alert.status
+      ignore_missing: true
+  - rename:
+      field: json.alert_outcome
+      target_field: blacklens.alert.outcome
+      ignore_missing: true
+  - rename:
+      field: json.details.engine
+      target_field: blacklens.alert.type
+      ignore_missing: true
+  - rename:
+      field: json.type_id
+      target_field: blacklens.alert.type_id
+      ignore_missing: true
+  - rename:
+      field: json.details.title
+      target_field: blacklens.alert.title
+      ignore_missing: true
+  - rename:
+      field: json.alert_payload
+      target_field: blacklens.alert.details
+      ignore_missing: true
+  - remove:
+      field: json
+      ignore_missing: true
+  - script:
+      description: Drops null/empty values recursively.
+      lang: painless
+      source: |
+        boolean dropEmptyFields(Object object) {
+          if (object == null || object == '') {
+            return true;
+          } else if (object instanceof Map) {
+            ((Map) object).values().removeIf(value -> dropEmptyFields(value));
+            return (((Map) object).size() == 0);
+          } else if (object instanceof List) {
+            ((List) object).removeIf(value -> dropEmptyFields(value));
+            return (((List) object).length == 0);
+          }
+          return false;
+        }
+        dropEmptyFields(ctx);
+on_failure:
+  - set:
+      field: event.kind
+      value: pipeline_error
+  - append:
+      field: error.message
+      value: '{{{ _ingest.on_failure_message }}}'

packages/blacklens/data_stream/alerts/fields/base-fields.yml

@@ -0,0 +1,20 @@
+- name: data_stream.type
+  type: constant_keyword
+  description: Data stream type.
+- name: data_stream.dataset
+  type: constant_keyword
+  description: Data stream dataset.
+- name: data_stream.namespace
+  type: constant_keyword
+  description: Data stream namespace.
+- name: event.module
+  type: constant_keyword
+  description: Event module.
+  value: blacklens
+- name: event.dataset
+  type: constant_keyword
+  description: Event dataset.
+  value: blacklens.alerts
+- name: '@timestamp'
+  type: date
+  description: Event timestamp.

packages/blacklens/data_stream/alerts/fields/beats.yml

@@ -0,0 +1,3 @@
+- name: input.type
+  type: keyword
+  description: Type of filebeat input.

packages/blacklens/data_stream/alerts/fields/ecs.yml

@@ -0,0 +1,10 @@
+# Define ECS constant fields as constant_keyword
+- name: event.kind
+  type: constant_keyword
+  value: alert
+- name: observer.product
+  type: constant_keyword
+  value: blacklens.io
+- name: observer.vendor
+  type: constant_keyword
+  value: blacklens.io

packages/blacklens/data_stream/alerts/fields/fields.yml

@@ -0,0 +1,30 @@
+- name: blacklens.alert
+  type: group
+  fields:
+    - name: updated_date
+      type: date
+      description: Activity last updated time (UTC).
+    - name: id
+      type: integer
+      description: Unique Alert ID
+    - name: severity
+      type: keyword
+      description: Alert Severity
+    - name: status
+      type: keyword
+      description: Current Status of the Alert
+    - name: outcome
+      type: keyword
+      description: Determines whether the current alert triggers further events
+    - name: type
+      type: keyword
+      description: Alert Type (Engine)
+    - name: type_id
+      type: integer
+      description: Alert Type ID (Engine)
+    - name: title
+      type: keyword
+      description: Title/Description of the given Alert
+    - name: details
+      type: nested
+      description: Alert Details