Initial release of PingFederate (#12113)

Create New integration package ping_federate.

Added admin and audit data stream.
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added pipeline test for the data streams.
Added system test cases for the data streams.

.github/CODEOWNERS

@@ -314,6 +314,7 @@
 /packages/pfsense @elastic/sec-deployment-and-devices
 /packages/php_fpm @elastic/obs-infraobs-integrations
 /packages/ping_one @elastic/security-service-integrations
+/packages/ping_federate @elastic/security-service-integrations
 /packages/platform_observability @elastic/stack-monitoring
 /packages/postgresql @elastic/obs-infraobs-integrations
 /packages/pps @elastic/security-service-integrations

packages/ping_federate/_dev/build/build.yml

@@ -0,0 +1,3 @@
+dependencies:
+  ecs:
+    reference: [email protected]

packages/ping_federate/_dev/build/docs/README.md

@@ -0,0 +1,98 @@
+# PingFederate
+
+## Overview
+
+[PingFederate](https://www.pingidentity.com/en/platform/capabilities/authentication-authority/pingfederate.html) is a key component of the [PingIdentity](https://www.pingidentity.com/en.html) platform, which is a suite of solutions for identity and access management (IAM). Specifically, Ping Federate is an enterprise-grade federated identity server designed to enable secure single sign-on (SSO), identity federation, and access management for applications and services.
+
+## Compatibility
+
+This module has been tested with the latest version of PingFederate, **12.1.4(November 2024)**.
+## Data streams
+
+The PingFederate integration collects two types of logs:
+
+**[Admin](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_admin_audit_loggin.html)** - Record actions performed within the PingFederate Administrative Console and via the Administrative API.
+
+**[Audit](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_security_audit_loggin.html)** - Provides a detailed record of authentication, authorization, and federation transactions.
+
+**Note**:
+
+1. In the Admin datastream, only logs from the admin.log file are supported via filestream in the pipe format. The log pattern is as follows:
+```
+<pattern>%d | %X{user} | %X{roles} | %X{ip} | %X{component} | %X{event} | %X{eventdetailid} | %m%n</pattern>
+```
+Sample Log:
+```
+2024-11-28 5:58:55,832 | Administrator | UserAdmin,Admin,CryptoAdmin,ExpressionAdmin | 81.2.69.142 | A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nxxxxyn6H4 | LICENSE | ROTATE | - Login was successful
+```
+
+2. Audit logs are supported through filestream, TCP, and UDP in the CEF format. The log pattern is as follows:
+```
+<pattern>%escape{CEF}{CEF:0|Ping Identity|PingFederate|%X{pfversion}|%X{event}|%X{event}|0|rt=%d{MMM dd yyyy HH:mm:ss.SSS} duid=%X{subject} src=%X{ip} msg=%X{status} cs1Label=Target Application URL cs1=%X{app} cs2Label=Connection ID cs2=%X{connectionid} cs3Label=Protocol cs3=%X{protocol} dvchost=%X{host} cs4Label=Role cs4=%X{role} externalId=%X{trackingid} cs5Label=SP Local User ID cs5=%X{localuserid} cs6Label=Attributes cs6=%X{attributes} %n}</pattern>
+```
+Sample Log:
+```
+CEF:0|Ping Identity|PingFederate|6.4|AUTHN_SESSION_DELETED|AUTHN_SESSION_DELETED|0|rt=May 18 2012 11:41:48.452 duid=joe src=89.160.20.112 msg=failure cs1Label=Target Application URL cs1=http://www.google.ca&landingpage\=pageA cs2Label=Connection ID cs2=sp:cloud:saml2 cs3Label=Protocol cs3=SAML20 dvchost=hello cs4Label=Role cs4=IdP externalId=tid:ae14b5ce8 cs5Label=SP Local User ID cs5=idlocal cs6Label=Attributes cs6={SAML_SUBJECT\=joe, ognl\=tom}
+```
+
+## Requirements
+
+- Elastic Agent must be installed.
+- You can install only one Elastic Agent per host.
+- Elastic Agent is required to stream data through the Filestream or TCP/UDP and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.
+
+### Installing and managing an Elastic Agent:
+
+You have a few options for installing and managing an Elastic Agent:
+
+### Install a Fleet-managed Elastic Agent (recommended):
+
+With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
+
+### Install Elastic Agent in standalone mode (advanced users):
+
+With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
+
+### Install Elastic Agent in a containerized environment:
+
+You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
+
+There are some minimum requirements for running Elastic Agent. For more information, refer to the Elastic Agent [installation guide](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html).
+
+## Setup
+
+1. For step-by-step instructions on how to configure log files in PingFederate instance, see the [Log4j 2 logging service and configuration](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_log4j_2_loggin_service_and_config.html) guide.
+2. To write the audit logs in cef format, see the [Writing audit log in CEF](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_writin_audit_log_cef.html) guide.
+
+### Enabling the integration in Elastic:
+
+1. In Kibana go to Management > Integrations.
+2. In "Search for integrations" search bar, type PingFederate.
+3. Click on the "PingFederate" integration from the search results.
+4. Click on the "Add PingFederate" button to add the integration.
+5. Select the toggle for the data stream for which you want to collect logs.
+6. Enable the data collection mode from the following: Filestream, TCP, or UDP. (Admin logs are only supported through Filestream)
+7. Add all the required configuration parameters, such as paths for the filestream or listen address and listen port for the TCP and UDP.
+8. Click on "Save and Continue" to save the integration.
+
+## Logs Reference
+
+### Admin
+
+This is the `Admin` dataset.
+
+#### Example
+
+{{event "admin"}}
+
+{{fields "admin"}}
+
+### Audit
+
+This is the `Audit` dataset.
+
+#### Example
+
+{{event "audit"}}
+
+{{fields "audit"}}

packages/ping_federate/_dev/deploy/docker/docker-compose.yml

@@ -0,0 +1,29 @@
+version: '2.3'
+services:
+  ping_federate-tcp-audit:
+    image: docker.elastic.co/observability/stream:v0.10.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9598 -p=tcp /sample_logs/test-audit.log
+  ping_federate-udp-audit:
+    image: docker.elastic.co/observability/stream:v0.10.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9599 -p=udp /sample_logs/test-audit.log
+  ping_federate-logfile-audit:
+    image: alpine
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+      - ${SERVICE_LOGS_DIR}:/var/log
+    command: /bin/sh -c "cp /sample_logs/* /var/log/"
+  ping_federate-tls-audit:
+    image: docker.elastic.co/observability/stream:v0.10.0
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+    command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9598 -p=tls --insecure /sample_logs/test-audit.log
+  ping_federate-logfile-admin:
+    image: alpine
+    volumes:
+      - ./sample_logs:/sample_logs:ro
+      - ${SERVICE_LOGS_DIR}:/var/log
+    command: /bin/sh -c "cp /sample_logs/* /var/log/"

packages/ping_federate/_dev/deploy/docker/sample_logs/test-admin.log

@@ -0,0 +1 @@
+2024-11-28 5:58:55,832 | Administrator | UserAdmin,Admin,CryptoAdmin,ExpressionAdmin | 81.2.69.142 | A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nef5uKyn6H4 | LICENSE | ROTATE | - Login was successful

packages/ping_federate/_dev/deploy/docker/sample_logs/test-audit.log

@@ -0,0 +1 @@
+CEF:0|Ping Identity|PingFederate|6.4|AUTHN_SESSION_DELETED|AUTHN_SESSION_DELETED|0|rt=May 18 2012 11:41:48.452 duid=joe src=192.168.6.130 msg=failure cs1Label=Target Application URL cs1=http://www.google.ca&landingpage\=pageA cs2Label=Connection ID cs2=sp:cloud:saml2 cs3Label=Protocol cs3=SAML20 dvchost=hello cs4Label=Role cs4=IdP externalId=tid:ae14b5ce8 cs5Label=SP Local User ID cs5=idlocal cs6Label=Attributes cs6={SAML_SUBJECT\=joe, ognl\=tom}

packages/ping_federate/changelog.yml

@@ -0,0 +1,6 @@
+# newer versions go on top
+- version: "0.1.0"
+  changes:
+    - description: Initial Release.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/12113

packages/ping_federate/data_stream/admin/_dev/test/pipeline/test-admin.log

@@ -0,0 +1 @@
+2024-11-28 5:58:55,832 | Administrator | UserAdmin,Admin,CryptoAdmin,ExpressionAdmin | 81.2.69.142 | A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nef5uKyn6H4 | LICENSE | ROTATE | - Login was successful

packages/ping_federate/data_stream/admin/_dev/test/pipeline/test-admin.log-expected.json

@@ -0,0 +1,83 @@
+{
+    "expected": [
+        {
+            "@timestamp": "2024-11-28T16:58:55.832+11:00",
+            "ecs": {
+                "version": "8.16.0"
+            },
+            "event": {
+                "action": "rotate",
+                "category": [
+                    "configuration"
+                ],
+                "id": "A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nef5uKyn6H4",
+                "kind": "event",
+                "original": "2024-11-28 5:58:55,832 | Administrator | UserAdmin,Admin,CryptoAdmin,ExpressionAdmin | 81.2.69.142 | A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nef5uKyn6H4 | LICENSE | ROTATE | - Login was successful",
+                "timezone": "+11:00",
+                "type": [
+                    "change"
+                ]
+            },
+            "message": "- Login was successful",
+            "observer": {
+                "product": "PingFederate",
+                "vendor": "Ping Identity"
+            },
+            "ping_federate": {
+                "admin": {
+                    "component": "LICENSE",
+                    "event": {
+                        "detail_id": "A-rBnNPcJffxBiizBWDOWxq_Ek8cYxg3nef5uKyn6H4",
+                        "type": "ROTATE"
+                    },
+                    "ip": "81.2.69.142",
+                    "message": "- Login was successful",
+                    "roles": [
+                        "UserAdmin",
+                        "Admin",
+                        "CryptoAdmin",
+                        "ExpressionAdmin"
+                    ],
+                    "timestamp": "2024-11-28T16:58:55.832+11:00",
+                    "user": "Administrator"
+                }
+            },
+            "related": {
+                "ip": [
+                    "81.2.69.142"
+                ],
+                "user": [
+                    "Administrator"
+                ]
+            },
+            "source": {
+                "geo": {
+                    "city_name": "London",
+                    "continent_name": "Europe",
+                    "country_iso_code": "GB",
+                    "country_name": "United Kingdom",
+                    "location": {
+                        "lat": 51.5142,
+                        "lon": -0.0931
+                    },
+                    "region_iso_code": "GB-ENG",
+                    "region_name": "England"
+                },
+                "ip": "81.2.69.142"
+            },
+            "tags": [
+                "preserve_original_event",
+                "preserve_duplicate_custom_fields"
+            ],
+            "user": {
+                "name": "Administrator",
+                "roles": [
+                    "UserAdmin",
+                    "Admin",
+                    "CryptoAdmin",
+                    "ExpressionAdmin"
+                ]
+            }
+        }
+    ]
+}

packages/ping_federate/data_stream/admin/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1,6 @@
+fields:
+  tags:
+    - preserve_original_event
+    - preserve_duplicate_custom_fields
+  _conf:
+    tz_offset: "+11:00"

packages/ping_federate/data_stream/admin/_dev/test/system/test-logfile-config.yml

@@ -0,0 +1,15 @@
+service: ping_federate-logfile-admin
+input: filestream
+data_stream:
+  vars:
+    paths:
+      - "{{SERVICE_LOGS_DIR}}/*admin*.log"
+    tz_offset: "+11:00"
+    preserve_original_event: true
+    preserve_duplicate_custom_fields: true
+numeric_keyword_fields:
+  - log.file.device_id
+  - log.file.inode
+  - log.file.idxhi
+  - log.file.idxlo
+  - log.file.vol

packages/ping_federate/data_stream/admin/agent/stream/filestream.yml.hbs

@@ -0,0 +1,32 @@
+paths:
+{{#each paths as |path|}}
+  - {{path}}
+{{/each}}
+exclude_files: ['\.gz$']
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#if preserve_duplicate_custom_fields}}
+  - preserve_duplicate_custom_fields
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+processors:
+- rename:
+    fields:
+      - {from: "message", to: "event.original"}
+{{#if tz_offset}}
+fields_under_root: true
+fields:
+  _conf:
+    tz_offset: "{{tz_offset}}"
+{{/if}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}