[ProxySG] Support 'bcreportermain_v1', 'bcreporterssl_v1', and 'ssl' …

…formats (#11609)

In the ProxySG integration, add support for 'bcreportermain_v1', 'bcreporterssl_v1', and 'ssl' log formats. These formats are defined here: https://techdocs.broadcom.com/us/en/symantec-security-software/web-and-network-security/edge-swg/7-3/getting-started/page-help-administration/page-help-logging/log-formats/default-formats.html

As it's not possible to automatically detect what log format is being used, users of the integration will need to select the log format to be processed beforehand when setting up the input (this selector dropdown already exists, but previously only had one option).

packages/proxysg/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: 0.2.0
+  changes:
+    - description: Add support for 'bcreportermain_v1', 'bcreporterssl_v1' and 'ssl' formats
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/11609
 - version: 0.1.0
   changes:
     - description: Initial creation of the integration.

packages/proxysg/data_stream/log/_dev/test/pipeline/test-proxy-bcreportermainv1.log

@@ -0,0 +1,5 @@
+2024-11-01 14:23:54 1 10.0.1.2 - - - - "None" - authentication_redirect_to_virtual_host DENIED "Web Infrastructure" -  302 TCP_AUTH_REDIRECT GET - "http" www.msftconnecttest.com 80 /connecttest.txt - txt "Microsoft NCSI" 10.0.1.3 1033 111 1 2 3 4 5 "none" "none" "none" unavailable - - "2aaf931f4d5c270c-0000000000024973-000000006724e47a" - -
+2024-11-01 14:24:28 6 10.0.1.2 - - - - "None" - authentication_failed DENIED "Chat (IM)/SMS;Social Networking" https://www.regions.com/  401 TCP_DENIED OPTIONS - "https" tr6.snapchat.com 443 /p - - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 10.0.1.3 1297 582 - - - - - "Snapchat" "none" "Instant Messaging;Photo Sharing" unavailable - - "2aaf931f4d5c270c-0000000000024978-000000006724e49c" - -
+2024-11-01 14:25:53 6 10.0.1.2 - - - - "None" - authentication_failed DENIED "Web Infrastructure" -  401 TCP_DENIED GET - "http" ctldl.windowsupdate.com 80 /msdownload/update/v3/static/trustedr/en/pinrulesstl.cab ?1fcb4f4be3fc14ce cab "Microsoft-CryptoAPI/10.0" 10.0.1.3 1297 291 - - - - - "Microsoft Update" "Update Software" "none" unavailable - - "2aaf931f4d5c270c-0000000000024995-000000006724e4f1" - -
+2024-11-01 14:25:56 1 10.0.1.2 - - - - "None" - authentication_redirect_to_virtual_host DENIED "Web Infrastructure" -  302 TCP_AUTH_REDIRECT GET - "http" www.msftconnecttest.com 80 /connecttest.txt - txt "Microsoft NCSI" 10.0.1.3 1033 111 - - - - - "none" "none" "none" unavailable - - "2aaf931f4d5c270c-0000000000024996-000000006724e4f4" - -
+2024-11-01 14:41:40 4 10.0.1.2 - - - - "None" - authentication_failed DENIED "Web Ads/Analytics" https://tpc.googlesyndication.com/  401 TCP_DENIED OPTIONS - "https" dt.adsafeprotected.com 443 /dt ?advEntityId=3333333&asId=5a332ac0-c0bb-6109-dd47-7b98aaf9fde0&tv=%7Bc:sNKJGr,pingTime:-1,time:4024544,type:u,clog:%5B%7Bpiv:-1,vs:o,r:h.v,w:0,h:0,t:87%7D,%7Bpiv:0,r:l.h.v,t:165%7D,%7Br:l.h,t:2175%7D,%7Bpiv:100,vs:i,r:,w:400,h:225,t:3929%7D,%7Bvs:o,r:f,t:4798%7D,%7Br:v,t:5739%7D,%7Bvs:i,r:,t:5931%7D,%7Bvs:o,r:f,t:8873%7D,%7Br:f.v,t:10024%7D,%7Br:v,t:10739%7D,%7Bvs:i,r:,t:11329%7D,%7Bvs:o,r:f,t:14037%7D,%7Br:f.v,t:14618%7D,%7Br:v,t:2847110%7D,%7Br:f,t:2847613%7D,%7Br:f.v,t:2849615%7D%5D,ve:%7BvEventCount:20,vEvents:%5B%7Bt:-1293,tp:adLoaded,sl:o,ad_duration:15,width:0,height:0,volume:-2%7D,%7Bt:-869,tp:adStarted,sl:o,ad_duration:15,width:0,height:0,volume:-2%7D,%7Bt:1791,tp:adDurationChange,sl:o,ad_duration:15.232,width:0,height:0,volume:-2%7D,%7Bt:2043,tp:adImpression,sl:o,ad_duration:15.232,width:0,height:0,volume:-2,integral_timeToDecision:1948,integral_didBlock:false,viewMode:normal,x_vv:3.8.12renddet:env,lt:1,siq:92%7D&br=c - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 10.0.1.3 1316 510 - - - - - "none" "none" "none" unavailable - - "2aaf931f4d5c270c-0000000000024a6c-000000006724e8a4" - -

packages/proxysg/data_stream/log/_dev/test/pipeline/test-proxy-bcreportermainv1.log-config.yml

@@ -0,0 +1,3 @@
+fields:
+  _temp_:
+    _conf: bcreportermain_v1

packages/proxysg/data_stream/log/_dev/test/pipeline/test-proxy-bcreportersslv1.log

@@ -0,0 +1,5 @@
+2024-11-01 13:26:42 9 10.0.1.1 - - - - "None" - authentication_redirect_to_virtual_host DENIED "Web Ads/Analytics" 307 TCP_AUTH_REDIRECT POST - "https" server.googleapis.com 443 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0" 10.0.1.2 1039 11343 - - - - - none - - high *.googleapis.com "Web Ads/Analytics" 2 2 - -
+2024-11-01 13:26:42 33 10.0.1.1 user1 - - - "None" - authentication_success DENIED "Web Ads/Analytics" 307 TCP_DENIED GET - "https" px.moatads.com 443 gif "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0" 10.0.1.2 1231 720 - - - - - none - - high moatads.com "Web Ads/Analytics" 3 4 - -
+2024-11-01 13:26:42 31 10.0.1.1 - - - - "None" - authentication_redirect_to_virtual_host DENIED "Web Ads/Analytics" 307 TCP_AUTH_REDIRECT POST - "https" t.txt.com 443 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0" 10.0.1.2 1167 7791 - - - - - none - - high *.3lift.com "Web Ads/Analytics" 2 2 - -
+2024-11-01 13:26:47 63703 10.0.1.1 user1 - secure.espn.com 10.1.1.2 "None" - - OBSERVED "Sports/Recreation" 0 TUNNELED unknown - "ssl" secure.espn.com 443 - - 10.0.1.3 42796 3174 - - - - - none - - high a.espncdn.com "Content Delivery Networks" 1 1 - -
+2024-11-01 13:26:48 8 10.0.1.1 - - - - "None" - authentication_redirect_to_virtual_host DENIED "Technology/Internet" 307 TCP_AUTH_REDIRECT GET - "https" edge.microsoft.com 443 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0" 10.0.1.2 1323 764 - - - - - none - - high edge.microsoft.com "Technology/Internet" 1 1 - -

packages/proxysg/data_stream/log/_dev/test/pipeline/test-proxy-bcreportersslv1.log-config.yml

@@ -0,0 +1,3 @@
+fields:
+  _temp_:
+    _conf: bcreporterssl_v1