Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial release of Falco integration #9619

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
bb50e86
work in progress on Falco integration
brewcore Apr 16, 2024
285a847
- Significant updates to pipeline and supporting files based on PR fe…
cole-labar May 2, 2024
7591409
Updates to field definition files and pipeline to satisfy test suites…
cole-labar May 3, 2024
71406fe
Automatic linting fixes
brewcore May 7, 2024
9c78136
Fix pipeline test
brewcore May 7, 2024
5e4a0ac
Regenerated sample event
brewcore May 7, 2024
b050aaf
Work in progress - ingest pipeline refactor and cleanup
brewcore May 9, 2024
30e3dca
remove undocumented fields, updated sample event
brewcore May 13, 2024
a5a81fa
move to tcp input with syslog processor
brewcore Jun 4, 2024
cfb4c57
Reworked field definitions and updated pipeline processors related to…
cole-labar Jun 11, 2024
6e0576b
Updated fieldspec to fix build issue
cole-labar Jun 11, 2024
9a5cb0d
Updated dashboard and added data view. Updated pipeline with timestam…
cole-labar Jun 12, 2024
e85d2dc
Updated docs, pipeline, and sample events from remaining PR feedback
cole-labar Jun 13, 2024
f0c9cf1
add failing pipeline test to match mapping spec for container.image data
brewcore Jun 17, 2024
b21ecdc
Updated dashboard and included Raw Events stream component
cole-labar Jun 18, 2024
308bd17
Adjusted documentation to be more accurate and updated pipeline / fie…
cole-labar Jun 24, 2024
72180a8
Significant updates across the board:
cole-labar Jun 28, 2024
95b132e
Updated from PR review. Updated documentation, field definitions, and…
cole-labar Jun 28, 2024
19a0a91
Merge branch 'bugfix/update-pipeline-dot-processor' into feature/ESI-…
cole-labar Jun 28, 2024
a6bf157
Re-ran tests and build process following merge to ensure updated.
cole-labar Jun 28, 2024
b75b272
Updated dashboard with additional fields for when original event is n…
cole-labar Jul 2, 2024
ba6f9a2
Use template variables in docs
brewcore Jul 2, 2024
3b84bc6
version number set for initial release
brewcore Jul 2, 2024
98f24b9
Updated manifest to remove license
cole-labar Jul 8, 2024
c772e59
Updated data stream naming convention and added in dynamic mapping se…
cole-labar Jul 15, 2024
d0c9a03
Removed ECS field definitions as per new mapping
cole-labar Jul 15, 2024
d2d5904
Updated port assignment
cole-labar Jul 15, 2024
510a6bb
Update packages/falco/changelog.yml
cole-labar Jul 15, 2024
e4d55c4
Update packages/falco/manifest.yml
cole-labar Jul 15, 2024
56b22e9
Update packages/falco/manifest.yml
cole-labar Jul 15, 2024
42bb494
Update packages/falco/manifest.yml
cole-labar Jul 15, 2024
b87004b
Update packages/falco/manifest.yml
cole-labar Jul 15, 2024
8611e2b
Removed index pattern and updated dashboard elements. Updated port se…
cole-labar Jul 16, 2024
ed95cea
Updated remove processor
cole-labar Jul 16, 2024
1c9e459
Updated dashboard
cole-labar Jul 17, 2024
bcd26c2
Merge branch 'main' into feature/ESI-21-Mappings
cole-labar Jul 17, 2024
d1d1986
Updated the namespace base field
cole-labar Jul 17, 2024
66233ab
Updated screenshot, dashboard, and field removal for better UX
cole-labar Jul 18, 2024
44f8a9e
Updated namespace field definition
cole-labar Jul 18, 2024
f4303ac
Updated agent.yml fieldspec to remove ecs fields
cole-labar Jul 19, 2024
faf8b3b
Added in handler for MITRE subtechnique field and adjusted field mapp…
cole-labar Jul 19, 2024
dad393b
Updated removed field list
cole-labar Jul 19, 2024
793dccb
Updated observer fields
cole-labar Jul 19, 2024
a4f4743
Updated event.start to use a date processor
cole-labar Jul 19, 2024
a117225
Updated pipeline and dashboard from comments.
cole-labar Jul 23, 2024
ce15758
Updated test suites to mock data and updated example logs
cole-labar Aug 1, 2024
787e427
Updated to handle evt.time discrepency and updated template file
cole-labar Aug 2, 2024
47913ba
Updated to split out sample logs into their respective test suites an…
cole-labar Aug 7, 2024
78933d5
Merge branch 'feature/falco-testing-updates' into feature/ESI-21-Mapp…
cole-labar Aug 7, 2024
25049b6
Update from build test
cole-labar Aug 7, 2024
551c423
Updated with changes from "clean slate" branch
cole-labar Aug 8, 2024
90ffbb1
Updated from build process
cole-labar Aug 8, 2024
4276e25
Updated README
cole-labar Aug 9, 2024
b6866f4
Updated with syslog fix and adjusted pipeline for event.original
cole-labar Aug 13, 2024
f963586
Updated test definition files
cole-labar Aug 13, 2024
e6882db
Added in Falco Sidekick handling
cole-labar Aug 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@
/packages/etcd @elastic/obs-infraobs-integrations
/packages/f5 @elastic/security-service-integrations
/packages/f5_bigip @elastic/security-service-integrations
/packages/falco @elastic/security-service-integrations
/packages/fim @elastic/sec-linux-platform
/packages/fireeye @elastic/security-service-integrations
/packages/fleet_server @elastic/fleet
Expand Down
93 changes: 93 additions & 0 deletions packages/falco/LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
Elastic License 2.0

URL: https://www.elastic.co/licensing/elastic-license

## Acceptance

By using the software, you agree to all of the terms and conditions below.

## Copyright License

The licensor grants you a non-exclusive, royalty-free, worldwide,
non-sublicensable, non-transferable license to use, copy, distribute, make
available, and prepare derivative works of the software, in each case subject to
the limitations and conditions below.

## Limitations

You may not provide the software to third parties as a hosted or managed
service, where the service provides users with access to any substantial set of
the features or functionality of the software.

You may not move, change, disable, or circumvent the license key functionality
in the software, and you may not remove or obscure any functionality in the
software that is protected by the license key.

You may not alter, remove, or obscure any licensing, copyright, or other notices
of the licensor in the software. Any use of the licensor’s trademarks is subject
to applicable law.

## Patents

The licensor grants you a license, under any patent claims the licensor can
license, or becomes able to license, to make, have made, use, sell, offer for
sale, import and have imported the software, in each case subject to the
limitations and conditions in this license. This license does not cover any
patent claims that you cause to be infringed by modifications or additions to
the software. If you or your company make any written claim that the software
infringes or contributes to infringement of any patent, your patent license for
the software granted under these terms ends immediately. If your company makes
such a claim, your patent license ends immediately for work on behalf of your
company.

## Notices

You must ensure that anyone who gets a copy of any part of the software from you
also gets a copy of these terms.

If you modify the software, you must include in any modified copies of the
software prominent notices stating that you have modified the software.

## No Other Rights

These terms do not imply any licenses other than those expressly granted in
these terms.

## Termination

If you use the software in violation of these terms, such use is not licensed,
and your licenses will automatically terminate. If the licensor provides you
with a notice of your violation, and you cease all violation of this license no
later than 30 days after you receive that notice, your licenses will be
reinstated retroactively. However, if you violate these terms after such
reinstatement, any additional violation of these terms will cause your licenses
to terminate automatically and permanently.

## No Liability

*As far as the law allows, the software comes as is, without any warranty or
condition, and the licensor will not be liable to you for any damages arising
out of these terms or the use or nature of the software, under any kind of
legal claim.*

## Definitions

The **licensor** is the entity offering these terms, and the **software** is the
software the licensor makes available under these terms, including any portion
of it.

**you** refers to the individual or entity agreeing to these terms.

**your company** is any legal entity, sole proprietorship, or other kind of
organization that you work for, plus all organizations that have control over,
are under the control of, or are under common control with that
organization. **control** means ownership of substantially all the assets of an
entity, or the power to direct its management and policies by vote, contract, or
otherwise. Control can be direct or indirect.

**your licenses** are all the licenses granted to you for the software under
these terms.

**use** means anything you do with the software requiring one of your licenses.

**trademark** means trademarks, service marks, and similar rights.
3 changes: 3 additions & 0 deletions packages/falco/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
dependencies:
ecs:
reference: [email protected]
46 changes: 46 additions & 0 deletions packages/falco/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Falco Integration
This integration allows for the shipping of [Falco](https://falco.org/) alerts to Elastic for observability and organizational awareness. Alerts can then be analyzed by using either the dashboard included with the integration or via the creation of a custom dashboard within Kibana.

## Data Streams
The Falco integration collects one type of data stream: logs.

**Logs** The Logs data stream collected by the Falco integration is comprised of Falco Alerts. See more details about Falco Alerts in [Falco's Outputs Documentation](https://falco.org/docs/outputs/). A complete list of potential fields used by this integration can be found in the [Logs reference](#logs-reference)

## Requirements

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it.
You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Falco must be configured to output alerts to a supported output channel as defined in [Setup](#setup). The system will only receive fields output by Falco's rules. If a rule does not include a desired field the rule must be edited in Falco to add the field.

This integration is compatible with Falco version 0.37 and above, and should not be expected to perform successfully in lower versions.

## Setup

For step-by-step instructions on how to set up an integration, see the {{ url "getting-started-observability" "Getting started" }} guide.

In order to capture alerts from Falco you **must** configure Falco to output Alerts as JSON to one of the supported channels: [Logfile](#logfile-input) or [TCP Syslog](#tcp-syslog-input).

**Required:** To configure Falco to output JSON, set the config properties `json_output=true` and `json_include_output_property=true` in Falco's config. See the examples in Falco's [Output Channels documentation](https://falco.org/docs/outputs/channels/#http-output).

### Logfile Input

The logfile input reads data from one or more Falco log files using the Elastic Agent. Use this input when the Elastic Agent will be deployed to the same machine as Falco or when Falco's log files are available via a mounted filesystem.

To use this input Falco must be configured to output alerts to a log file. See Falco's [File Output](https://falco.org/docs/outputs/channels/#file-output) documentation for details.

### TCP Syslog Input

The TCP Syslog input allows the Elastic Agent to receive Falco Alerts via remote syslog. Use this input when you want to send data via [Falco Sidekick](https://github.com/falcosecurity/falcosidekick).

To use this input you will need to deploy the Elastic Agent *first* and then configure and deploy Falco Sidekick to send Alerts to the Agent via Syslog. See [Syslog Output](https://github.com/falcosecurity/falcosidekick/blob/master/docs/outputs/syslog.md) and [Connecting Falco to Sidekick](https://github.com/falcosecurity/falcosidekick?tab=readme-ov-file#connect-falco) for more details.

## Logs Reference

### alerts

Falco alerts can contain a multitude of various fields pertaining to the type of activity on the host machine.

{{ fields "alerts" }}

{{ event "alerts" }}
13 changes: 13 additions & 0 deletions packages/falco/_dev/deploy/docker/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
services:
falco-tcp:
image: docker.elastic.co/observability/stream:v0.15.0
volumes:
- ./sample_syslog:/sample_logs:ro
- ${SERVICE_LOGS_DIR}:/var/log
command: log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9030 -p=tcp /sample_logs/*.log
falco-logfile:
image: alpine
volumes:
- ./sample_logfile:/sample_logs:ro
- ${SERVICE_LOGS_DIR}:/var/log
command: /bin/sh -c "cp /sample_logs/* /var/log/"
Loading