Elastic Integration plugin for Logstash

This is a plugin for Logstash, and is not meant to be used on its own.

It can be used in a Logstash pipeline to perform the transformations that are applied by many Elastic Integrations before sending the events to Elasticsearch.

Documentation

Logstash provides infrastructure to automatically generate documentation for this plugin from its asciidoc source. All plugin documentation are placed under one central location.

Development

Prerequisites

• Clone the plugin repo
• Check out the branch you would like to build, such as 8.x or 9.x/main, etc...
  • Note that, 8.series branches require JDK17 and 9.series/main requires JDK21.
• Set the Logstash source path in the gradle.properties file. Open the file and update the LOGSTASH_PATH.
• Make sure Logstash is compiled.
• Point to the Elasticsearch branch/version to build the plugin with. Open the gradle.properties file and update ELASTICSEARCH_TREEISH.

Build

We use Gradle tool to build this plugin. Gradle is configured to load the Logstash core jars from the LOGSTASH_PATH path, download the Elasticsearch version defined with ELASTICSEARCH_TREEISH and build the plugin. The build process may fail if the Elasticsearch interfaces the plugin is using have changed.

The following command builds plugin and generates the jar file, can be locally installed in Logstash core and verified:

./gradlew clean vendor localGem

Plugin test

The recommendation to run tests to align on our common plugins/.ci tool. Run the following command under the plugin source dir to download the CI tool:

mkdir -p .ci-test && curl -sL --retry 5 --retry-delay 5 https://github.com/logstash-plugins/.ci/archive/1.x.tar.gz | tar zxvf - --keep-old-files --strip-components=1 -C .ci-test

Export environment variables:

SNAPSHOT=false                # set to true if targeting the SNAPSHOT
ELASTIC_STACK_VERSION=8.17.0  # stack version to run the plugin on
INTEGRATION=false             # set to true if you want integration tests to run, it also requires SECURE_INTEGRATION=true

Run tests on the local docker environment, make sure your docker-engine is running.

./.ci/docker-setup.sh && ./.ci/docker-run.sh

Running your unpublished plugin in Logstash

Run in a local Logstash clone

• Edit Logstash Gemfile and add the local plugin path, for example:

gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"

• Install plugin

bin/logstash-plugin install --no-verify

• Run Logstash with your plugin

bin/logstash -e 'filter {awesome {}}'

At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.

Run in an installed Logstash

• Edit Logstash Gemfile and add the local plugin path, for example:

gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"

• Build your plugin gem

./gradlew clean vendor localGem

• Install the plugin from the Logstash home

bin/logstash-plugin install --no-verify

• Start Logstash and proceed to test the plugin