Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add the otelcol syslog receiver #2263

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1aec317
Implementation of otelcol syslog receiver wrapper
dehaansa Dec 11, 2024
69a93f3
update unmarshal code
dehaansa Dec 11, 2024
306a43f
Merge branch 'main' into feat/otelcol-syslog-receiver
dehaansa Dec 11, 2024
ad69096
Add changelog entry for syslogreceiver
dehaansa Dec 11, 2024
cefb4c4
Change field key, add converter
dehaansa Dec 11, 2024
587d4a1
Apply suggestions from code review
dehaansa Dec 11, 2024
85b2cd0
Update whitespace attribute descriptions
dehaansa Dec 11, 2024
b22ebfb
Clean up commented out code
dehaansa Dec 11, 2024
520eeea
Clean up comments
dehaansa Dec 11, 2024
ad716de
Update autogenerated docs
dehaansa Dec 11, 2024
d00f726
Update docs/sources/reference/components/otelcol/otelcol.receiver.sys…
dehaansa Dec 11, 2024
47deaad
Update docs/sources/reference/components/otelcol/otelcol.receiver.sys…
dehaansa Dec 11, 2024
f79a9ac
Put a little more maximum delay in syslog test
dehaansa Dec 11, 2024
3a438a7
clean up validation, try not to use localhost for ci tests
dehaansa Dec 11, 2024
1072d10
Update docs/sources/reference/components/otelcol/otelcol.receiver.sys…
dehaansa Dec 12, 2024
27dcf40
Fix syslogreceiver converter
dehaansa Dec 12, 2024
d0e59ac
Merge branch 'feat/otelcol-syslog-receiver' of github.com:dehaansa/al…
dehaansa Dec 12, 2024
1d0b32a
Add test sleep
dehaansa Dec 12, 2024
f2942e4
Update docs/sources/reference/components/otelcol/otelcol.receiver.sys…
dehaansa Dec 12, 2024
ad373b7
Address some PR feedback
dehaansa Dec 12, 2024
70997f7
Update docs for rfc6587 arguments
dehaansa Dec 12, 2024
f278703
Add todo to test comment
dehaansa Dec 12, 2024
548b173
Update docs/sources/reference/components/otelcol/otelcol.receiver.sys…
dehaansa Dec 12, 2024
a9fc448
Apply suggestions from code review
dehaansa Dec 12, 2024
d0a0fb8
Merge branch 'main' into feat/otelcol-syslog-receiver
dehaansa Dec 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ Main (unreleased)

### Features

- Add `otelcol.receiver.syslog` component to receive otel logs in syslog format (@dehaansa)

- Add support for metrics in `otelcol.exporter.loadbalancing` (@madaraszg-tulip)

- Add `add_cloudwatch_timestamp` to `prometheus.exporter.cloudwatch` metrics. (@captncraig)
Expand Down
272 changes: 272 additions & 0 deletions docs/sources/reference/components/otelcol/otelcol.receiver.syslog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
---
canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.syslog/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clayton-cornell do we need to add a tag for public preview?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I missed that :-( I saw the shared lookup... forgot to make sure the badge was added.

Parallel to this, we are looking at better ways of adding badges to the docs... so we can avoid HTML spans.

description: Learn about otelcol.receiver.syslog
title: otelcol.receiver.syslog
---

# otelcol.receiver.syslog
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

`otelcol.receiver.syslog` accepts syslog messages over the network and forwards them as logs to other `otelcol.*` components.
It supports syslog protocols [RFC5424][] and [RFC3164][] and can receive data over `TCP` or `UDP`.

> **NOTE**: `otelcol.receiver.syslog` is a wrapper over the upstream
> OpenTelemetry Collector `syslog` receiver. Bug reports or feature requests will
> be redirected to the upstream repository, if necessary.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

Multiple `otelcol.receiver.syslog` components can be specified by giving them
different labels.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

[RFC5424]: https://www.rfc-editor.org/rfc/rfc5424
[RFC3164]: https://www.rfc-editor.org/rfc/rfc3164

## Usage

```alloy
otelcol.receiver.syslog "LABEL" {
tcp { ... }
udp { ... }

output {
logs = [...]
}
}
```




## Arguments
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

The following arguments are supported:

| Name | Type | Description | Default | Required |
|-----------------------------------|----------|--------------------------------------------------------------------|-----------|----------|
| `protocol` | `string` | The syslog protocol that the syslog server supports. | `rfc5424` | no |
| `location` | `string` | The geographic time zone to use when parsing an rfc3164 timestamp. | `UTC` | no |
| `enable_octet_counting` | `bool` | Whether to enable rfc6587 octet counting. | `false` | no |
| `max_octets` | `int` | The maximum octets for messages when octet counting is enabled. | `8192` | no |
| `allow_skip_pri_header` | `bool` | Allow parsing records without a priority header. | `false` | no |
| `non_transparent_framing_trailer` | `string` | The framing trailer when using rfc6587 Non-Transparent-Framing. | `false` | no |
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

The `protocol` argument specifies the syslog format supported by the receiver.
`protocol` must be one of `rfc5424`, `rfc3164`
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

The `location` argument specifies a Time Zone identifier. The available locations depend on the local IANA Time Zone database.
See [this wikipedia entry][tz-wiki] for a non-comprehensive list.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

The `non_transparent_framing_trailer` argument must be one of `LF`, `NUL`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some context around this? Like how can the default be nil? Even if its a string it should be "" but that seems to not be LF or NUL.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's fair. It's a *string, so it's nil if not set. If it is set, then the character chosen is expected to terminate the message. If not set, it's expected that you're using octet counting or that the syslog frame contains only one message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@clayton-cornell I'd appreciate any feedback you have on this docs change as well as Matt's.


[tz-wiki]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

## Blocks

The following blocks are supported inside the definition of
`otelcol.receiver.syslog`:


| Hierarchy | Block | Description | Required |
|------------------|----------------------|-------------------------------------------------------------------------------------------------|----------|
| udp | [udp][] | Configures a UDP syslog server to receive syslog messages. | no* |
| udp > multiline | [multiline][] | Configures rules for multiline parsing of incoming messages. | no |
| udp > async | [async][] | Configures rules for asynchronous parsing of incoming messages. | no |
| tcp | [tcp][] | Configures a TCP syslog server to receive syslog messages. | no* |
| tcp > multiline | [multiline][] | Configures rules for multiline parsing of incoming messages | no |
| tcp > tls | [tls][] | Configures TLS for the TCP syslog server. | no |
| retry_on_failure | [retry_on_failure][] | Configures the retry behavior when the receiver encounters an error downstream in the pipeline. | no |
| debug_metrics | [debug_metrics][] | Configures the metrics that this component generates to monitor its state. | no |
| output | [output][] | Configures where to send received telemetry data. | yes |

A syslog receiver must have either a `udp` or `tcp` block configured.

The `>` symbol indicates deeper levels of nesting. For example, `tcp > tls`
refers to a `tls` block defined inside a `tcp` block.

[tls]: #tls-block
[udp]: #udp-block
[tcp]: #tcp-block
[multiline]: #multiline-block
[async]: #async-block
[retry_on_failure]: #retry-on-failure-block
[debug_metrics]: #debug_metrics-block
[output]: #output-block

### udp block

The `udp` block configures a UDP syslog server.
The following arguments are supported:

| Name | Type | Description | Default | Required |
|---------------------------------|----------|--------------------------------------------------------------------------------------------------------------|---------|----------|
| `listen_address` | `string` | The `<host:port>` address to listen to for syslog messages. | | yes |
| `one_log_per_packet` | `bool` | Skip log tokenization, improving performance when messages always contain one log and multiline is not used. | `false` | no |
| `add_attributes` | `bool` | Add net.* attributes to log messages according to OpenTelemetry semantic conventions. | `false` | no |
| `encoding` | `string` | The encoding of the syslog messages. | `utf-8` | no |
| `preserve_leading_whitespaces` | `bool` | Trims leading whitespace on messages unless this is set to `true`. | `false` | no |
| `preserve_trailing_whitespaces` | `bool` | Trims trailing whitespace on messages unless this is set to `true`. | `false` | no |
clayton-cornell marked this conversation as resolved.
Show resolved Hide resolved

The `encoding` argument specifies the encoding of the incoming syslog messages.
`encoding` must be one of `utf-8`, `utf-16le`, `utf-16be`, `ascii`, `big5`, `nop`.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved
See the upstream receiver [documentation][encoding-documentation] for more details.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

### multiline block

The `multiline` block configures logic for splitting incoming log entries.
The following arguments are supported:

| Name | Type | Description | Default | Required |
|----------------------|----------|-----------------------------------------------------------------|---------|----------|
| `line_start_pattern` | `string` | A regular expression that matches the beginning of a log entry. | | no |
| `line_end_pattern` | `string` | A regular expression that matches the end of a log entry. | | no |
| `omit_pattern` | `bool` | Omit the start/end pattern from the split log entries. | `false` | no |

A `multiline` block must contain either `line_start_pattern` or `line_end_pattern`.

If a `multiline` block is not set, log entries will not be split.

### async block

The `async` block configures concurrent asynchronous readers for a UDP syslog server.
The following arguments are supported:

| Name | Type | Description | Default | Required |
|--------------------|-------|----------------------------------------------------------------------------------|---------|----------|
| `readers` | `int` | The number of goroutines to concurrently read from the UDP syslog server. | `1` | no |
| `processors` | `int` | The number of goroutines to concurrently process logs before sending downstream. | `1` | no |
| `max_queue_length` | `int` | The maximum number of messages to wait for an available processor. | `100` | no |

If `async` is not set, a single goroutine will read and process messages synchronously.

### tcp block

The `tcp` block configures a TCP syslog server.
The following arguments are supported:

| Name | Type | Description | Default | Required |
|---------------------------------|----------|--------------------------------------------------------------------------------------------------------------|---------|----------|
| `listen_address` | `string` | The `<host:port>` address to listen to for syslog messages. | | yes |
| `max_log_size` | `int` | The maximum size of a log entry to read before failing. | `1MiB` | no |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a string.

| `one_log_per_packet` | `bool` | Skip log tokenization, improving performance when messages always contain one log and multiline is not used. | `false` | no |
| `add_attributes` | `bool` | Add net.* attributes to log messages according to OpenTelemetry semantic conventions. | `false` | no |
| `encoding` | `string` | The encoding of the syslog messages. | `utf-8` | no |
| `preserve_leading_whitespaces` | `bool` | Trims leading whitespace on messages unless this is set to `true`. | `false` | no |
| `preserve_trailing_whitespaces` | `bool` | Trims trailing whitespace on messages unless this is set to `true`. | `false` | no |
clayton-cornell marked this conversation as resolved.
Show resolved Hide resolved

The `encoding` argument specifies the encoding of the incoming syslog messages.
`encoding` must be one of `utf-8`, `utf-16le`, `utf-16be`, `ascii`, `big5`, `nop`.
See the upstream receiver [documentation][encoding-documentation] for more details.

The `max_log_size` argument has a minimum value of `64KiB`

### tls block

The `tls` block configures TLS settings used for a server. If the `tls` block
isn't provided, TLS won't be used for connections to the server.

{{< docs/shared lookup="reference/components/otelcol-tls-server-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

### retry on failure block

The `retry_on_failure` block configures the retry behavior when the receiver encounters an error downstream in the pipeline.
A backoff algorithm is used to delay the retry upon subsequent failures.
The following arguments are supported:

| Name | Type | Description | Default | Required |
|--------------------|------------|-----------------------------------------------------------------------------------------------------------|--------------|----------|
| `enabled` | `bool` | If true, the receiver will pause reading a file and attempt to resend the current batch of logs on error. | `false` | no |
| `initial_interval` | `duration` | The time to wait after first failure to retry. | `1 second` | no |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Durations should be in formatted to 1s or 5m

| `max_interval` | `duration` | The maximum time to wait after applying backoff logic. | `30 seconds` | no |
| `max_elapsed_time` | `duration` | The maximum age of a message before the data is discarded. | `5 minutes` | no |

If `max_elapsed_time` is set to `0` data will never be discarded.

### debug_metrics block

{{< docs/shared lookup="reference/components/otelcol-debug-metrics-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

### output block

{{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="<ALLOY_VERSION>" >}}

## Exported fields

`otelcol.receiver.syslog` does not export any fields.

## Component health

`otelcol.receiver.syslog` is only reported as unhealthy if given an invalid
configuration.

## Debug information

`otelcol.receiver.syslog` does not expose any component-specific debug
information.

## Debug metrics

`otelcol.receiver.syslog` does not expose any component-specific debug metrics.

## Example

This example proxies syslog messages from the `otelcol.receiver.syslog` receiver to the
`otelcol.exporter.syslog` component, and then sends them on to a `loki.source.syslog` component
before being logged by a `loki.echo` component. This shows how the `otelcol` syslog components
can be used to proxy syslog messages before sending them to another destination.

Using the `otelcol` syslog components in this way results in the messages being forwarded as sent,
attempting to use the `loki.source.syslog` component for a similar proxy use case requires
careful mapping of any structured data fields through the `otelcol.processor.transform` component. A
very simple example of that can be seen in the [`otelcol.exporter.syslog`][exporter-examples] documentation.
dehaansa marked this conversation as resolved.
Show resolved Hide resolved

```alloy
otelcol.receiver.syslog "default" {
protocol = "rfc5424"
tcp {
listen_address = "localhost:1515"
}
output {
logs = [otelcol.exporter.syslog.default.input]
}
}

otelcol.exporter.syslog "default" {
endpoint = "localhost"
network = "tcp"
port = 1514
protocol = "rfc5424"
enable_octet_counting = false
tls {
insecure = true
}
}

loki.source.syslog "default" {
listener {
address = "localhost:1514"
protocol = "tcp"
syslog_format = "rfc5424"
label_structured_data = true
use_rfc5424_message = true
}
forward_to = [loki.echo.default.receiver]
}

loki.echo "default" {}
```

[exporter-examples]: ../otelcol.exporter.syslog/#use-the-otelcolprocessortransform-component-to-format-logs-from-lokisourcesyslog
[encoding-documentation]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/syslogreceiver/README.md#supported-encodings
<!-- START GENERATED COMPATIBLE COMPONENTS -->

## Compatible components

`otelcol.receiver.syslog` can accept arguments from the following components:

- Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters)


{{< admonition type="note" >}}
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
Refer to the linked documentation for more details.
{{< /admonition >}}

<!-- END GENERATED COMPATIBLE COMPONENTS -->
3 changes: 3 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatatest v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/pdatautil v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.112.0
Expand All @@ -143,6 +144,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.112.0
github.com/ory/dockertest/v3 v3.8.1
Expand Down Expand Up @@ -845,6 +847,7 @@ require (
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/kafka/topic v0.112.0 // indirect
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/influxdbreceiver v0.112.0
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/valyala/fastjson v1.6.4 // indirect
go.opentelemetry.io/collector/connector/connectorprofiles v0.112.0 // indirect
go.opentelemetry.io/collector/consumer/consumererror v0.112.0 // indirect
go.opentelemetry.io/collector/consumer/consumererror/consumererrorprofiles v0.112.0 // indirect
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2611,6 +2611,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/extension/oauth2client
github.com/open-telemetry/opentelemetry-collector-contrib/extension/oauth2clientauthextension v0.112.0/go.mod h1:F7NU4rHqbTrkfOFH6ZtbSHoD+vLNgfu0MrqDw5n1I8Y=
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.112.0 h1:XYTTlyT5xjZKcUaQT05ffltMahw2S2wU7qIWtjLp7XE=
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.112.0/go.mod h1:wlnJiEwFYq3920DXOgDby5w6ctv57GJUTkvH4gHoBVA=
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage v0.112.0 h1:OtZFEz8PEAcGJFGAI3m1bu8gC3rS8IbtnB5mO8B9AAU=
github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage v0.112.0/go.mod h1:ykfaFUQlOIuWrWSwc4wtY0TDwWjjGHF/8jNm3lFH0cM=
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/ecsutil v0.112.0 h1:3zGjQ0pRszCibVGvjqTWDVEDT0D+d5pY8JRy8rV8X9k=
github.com/open-telemetry/opentelemetry-collector-contrib/internal/aws/ecsutil v0.112.0/go.mod h1:0vf3+lFg/yOaXwQ17MAF2JmBkTGeq09qR+ftaJQqN08=
github.com/open-telemetry/opentelemetry-collector-contrib/internal/common v0.112.0 h1:PVgAm7sIQUOS8TtX5ANV+hHn67vW6cW6uVy3qifccKc=
Expand Down Expand Up @@ -2715,6 +2717,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusrec
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.112.0/go.mod h1:q2lFBHfnG+ar2DJJlIU6RviOFXDeFur9vJ083NvOMQs=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0 h1:cHk8vS/D1pjeZ0o4LJJAENP847HHWjTXFe4y1RJYlfo=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/solacereceiver v0.112.0/go.mod h1:2CK7Hh6UGLnBSGW7Y0nopvEhoo25D6t/395jFEephEs=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.112.0 h1:GNWvYGjT08ByMbKuvY/uB57TQYrPJc/aF+nnpraELgU=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/syslogreceiver v0.112.0/go.mod h1:U0XNYcs+DJTwElKNKXADGBpQLIFrrEKAI78PzqOVl/E=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0 h1:Vv1FDwd7pykzj8Wmuc7yj7bcN0qUv1mGBb/dcTMPfNE=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver v0.112.0/go.mod h1:lklLK8ELD2Wk5z7ywjaf6XEbbViDtf7uK8jAExjRlls=
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.112.0 h1:XhKHjEpQJQMaUuWVhWS1FEuaY4LJDwBgsGXE166j9SY=
Expand Down
Loading
Loading